One of my friends, a keen triathlete/cyclist, once told me that the main reason for him shaving his legs was that the presence of hair in a fall/scrape causes a burn. Without the hair, the majority of the wound is a cut. I.e. the carpet of hair is the first to heat up and causes a wide(r)spread burn than simply hitting concrete with fresh skin.
Shaving for swimming is more for aerodynamics (or should that be aquadynamics?) but I don't think I'd see any difference doing 12 lengths of breaststroke with vs without my hairy legs and hobbit feet!