While the destination has earned a reputation for its sometimes-stuffy, seen-and-be-seen vibe, there are some more low-key spots worth exploring. Take Sag Harbor, for example: Part of both Southampton and East Hampton, this unincorporated village is most famous for its whaling history and laid-back atmosphere.