If it's hot enough to swim and you're in a remote location you'll probably go for a dip regardless of whether or not you brought your swimming costume.
It's your choice.
One thing to bear in mind though is that mountain lakes can be deep and very cold, much colder than the sea. This can induce cramp. So think twice before swimming out.