Absolutely true Matt, there is no magic mojo. Me personally, I enjoy cold weather. Within reason, the colder the better. I also enjoy diving cold weather. My first ice dive were in a wet suit and it was years before I went dry on ice dives. In fact those in dry suits were initially referred to as whimps. Now, I call them the smart ones.
I have found though that those who enjoy the colder weather and colder temps far tolerate the cold water better than others.
I do a some winter camping and have done a few winter exercises with the USMC and I have always told my folks that cold is an attitude! If you look at the thermometer and say "Cool! -15*!" you do a lot better than the person that says "Holy Crap!!! it's 15*!!!!" Being prepared is another part of it too.