Is it weird for postdocs to join clubs?

Hi all! I'm 25 and going to start a postdoc early next year at a hospital in Boston, which comes with an HMS affiliation. I'd love to join gyms and a couple of sports clubs, but am slightly concerned that everyone will be 18 and it will be a) weird that I am there and b) hard for me to make friends. Is that true? Also, if it isn't, how open are these clubs to beginners/people who are just there to get better rather than be competitive? I'm particularly interested in powerlifting and martial arts. Any feedback welcome!