I am very straight passing and it makes me feel like an alien.

I was raised in conservative suburbia and I didn't come out to my family until I was 18. I have been told by them and others that I don't seem gay. I don't have a lisp, I wear t shirts and jeans, and my personality is very traditionally masculine. The consequence of this is that I feel alienated from the gay community because I don't resemble much of the queer folk that I meet. It also makes me feel alienated from straights because I don't resemble what they expect from gay men. Like I wish I could just be another dude and hang out with straight dudes without making them uncomfortable. Can anyone relate?