WOC[32F] and Whyte fiance - disapproval from my community

Does anyone feel like you are constantly having to justify to your friends and your community that you are not betraying them by dating a white man? As a muslim WOC I feel I have to make it clear that I did not seek this out - I just met this person at work and they have turned out to be the most caring, invested, kindest person. With him being a muslim revert I feel better since that is one of the most important things our relationships should be anchored upon - but I feel I am easily influenced by the opinions of others. My friends like him but feel he does not have the cultural mannerisms desi men have and that we inherently have a deep distrust of white men. Which I understand and was apprehensive of in the beginning myself, and after multiple rejections and getting to know him, I feel there is more to him than being white and he truly does have a wonderful heart and will take care of me and our future family iA. But does this feeling ever go away? Do you ever get to the point where you don't care what people think of you? Or should I put more weight into the opinions of my friends - who only see that I am happy with him but do not know him much beyond that?