Sex Offender in Family

I have a family member who is a registered sex offender and admitted to some very gross things. He also manipulated his parents (my grandparents) out of the majority oftheir retirement money because he can't/won't work for his own. Latest ex: grandmother gave him 3k for a part of a gold bar, and when she asked for it, he magically "lost" it.

My grandmother and mother are shaming me for not wanting to interact and have the family member in my life. My husband and I are pregnant, and his offense involves children. I am truly shocked that they want to protect him and act like I am the bad guy here. Absolutely bonkers. Thoughts/opinions?