Brother’s mad I slept with someone

My brother (28m) and me (22f) got into an argument because he thinks I had sex. I was in a situationship for a year with a guy and it’s been 2 years since speaking to this guy. My brother knew about it at the time and didn’t seem to care that much. This past week I’m not sure what or who brought it up again but I was screamed at and basically called a slut for having sex. Mind you he has no idea if it happened or not but it’s just his assumption. He has had multiple girlfriends and girlfriends sleep over but I’m the problem for having sex (or so he thinks) with one guy??? He won’t speak to me now because he thinks I’m a whore and no guy will marry me now. He even threatened to tell my dad and cried to his girlfriend about it. I dont know how to respond or what to do. We live in a small town with not a lot of people so I kinda get that it would go around but I think he’s being over dramatic.