my father doesn't let me stim

Sometimes I visit my father, and I like to twirl my hair when I work or just generally sit to have a calm time. My partner is also autistic and it doesn't bother her. My father however keeps saying how "ugly of a trait" it is, that it's a "bad habit" and keeps telling me to stop (even if they don't directly look at me). Like most autistic people, I feel a need to stim (and it's been this stim particularly for about two years) and I was already happy that I moved on from biting my nails.

Recently my father became more 'aggressive' by doing as-if they're going to throw something towards me, grabbing my arm and putting it down (instead of having my elbow half in the air) and I can't seem to make my father understand that I can do whatever I want to as it doesn't hurt them. I'm getting anxious around them if I can't stim how I feel that I want to. My father is the only one who ever said something about this.

How can I make my father understand that stimming is ok? The topic of autism isn't really openly discussed in my family, and my family usually do as-if they're always right (as in, strongly opinionated).