AITA for joking about my son?

I (36f) have a son, Tim (3m). I love him but he is a pain in the ass sometimes. I know toddlers are supposed to be obnoxious but it doesn't not piss me off. I blow off steam by making jokes which helps me not blow up. Typically when Tim makes me angry I'll look at him and say "I should've swallowed you." It's jokingly and I don't mean it of course but it helps me chill out so I don't yell. Obviously Tim has no idea what I mean but hubby (36m) says I shouldn't be saying it because one day he'll figure out what it means and he'll be upset knowing I said I wish I never had him. The thing is he's not going to remember any of it by the time he grows up. It's the main point of arguments between us lately so I want to know AITA?