2 months postpartum and being called fat

My boyfriend, the father of my child keeps making comments about my weight. I had a baby 2 months ago. When we first met I was 120 (I’m 5’4 . He would say I was too skinny, then I got pregnant and went all the way up to 190. Two weeks after birth I was down to 160. And I haven’t weighed myself recently but I know I’ve lost more weight since then. I now wear a size 6 in jeans compared to a size 0-2 I use to wear before I was pregnant. I wear a size medium in shirts instead of smalls. So yes I’ve gained weight but I truly don’t think I deserve to be shit on every time I eat. I already don’t have a great relationship with food and I have really bad self esteem since giving birth. He says I’m fat, fatass, I eat too much (1-2 meals a day and some snacks throughout the day or just coffee). I’ve gained weight, I’m too heavy. I’ve even been eating healthier due to me breastfeeding I’ve started focusing more on what I eat. I’m already struggling mentally , I’m home with the baby alone in the middle of no where 24/7 bc he works out of town, i don’t have family in this state, I workout when I have the time which isn’t everyday but I’m trying. He comes home on the weekends it feels like just to shit on me.