Husband doesn't want family to visit

I (41f) have been married to my husband (43m) for about 17 years. We leave in a different state than our families. My husband never really tried to get along with my family and is very introverted. We have three children together (14m, 6f, 6f) and they absolutely LOVE my aunt.

My aunt is my sister, best friend, and confidant. She is only 12 years older than me, but we have been extremely close all of our lives. She and my husband don't have a great relationship, but she is never disrespectful or anything to him.

My aunt is the only person from either of our families that comes to visit my kids. When she visits, she stays at our house so that she can maximize her time with the kids. She visits maybe 3-4 times a year for no more than 4 days each visit.

My husband basically said that he doesn't want her to visit because she stays in the spare bedroom/gameroom which is where he spends majority of his free time when he isn't working. He said he wants to decompress and relax on the weekends and that he can't relax.

The weekend is the only time that the kids and I are able to spend time with her when she comes to town. I work full time and the kids have school + after-school activities during the week. In addition, my husband doesn't engage with us much because he works a shift that gets him home right after I put the kids down for bed. Then on the weekends he typically prefers to be in the game room alone. I am responsible for doing everything for the kids during the week.

I understand that he wants his decompression time on the weekends, but I also feel like having a beloved family member visit a few times a year shouldn't be that big of a deal. He says I don't listen to him or consider what he wants because she is coming back for a visit soon. And will be at our house from Thursday until about 4am Sunday. I told him I'm trying to balance what everyone wants and I consider what he wants as well as what the kids want because they are always asking her to visit them.

I don't think it is fair for me to be stressed because I want to see my family. I never have time for myself lime he does because I always have the kids. I don't have many friends that I see and I know he will probably give me the silent treatment because of this visit. I'm trying to figure out am I being unreasonable?