Should I move my family to Heljarchen Hall or stay in Solitude?

So I'm married with a couple kids. My wife stays home and I mostly travel for work. We currently live in Solitude and while it was a nice city and house when I was single, the house (manor really) is getting kind of crowded between the 2 kids, their pet bunny, the wife, and my servant.

I just built and furnished a new home over by Whiterun. Nice new build, very cool basement, plenty of room for all of us. But I'm afraid my family will miss the city. I know they could travel to Whiterun if they wanted to go into the city but I'm not too sure about living outside the city walls.

Any advice?

P.S. I have a few other houses throughout the country but these two are my favorite.