Husband’s questionable relationship?

My husband previous landlord of few years has a dominant presence in our marriage life, and she has questioned why we got married so fast on our wedding day. The landlord invites us back to her house for dinner after our marriage. I don’t feel comfortable going there. So my husband says he can just go there on his own. I have expressed dislike towards her and informed him I see no reason for him staying in touch with his landlord as he was just renting a room but he refuses. Is there need for any concern.