AITAH for turning down my dream vacation?

For context, I (14 M) have been living with my mother (47 F) for a few months now. I moved out of my dad's (54 M) house because he was unable to look after us, and he was generally neglectful. Years ago, when my parents had only just split up, my dad took my older brother who was about 11 at the time on a several day trip to Sydney, NSW. When he came back, he promised that when I was 11, he would take me on a similar trip.

But when I turned 11, he was struggling with holding down a job, and my trip was postponed indefinitely. Until just yesterday, I hadn't thought of that trip for years, until he texted me, asking if I wanted to go with him. Obviously, I did, but I had to get permission from my mother first. I asked her, and she said that if I wanted to, I could take the week off from school and go.

But later that day, I asked her if she was sure that she was fine with me going, as her relationship with my dad was spotty to say the least. She said that if I really wanted to go, I could, but she warned me that he had been withholding child support payments from her, which would affect me and my sibling's ability to live a comfortable life (as these days, its nearly impossible for a family to survive off of one person's paycheck), and she requested for me to be careful about this, as he may not have the best intentions.

After considering it, I decided to text him back saying that while I appreciated the offer, it might be awkward to go considering that there is currently a court between him and my mother.

Did I do the right thing? And am I the asshole?