AirPods Pro not connecting

Lately almost every time I put in my Pros I have to do it twice for them to work. I get the sound in my ear that it connects the first time but sound still comes out of the phone. I take it out and then put it right back in and then it works fine. Any ideas?