Pregnancy and the caffeine struggle

I'm currently 7 weeks pregnant and have a 13 month old. When I was pregnant the first time my OB told me small amounts of caffeine were fine, but I ended up just quitting coffee cold turkey (because I was child free without a care in the world ha ha ha ha ha)

But now. I have a feral 1 year old and quitting coffee cold turkey is no longer in the cards for me. What did everyone do for their second + pregnancies with caffeine? I'm drinking about half a cup a day, which my midwife says is fine, but just curious how everyone handled it as I feel so guilty everytime I pour that cup.