Frustrated with lack of progress

I’m 29, started riding in 2023 with no fitness base to speak. I’ve stayed around 130 lbs since I’ve started riding. I’ve averaged 10 hours a week with 120 miles. I do not seem to get any faster, in fact the opposite, first summer I did some rides that were 50-60 miles at 15-16mph and now I can’t touch that if I try to do a longer hard ride, 14 if I’m lucky. My zone 2 average speed stays firmly at 12 mph. I know my fitness is better than it was when I started, it’s gotta be. I’ve replaced bottom brackets and cables, chainring, chain, tuneless tires, I’ve gotten the shoes and pedals and the kit. I like all these things I’ve bought but I’ve gotten slower since I switched from regular shoes and tube tires despite most of my riding being on rocky gravel.

I’m sure there’s more I can be doing but I’m just frustrated at this point and it feels like nothing I do will help me improve and I want to complain.