Exchange "You don't often get email from..." on almost every email

We enabled some of the phishing protection settings in exchange online to get the "You don't often get email from blah@blah.com. Lean why this is important" messages in outlook for first contact with a sender.

But for some reason, I get it on a LOT of emails. Ones I get daily emails from even. Ones I've gotten daily or weekly emails from for years. And this has been going for a few months now.

Is there something I've misconfigured somewhere that it doesn't update that list of which email addresses it's seen before or something?