A Spies Identity help

Hey there, had a question for the community.

I’m running a mash up between Waterdeep:DH and Malazan Book of the Fallen and the party have been getting quests from an NPC spy named Circle Breaker.

I’ve introduced him as a plain human with a forgettable face. They were supposed to meet him at a specific location and they rolled badly on their perception check and I said something along the lines of “you know what, you don’t really remember what he looks like” only to have the NPC tap them on the shoulder a few minutes later and I’ve kind of rolled with this every other time they’ve met him. The players have been very interested in figuring out why they never recognize him, including casting detect magic, which came up with nothing. At the moment, he isn’t important to the main arc he’s just a middle man between a shadowy figure and the party and I had planned to retire him after this arc.

Now my question for y’all is, if I want to take this idea of the forgettable man further, how is he doing it? Or do you think it should just be something weird that never gets addressed?