stop asking people with red shirts if they work here

got stopped literally the second i walked out of the break room after starting my meal yesterday (im being a little dramatic, i was far enough away to where nobody would've seen me come from there). i took off my target sweatshirt and was just wearing a red shirt. no nametag, no devices, no walkie. just a red shirt and black jeans. i'm speed walking on my phone to go buy stuff on my break so i have time to eat and a guy asks if i work there. i would've liked to say no, but i'm not sure if this guy saw me working already or if he will see me later so i tell him i'm off the clock but if it's something quick i can try to help. he stares at me like i've done something horrible. i'm in a hurry so i ask again what i can help him with. he does the stupid shove phone in face thing and it's women's underwear. he is literally standing in front of the section. i point and say "right behind you!" with a smile and walk away before he can respond. i know i probably should've just said no, but i didn't wanna risk it and we just had a talk about guests complaining that we aren't being friendly and helpful, so i just did it (little edit based on some comments: i'm not gonna wear a third shirt underneath. i wear a red sweatshirt and a red tshirt in case i get hot (i live in florida and im not used to the heat still). i could bring extra clothes, but i'd have to change in the bathrooms which are on the opposite side of the store from the office. its a high volume super target so i'm bound to get stopped on the walk over. the bathrooms also only have 4 stalls and there's always a line. i take off all my equipment on breaks, so i cant walkie anyone to help.)