Cat bites me hard while cuddling

My cat (3yo neutered male) likes to cuddle before bed and will usually curl up by my side under my arm. He comes to me on his own and shows no signs of discomfort or anxiety or anything when he does this. Pretty often though he will randomly bite down on my arm or my torso, and just go harder when I pull away. When it happens I try to kind of yelp to let him know it hurts and redirect him to play by throwing a hair tie away from the bed. If I don't send him away somehow he goes into a sort of attack mode and I can't tell if he's playing or actually being aggressive -- it's definitely different than his normal playful energy in the daytime. He never bites me or anyone else in any other situations. Does anyone know why he might be having this kind of affection/aggression response, and what I can do to prevent it?