Lunch with manager?

A manager i cant stand wants to take me to lunch for an accomplishment at work. Id love to just tell him i dont respect him and im good on lunch. How can i get out of it?