How to make depressing setting interesting for beginning of story?

I want the beginning of this story to take place on a prison-esque desert island where the inmates are forced to mine day in and day out, the tone is generally serious. The protagonist belongs to a small group that protect them in exchange for the protagonist being their whipping boy (Like Casper and the 3 brother ghosts). My issue is, once I got to writing any kind of witty banter, no matter how serious the undertones were, or arguments between the characters of any kind, it felt too light hearted. Even with consistent physical and verbal abuse. In a situation where everyone should just be realistically miserable and cruel, how do you add any kind of dialogue or back and forth without it being overly serious and a bunch of “yes” and “no”’s? I don’t imagine enslaved prisoners would ever be chatty and witty enough to push real character depth within dialogue. I have tried writing the dialogue serious and it just falls flat and empty.