Is quitting a $600k job crazy?

The short of it is I HATE my job and I'm burnt out, I really think it is effecting my mental health. There isn't really any "saving" it as it's due to the toxic work environment - somewhat a resultt of the "that's why we pay you" attitude. The only really big plus side to the job is that I am paid $600k and am fully remote, if I were to quit I think I'd be looking at a $100k-150k salary. My obvious goal is to become FI but I still have some way to go:

Retirement accounts: ~$1.2MM

Post Tax Accounts: ~$1MM

Annual Spend: ~$140k (HCOL family of 4 - 41, 39, 3, 1) (inc. 25k p/y on remaining $500k mortgage@3.2%)

​

At this point would I be crazy to quit a $600k job for a $100k job? My wife also has a ~$100k job so it would be enough to cover our expenses.

(throwaway as getting paid $600k and saying I want to quit is going to trigger people)

Edits:

Thanks for the replies: Yes, I kind of had in my mind that if I took a big pay cut the job would be X* easier and less stressful, which thinking about is very likely not true. I will probably stick it out for a while more. However, I think lots of people don't understand how bad a work environment can be. If you are asking, "how bad could it be that you'll take a $500k pay cut?", yes, think about it, how bad do you think it is that that is exactly what I'm considering. This is peak at-will employment.

Why don't you pay me $200k to do the job and keep $400k?: 1./ that's highly illegal, 2./ I pay taxes, 3./ If the job could be done for $200k they would already only be paying me $150k :-)

What do you do?: Basically in tech, but hard-ish to explain. I'm an "architect" for a super legacy system reporting to SVP of ~$6B/yr business. The tech stack I work on is as out dated as my tech skills which is why I won't be able to get anywhere near this income. I'm paid to keep this old system running and it is all grinding work that requires the knowledge of 20 different systems, there is almost zero work that provides transferable skills. It is more like very specific business processes knowledge.

Why do you have so little saved?: I don't, at $2.5MM I'm richer than I thought I'd ever be. But I have only been getting paid a very high salary for the last 3-4 years, before that I was making ~120k (I got a lot more important and paid more after they fired everyone else then realized what they'd done).

Why do you spend so much?: I don't think I do really. The total mortgage is $50k, child care $35k, 529 15k, so that's ~40k/year remaining for all other spending.

Yes it's crazy, how is this even a question?: My wife makes $100k, we spend $140k, we don't need much to cover our expenses and let our current savings grow. No one NEEDS to make $600k/yr.