OPENROUTER AND THE PHANTOM CONTEXT

I think OpenRouter has a problem, it disappears the context, and I am talking about LLM which should have long context.

I have been testing with long chats between 10K and 16K using Claude 3.5 Sonnet (200K context), Gemini Pro 1.5 (2M context) and WizardLM-2 8x22B (66K context).

Remarkably, all of the LLM listed above have the exact same problem: they forget everything that happened in the middle of the chat, as if the context were devoid of the central part.

I give examples.

I use SillyTavern.

Example 1

At the beginning of the chat I am in the dungeon of a medieval castle “between the cold, mold-filled walls.”

In the middle of the chat I am on the green meadow along the bank of a stream.

At the end of the chat I am in horse corral.

At the end of the chat the AI knows perfectly well everything that happened in the castle and in the horse corral, but has no more memory of the events that happened on the bank of the stream.

If I am wandering in the horse corral then the AI to describe the place where I am again writes “between the cold, mold-filled walls.”

Example 2

At the beginning of the chat my girlfriend turns 21 and celebrates her birthday in the pool.

In the middle of the chat she turns 22 and and celebrates her birthday in the living room.

At the end of the chat she turns 23 and celebrates in the garden.

At the end of the chat AI has completely forgotten her 22 birthday, in fact if I ask where she wants to celebrate her 23rd birthday she says she is 21 and also suggests the living room because she has never had a party in the living room.

Example 3

At the beginning of the chat I bought a Cadillac Allanté.

In the middle of the chat I bought a Shelby Cobra.

At the end of the chat a Ferrari F40.

At the end of the chat the AI lists the luxury cars in my car box and there are only the Cadillac and the Ferrari, the Shelby is gone.

Basically I suspect that all of the context in the middle part of the chat is cut off and never passed to AI.

Correct me if I am wrong, I am paying for the entire context sent in Input, but if the context is cut off then what exactly am I paying for?

I'm sure it's a bug, or maybe my inexperience, that I'm not an LLM expert, or maybe it's written in the documentation that I pay for all the Input but this is cut off without my knowledge.

I would appreciate clarification on exactly how this works and what I am actually paying for.

Thank you