Every few months, a new announcement drops. “2 million tokens!” “Now supporting entire codebases!” The numbers keep getting bigger, and marketing departments keep getting bolder.

Here’s what they’re not telling you: that 128K, 1M, or 2M context window? It’s a theoretical maximum, not a practical reality. And if you’re building production systems based on those numbers, you’re heading for trouble.

We’ve spent the last eight months testing context windows in production at Rotavision. What we found isn’t pretty.

The Attention Decay Problem

Transformers don’t treat all tokens equally. They can’t. The attention mechanism that makes these models work has a dirty secret: it degrades in the middle.

Think of it like a conversation at a loud party. You remember how it started. You’re paying attention to what’s happening right now. But that thing someone said twenty minutes ago? Good luck.

graph LR
    subgraph "Lost in the Middle Phenomenon"
        A["START<br/>(Strong Attention)"] --> B["MIDDLE<br/>(Degraded Attention)"]
        B --> C["END<br/>(Strong Attention)"]
    end

    style A fill:#4ade80,stroke:#16a34a,color:#000
    style B fill:#fca5a5,stroke:#dc2626,color:#000
    style C fill:#4ade80,stroke:#16a34a,color:#000

Attention Strength by Position

This isn’t theoretical. In our testing, retrieval accuracy for information placed in the middle of a long context dropped by 40-60% compared to information at the beginning or end.

The Numbers Nobody Publishes

We ran a systematic evaluation across three major foundation models. The task was simple: retrieve specific facts placed at various positions within contexts of different lengths.

Retrieval Accuracy by Context Length (Middle Position)

* Results averaged across 3 major foundation models, 500+ test cases each

The pattern is consistent: meaningful retrieval accuracy starts degrading around 10K tokens and falls off a cliff past 32K. By the time you hit 100K+, you’re essentially playing retrieval roulette.

Why This Breaks Production Systems

Here’s where it gets painful. I’ve talked to engineering teams at a dozen Indian enterprises over the past year. The same story keeps repeating.

The demo works beautifully. You stuff 50 pages into the context, ask a question, get a perfect answer. Everyone’s impressed. The procurement team signs off.

Six months later, the support tickets start rolling in. “The AI missed this critical clause in the contract.” “It contradicted information from page 12.” “It confidently cited something that wasn’t in the document.”

What happened? The demo used carefully curated content with the important bits near the beginning. Production uses real documents where critical information is buried on page 47.

The Confidence Inversion Problem

This is the part that keeps me up at night.

When models fail on long-context tasks, they don’t fail gracefully. They don’t say “I’m not sure about this.” They hallucinate with complete confidence.

Confidence vs Accuracy in Long Contexts

■ Danger Zone: High Confidence + Low Accuracy

We call this “confident confabulation.” The model finds something in the context that’s vaguely related to the query, synthesizes a plausible-sounding response, and delivers it with the same confidence it would have for a correct answer.

For enterprise applications - legal documents, medical records, financial reports - this is catastrophic.

What Actually Works

After a lot of trial and error, here’s what we’ve learned works in production:

Keep working contexts under 10K tokens. Yes, really. The model can technically handle more, but your reliability drops with every additional token. If you need to process longer documents, you need a different architecture.

Chunk strategically, not arbitrarily. Most chunking implementations slice documents at fixed token counts. This is lazy and it shows. Chunk at semantic boundaries - section breaks, topic shifts, complete thoughts. Your retrieval accuracy will thank you.

Build retrieval-first architectures. Instead of stuffing everything into context and hoping the model finds it, use targeted retrieval to pull only what’s relevant. A smaller, focused context beats a large, noisy one every time.

Implement position-aware prompting. If you must use long contexts, put the most critical information at the very beginning and very end. Structure your prompts so the model encounters key facts before it gets lost in the middle.

Verify, don’t trust. Any claim the model makes about content from a long context should be verified. We’ve built verification layers that cross-check model responses against the source material. It catches a shocking number of confident errors.

The Guardian Approach

We’ve baked these learnings into Guardian, our AI reliability monitoring platform. It tracks attention patterns in production, flags when models are operating outside their reliable context range, and catches confident confabulation before it reaches users.

flowchart TD
    A[Input Document] --> B[Context Analysis]
    B --> C[Length Check]
    B --> D[Attention Mapping]

    C --> E{Within Safe Range?}
    D --> F[Position Analysis]

    E -->|Yes| G[Process Normally]
    E -->|No| H[Chunk & Retrieve]

    F --> I[Reliability Score]

    G --> I
    H --> I

    I --> J{Score > Threshold?}
    J -->|Pass| K[Serve Response]
    J -->|Warning| L[Flag for Review]
    J -->|Fail| M[Block & Retry]

    style K fill:#22c55e,stroke:#16a34a,color:#000
    style L fill:#eab308,stroke:#ca8a04,color:#000
    style M fill:#ef4444,stroke:#dc2626,color:#fff

The key insight: context window limits aren’t bugs to be fixed with bigger numbers. They’re fundamental constraints that require architectural solutions.

The Honest Conversation We Need

I’m not saying long context windows are useless. They’re genuinely useful for certain tasks - code review, document summarization where you need the gist rather than precise details, creative writing that benefits from broad context.

But for enterprise applications where accuracy matters, where a single missed clause or misattributed fact can cost millions, we need to stop pretending that “supports 1M tokens” means “reliably processes 1M tokens.”

The foundation model companies have commercial incentives to market impressive numbers. The evaluation benchmarks they publish are designed to show their models in the best light. Production reality is messier.

We need more honest conversations about what these systems can and can’t do. Not to discourage adoption - AI is genuinely transformative - but to ensure adoption happens in ways that actually work.

The context window is a specification, not a promise. Build accordingly.

Want to test your own long-context reliability? We’ve open-sourced our evaluation framework. Check it out on GitHub or reach out to discuss how Guardian can help monitor your production AI systems.