Data RAG

The Real Reason Your RAG App Hallucinates (It's Not Chunking)

Everyone's optimizing chunk size and embedding models. The problem is upstream. Your data pipeline strips context before it ever reaches the vector store.

Rotavision Team · January 9, 2026 · 10 min read

Your RAG application hallucinates. You've tried everything:

Smaller chunks (didn't help)
Larger chunks (made it worse)
Overlapping chunks (marginal improvement)
Better embedding models (expensive, minimal gain)
Hybrid search (added complexity, same hallucinations)
Reranking (helped a bit, still hallucinates)

You're optimizing the wrong thing.

The problem isn't retrieval. The problem is what you're retrieving. Your data pipeline strips context before it ever reaches the vector store — and no amount of retrieval optimization can recover what was never there.

The Hallucination Isn't Random

When a RAG app hallucinates, it's not making things up from nothing. It's filling gaps.

The LLM receives retrieved context that's almost sufficient to answer the question. But something's missing — a definition, a relationship, a qualifier, a timestamp. So the model fills the gap with plausible-sounding information.

The hallucination feels random. But it's actually predictable: it happens wherever your context has holes.

And those holes were created long before retrieval. They were created when you processed the data.

Where Context Dies

Let's trace a typical document's journey to your vector store.

Original document:

Q3 2025 Financial Report - CONFIDENTIAL
Prepared by: Finance Team
Last Updated: October 15, 2025

Revenue grew 12% YoY to $45.2M, exceeding guidance of $42M.
This growth was primarily driven by the APAC region (up 23%)
following the Singapore expansion announced in Q1.

Note: APAC figures exclude the divested Malaysia subsidiary
(see Appendix B for pro-forma comparisons).

After your ETL pipeline:

Revenue grew 12% YoY to $45.2M, exceeding guidance of $42M.
This growth was primarily driven by the APAC region (up 23%)
following the Singapore expansion announced in Q1.

What's lost?

It's a Q3 2025 report (temporal context)
It's confidential (access context)
APAC excludes Malaysia (scope context)
There's an Appendix B with more detail (relational context)

After chunking:

Chunk 1: "Revenue grew 12% YoY to $45.2M, exceeding guidance of $42M."
Chunk 2: "This growth was primarily driven by the APAC region (up 23%) following the Singapore expansion announced in Q1."

Now the chunks don't even know they're related to each other.

User asks: "What was APAC revenue growth in Q3?"

RAG retrieves: "APAC region (up 23%)"

Model answers: "APAC revenue grew 23% in Q3."

Reality: APAC grew 23%, but that excludes Malaysia due to a divestiture. The real comparable number is in Appendix B. But that context was stripped long ago.

This isn't a retrieval problem. It's a data problem.

The Context Graveyard

I've audited dozens of RAG implementations. The same context types die in every pipeline:

Temporal context

When was this written?
What time period does it cover?
Is this still current?

Your chunks don't know if they're from 2020 or 2025. The model treats them as equally valid.

Relational context

What else is this connected to?
What's the parent document?
What does "see above" refer to?

Your chunks are orphans. They've lost their relationships.

Scope context

What's included and excluded?
What assumptions apply?
What's the confidence level?

Your chunks state facts without qualifiers. The caveats were in a different paragraph — now a different chunk.

Source context

Who wrote this?
What's their authority?
Is this opinion or fact?

Your chunks are anonymous. A CEO statement and a blog comment look the same.

Semantic context

What do domain terms mean?
What's the implied meaning?
What would a domain expert understand?

Your chunks use terms without definitions. "Revenue" means something different in every company.

Why Traditional ETL Kills Context

Traditional ETL (Extract, Transform, Load) was designed to move structured data between systems. Its goal is clean, consistent, deduplicated records.

Context is messy. It doesn't fit in columns. It's implicit, relational, and often unstructured. So ETL strips it.

Extract: Pull the document. Metadata often lost here.

Transform: Clean the text. Headers, footers, annotations removed. "Noise" eliminated — except that "noise" was context.

Load: Store the clean text. Ready for chunking. Context-free.

By the time your data reaches the vector store, it's been sanitized of the very information that would prevent hallucination.

ETL-C: Context as a First-Class Citizen

The fix isn't better retrieval. It's better data processing.

We call this ETL-C: Extract, Transform, Load, Contextualize.

The key insight: Context isn't an afterthought. It's architectural. You need to preserve, extract, and enrich context throughout the pipeline — not just at the end.

Extract with context:

Capture document metadata (source, date, author, classification)
Preserve document structure (headers, sections, relationships)
Extract explicit context (definitions, assumptions, scope)

Transform with context:

Don't strip "noise" — evaluate what's context
Maintain relationships between sections
Preserve qualifiers and caveats with their claims

Load with context:

Store context alongside content
Create linkages between related chunks
Index context for retrieval

Contextualize:

Enrich with semantic context (what do terms mean here?)
Add temporal context (is this current?)
Link to related documents (what else matters?)

Practical Context Preservation

Here's what this looks like in practice:

Instead of this chunk:

{
  "text": "Revenue grew 12% YoY to $45.2M, exceeding guidance of $42M.",
  "embedding": [0.023, -0.142, ...]
}

Store this:

{
  "text": "Revenue grew 12% YoY to $45.2M, exceeding guidance of $42M.",
  "embedding": [0.023, -0.142, ...],
  "context": {
    "temporal": {
      "document_date": "2025-10-15",
      "period_covered": "Q3 2025",
      "as_of_date": "2025-09-30"
    },
    "source": {
      "document": "Q3 2025 Financial Report",
      "author": "Finance Team",
      "classification": "CONFIDENTIAL",
      "authority": "official"
    },
    "scope": {
      "includes": ["North America", "EMEA", "APAC ex-Malaysia"],
      "excludes": ["Malaysia subsidiary (divested)"],
      "notes": ["See Appendix B for pro-forma comparisons"]
    },
    "relationships": {
      "parent_section": "Financial Highlights",
      "related_chunks": ["chunk_id_456", "chunk_id_789"],
      "references": ["Appendix B"]
    },
    "semantic": {
      "terms": {
        "YoY": "Year over year comparison to Q3 2024",
        "guidance": "Previously announced forecast of $42M"
      }
    }
  }
}

Now when the model retrieves this chunk, it knows:

This is Q3 2025 data, as of September 30
It's from an official finance report
APAC excludes Malaysia
There's more detail in Appendix B

The hallucination opportunity shrinks dramatically.

The Retrieval Changes Too

With context-rich chunks, retrieval becomes smarter:

Context-aware retrieval:

User: "What was APAC revenue growth in Q3?"

1. Retrieve semantically similar chunks
2. Filter by temporal context (Q3 2025)
3. Surface scope context (APAC excludes Malaysia)
4. Include related chunks (Appendix B reference)
5. Pass context to model alongside content

Context-informed generation:

Based on the Q3 2025 Financial Report (as of September 30, 2025):

APAC revenue grew 23% YoY. Note that this figure excludes the divested Malaysia subsidiary. For pro-forma comparisons including the historical Malaysia figures, refer to Appendix B of the report.

No hallucination. Because the model had the context it needed.

The Unsexy Truth

Here's the truth nobody wants to hear: Fixing RAG hallucination is a data engineering problem, not an AI problem.

The sexy solutions are retrieval innovations — better embeddings, smarter reranking, advanced chunking strategies. The unsexy solution is fixing your data pipeline.

Context preservation isn't glamorous. It doesn't demo well. It requires careful work at the ETL layer — the part of the stack most AI teams ignore.

But it's where hallucinations are born. And it's where they need to die.

Where to Start

If your RAG app hallucinates, start here:

Audit your pipeline. Trace a document from source to vector store. What context is lost at each stage?
Categorize the losses. Temporal? Relational? Scope? Source? Semantic? Know what's missing.
Prioritize by hallucination type. Which missing context causes which hallucinations? Fix the biggest sources first.
Enrich at the source. Don't try to recover context after it's lost. Preserve and enrich during processing.
Store context with content. Your vector store needs to hold more than embeddings. Context is data too.
Retrieve context-aware. Use context in retrieval, not just content. Filter, boost, and inform based on context.

Stop optimizing chunk size. Start preserving context.

Your RAG app will thank you.

Want to fix your RAG hallucinations?

Rotavision's Context Engine implements ETL-C at scale — preserving and enriching context so your AI applications have the information they need.

Learn More About ETL-C