Why RAG is a data systems problem
Retrieval-augmented generation brings external information into a model's answer. In production, the challenging questions are often not about which vector database to pick: Are source documents complete? Has access changed? Can outdated records still appear? Can you explain why a result was retrieved?
1. Build a trustworthy ingestion pipeline
Record source identity, owner, content type, timestamps and source permissions. Handle retries and duplicate events idempotently. Capture parsing failures and make it possible to replay or backfill records without creating duplicate content.
2. Preserve meaningful retrieval units
Chunking should respect sections, tables, identifiers and context boundaries. Overly small chunks lose meaning; overly large chunks can reduce search precision and inflate downstream costs. Keep a traceable link from each chunk to its source document and version.
3. Apply authorization before content is exposed
Retrieval must not bypass document permissions. Index access metadata where appropriate, enforce authorization at query time, and ensure caches and derived stores respect revocation. Test shared and restricted documents across different user roles.
4. Define freshness and deletion behavior
Decide how quickly a source change must propagate to the index, and monitor that delay. Handle document deletions, permission revocations and legal retention requirements; stale embeddings are still data copies. A complete backfill strategy is as important as incremental updates.
5. Separate retrieval evaluation from response evaluation
Build representative search questions and expected relevant documents. Measure retrieval recall and ranking usefulness, then examine how the application uses retrieved context. A model cannot reliably cite information it never received. Add permission-denied and no-answer scenarios to the evaluation set.
| Control | What to measure | Failure example |
|---|---|---|
| Completeness | Ingestion success, parser exceptions, source coverage | Uploaded PDF silently missing sections |
| Freshness | Source-to-index lag, stale-record count | Policy changed yesterday, answer uses old version |
| Authorization | Permission-filter tests, revocation latency | Private file retrieved for unauthorized user |
| Retrieval relevance | Recall/precision on representative questions | Relevant policy not in top results |
| Provenance | Source/version linked per retrieved item | Generated answer cannot trace its evidence |
Who should own the work?
Data engineers can own ingestion, quality, provenance and source controls. Applied AI engineers can own the retrieval strategy, answer evaluation and model interaction. Both should share documented contracts and release thresholds.