Bill Tanker
Crazy Unicorns
Retrieval-Augmented Generation (RAG) has become the default pattern for grounding LLM outputs in enterprise data. The concept is straightforward: retrieve relevant documents, inject them into the prompt, and let the model generate an answer. But the gap between a working prototype and a reliable production system is significant. After deploying RAG pipelines for multiple enterprise clients, here are seven lessons we've learned the hard way.
Most teams spend weeks evaluating embedding models while using naive fixed-size chunking. In our experience, the chunking strategy has a larger impact on retrieval quality than the difference between top-tier embedding models. Semantic chunking — splitting documents at natural boundaries like paragraphs, sections, or topic shifts — consistently outperforms fixed-size approaches. We typically use a hybrid: semantic boundaries with a maximum token limit as a safety net.
One pattern that works well for technical documentation is hierarchical chunking. We create chunks at multiple granularity levels (section, paragraph, sentence) and store parent-child relationships. At retrieval time, we fetch fine-grained chunks for precision but expand to parent chunks for context. This gives the LLM enough surrounding information to generate coherent answers.
The single biggest mistake teams make is treating evaluation as an afterthought. Without a systematic evaluation framework, you're flying blind. Every change to chunking, retrieval, or prompting could improve one query type while degrading another. We build evaluation into the pipeline from the start using a three-layer approach: retrieval quality (are we fetching the right documents?), generation quality (is the answer correct and grounded?), and end-to-end metrics (does the user get what they need?).
For retrieval evaluation, we maintain a golden dataset of query-document pairs and measure recall@k and MRR. For generation, we use LLM-as-judge with structured rubrics for faithfulness, relevance, and completeness. The key insight: automate this into your CI/CD pipeline so every change is evaluated before deployment.
Pure vector search works well for semantic similarity, but enterprise queries often include specific identifiers — product codes, error numbers, policy references — where exact matching is critical. We've found that hybrid search combining BM25 (keyword) and vector similarity with reciprocal rank fusion consistently outperforms either approach alone. The improvement is especially pronounced for technical and regulatory content.
When a user asks about Q4 2025 financial results, you don't want the system retrieving Q4 2023 data just because the language is similar. Structured metadata filters — date ranges, document types, departments, access levels — are the most effective way to improve precision. We extract and index metadata during ingestion and expose it as pre-filters in the retrieval pipeline. This approach is computationally cheaper and more predictable than trying to encode temporal or categorical information into embeddings.
In production RAG systems, the prompt template is a critical piece of infrastructure. It needs versioning, testing, and monitoring just like any other component. We maintain prompt templates as code with semantic versioning. Each template includes explicit instructions for handling edge cases: what to do when retrieved documents are contradictory, how to express uncertainty, and when to decline answering. We also include few-shot examples of good and bad answers to calibrate the model's behavior.
When a RAG system gives a wrong answer, the root cause is usually in retrieval, not generation. But if you only monitor the final output, you can't distinguish between 'we retrieved the wrong documents' and 'we retrieved the right documents but the model hallucinated.' We instrument both stages independently: retrieval latency and relevance scores, plus generation faithfulness and citation accuracy. This separation makes debugging dramatically faster.
Enterprise knowledge bases are living systems. Documents get updated, deprecated, and replaced. A RAG system that doesn't handle freshness will gradually degrade as its index drifts from reality. We implement incremental indexing with change detection, version tracking for documents, and TTL-based cache invalidation. For critical use cases, we add a freshness indicator to the generated response so users know when the underlying data was last updated.
Building a RAG system that works in production requires treating it as a full engineering system — not just a prompt wrapper around a vector database. If you're planning a RAG deployment and want to avoid common pitfalls, book a strategy call with our team.
We build production-ready AI systems. Book a strategy call to discuss your requirements.
Hello! How can I help?
