Building RAG Systems That Actually Work in Production

Every "RAG tutorial" online follows the same pattern: take some documents, split them into chunks, embed them, store in a vector database, query with an LLM. It works on the demo dataset. Then you try it with real enterprise documents and the accuracy drops to 60%.

We built a RAG system for a legal services firm that searches through 50,000+ documents — contracts, case files, regulatory filings, internal memos — with 94% retrieval accuracy. Getting from 60% tutorial accuracy to 94% production accuracy required rethinking every assumption in the standard RAG architecture.

Here is what we learned.

Why Naive Chunking Destroys Retrieval Quality

The standard approach: split documents into 500-token chunks with 50-token overlap. This is fine for blog posts. It is catastrophic for structured documents.

Consider a legal contract. Section 4.2(a) references definitions from Section 1.3, exceptions listed in Exhibit B, and amendments documented in a rider attached three years later. A naive chunker splits these into isolated fragments that lose all relational context.

Our chunking strategy is document-type-aware:

TYPESCRIPT
interface ChunkingStrategy {
  type: 'fixed' | 'semantic' | 'structural'
  config: {
    maxTokens: number
    overlap: number
    preserveHeadings: boolean
    includeMetadata: boolean
  }
}

const STRATEGIES: Record<string, ChunkingStrategy> = {
  contract: {
    type: 'structural',
    config: {
      maxTokens: 1024,
      overlap: 0, // No overlap - sections are self-contained
      preserveHeadings: true,  // Keep "Section 4.2(a)" with its content
      includeMetadata: true    // Attach document title, date, parties
    }
  },
  email: {
    type: 'semantic',
    config: {
      maxTokens: 512,
      overlap: 100,
      preserveHeadings: false,
      includeMetadata: true  // Thread context matters
    }
  },
  report: {
    type: 'structural',
    config: {
      maxTokens: 768,
      overlap: 50,
      preserveHeadings: true,
      includeMetadata: true
    }
  }
}

function chunkDocument(
  content: string,
  documentType: string
): Chunk[] {
  const strategy = STRATEGIES[documentType] ?? STRATEGIES.report

  if (strategy.type === 'structural') {
    return structuralChunk(content, strategy.config)
  }
  return semanticChunk(content, strategy.config)
}

function structuralChunk(
  content: string,
  config: ChunkingStrategy['config']
): Chunk[] {
  // Split on heading patterns (##, Section X, Article X)
  const sections = content.split(
    /(?=^(?:#{1,3}|Section \d|Article \d|ARTICLE \d))/m
  )

  return sections.map((section, index) => {
    // Prepend parent heading chain for context
    const headingChain = extractHeadingChain(content, index)

    return {
      content: headingChain
        ? `${headingChain}\n\n${section.trim()}`
        : section.trim(),
      metadata: {
        sectionIndex: index,
        headings: headingChain,
        tokenCount: estimateTokens(section)
      }
    }
  })
}

The critical insight: every chunk should be independently understandable. If a chunk says "as described above" with no context about what "above" refers to, the LLM will hallucinate an answer.

We prepend the heading chain to every chunk. So a chunk from Section 4.2(a) starts with "Agreement > Article 4: Payment Terms > Section 4.2: Late Payment Penalties > (a) Grace Period" before the actual content. This costs a few extra tokens per chunk but dramatically improves retrieval relevance.

The Hybrid Search Architecture

Pure vector similarity search has a fundamental limitation: it finds semantically similar content, not necessarily relevant content. If a user asks "What is the termination clause?" and your documents use the phrase "exit provisions," pure semantic search might rank it lower than a document that happens to discuss employee terminations.

We use hybrid search: vector similarity combined with keyword (BM25) matching:

TYPESCRIPT
async function hybridSearch(
  query: string,
  options: SearchOptions = {}
): Promise<SearchResult[]> {
  const { topK = 10, alpha = 0.7 } = options

  // Run both searches in parallel
  const [vectorResults, keywordResults] = await Promise.all([
    vectorSearch(query, topK * 2),
    keywordSearch(query, topK * 2)
  ])

  // Reciprocal Rank Fusion
  const fusedScores = new Map<string, number>()
  const k = 60 // RRF constant

  vectorResults.forEach((result, rank) => {
    const score = alpha * (1 / (k + rank))
    fusedScores.set(
      result.id,
      (fusedScores.get(result.id) ?? 0) + score
    )
  })

  keywordResults.forEach((result, rank) => {
    const score = (1 - alpha) * (1 / (k + rank))
    fusedScores.set(
      result.id,
      (fusedScores.get(result.id) ?? 0) + score
    )
  })

  // Sort by fused score and return top K
  return Array.from(fusedScores.entries())
    .sort((a, b) => b[1] - a[1])
    .slice(0, topK)
    .map(([id, score]) => ({ id, score }))
}

The alpha parameter controls the balance between semantic and keyword search. For technical documentation where precise terminology matters, we set alpha to 0.5. For conversational search interfaces, we push it to 0.8 (favoring semantic).

Source Citations: The Non-Negotiable Feature

Every RAG system we build surfaces source citations. Not as a nice-to-have — as a core feature. Without citations, users cannot verify answers, and trust erodes within weeks.

TYPESCRIPT
const GENERATION_PROMPT = `Answer the user's question based ONLY on the provided context.

Rules:
1. If the context does not contain the answer, say "I could not find this information in the available documents."
2. NEVER make up information not present in the context.
3. Cite your sources using [Source: document_name, section] format.
4. If multiple sources agree, cite all of them.
5. If sources conflict, present both perspectives with their citations.

Context:
{context}

Question: {query}`

That first rule is the most important. An LLM that says "I don't know" is infinitely more trustworthy than one that confidently generates plausible-sounding nonsense. In production, our system returns "could not find" for about 8% of queries. Every one of those is a trust-building moment.

Evaluation: How We Measure and Improve Accuracy

You cannot improve what you do not measure. We maintain a test set of 200+ question-answer pairs generated from known documents. Every change to the chunking strategy, embedding model, or retrieval algorithm gets evaluated against this test set before deployment:

TYPESCRIPT
interface EvalResult {
  query: string
  expectedDocIds: string[]
  retrievedDocIds: string[]
  precision: number   // relevant retrieved / total retrieved
  recall: number      // relevant retrieved / total relevant
  mrr: number         // Mean Reciprocal Rank
}

async function evaluateRetrieval(
  testSet: TestCase[]
): Promise<EvalSummary> {
  const results: EvalResult[] = []

  for (const testCase of testSet) {
    const retrieved = await hybridSearch(testCase.query, { topK: 5 })
    const retrievedIds = retrieved.map(r => r.id)

    const relevant = retrievedIds.filter(
      id => testCase.expectedDocIds.includes(id)
    )

    const firstRelevantRank = retrievedIds.findIndex(
      id => testCase.expectedDocIds.includes(id)
    )

    results.push({
      query: testCase.query,
      expectedDocIds: testCase.expectedDocIds,
      retrievedDocIds: retrievedIds,
      precision: relevant.length / retrievedIds.length,
      recall: relevant.length / testCase.expectedDocIds.length,
      mrr: firstRelevantRank >= 0 ? 1 / (firstRelevantRank + 1) : 0
    })
  }

  return {
    avgPrecision: avg(results.map(r => r.precision)),
    avgRecall: avg(results.map(r => r.recall)),
    avgMRR: avg(results.map(r => r.mrr)),
    totalQueries: testSet.length,
    failedQueries: results.filter(r => r.recall === 0)
  }
}

When we built the legal document search, our initial naive implementation scored 62% recall. After switching to structural chunking: 78%. After adding hybrid search: 87%. After metadata enrichment and heading chains: 94%.

Each improvement was measurable because we had the evaluation framework in place from day one. Without it, you are just guessing.

The Architecture That Scales

Our production RAG systems follow a clear separation of concerns:

1. Ingestion pipeline (n8n workflow): watches for new documents, classifies type, chunks, embeds, stores
2. Vector store (PostgreSQL with pgvector): keeps embeddings close to the relational data they reference
3. Search layer (hybrid retrieval): combines vector and keyword search with reranking
4. Generation layer (OpenRouter): model-agnostic response generation with citation extraction
5. Evaluation service (scheduled): runs test suite weekly, alerts on accuracy regression

We chose pgvector over dedicated vector databases like Pinecone for a simple reason: it eliminates an entire infrastructure dependency. Your vectors live in the same PostgreSQL instance as your application data. One backup strategy, one connection pool, one less thing to monitor at 3 AM.

For organizations under 10 million vectors, pgvector performs comparably to dedicated solutions. Beyond that scale, a dedicated vector database starts to make sense — but we have yet to encounter a client who needed it.

The uncomfortable truth about enterprise RAG: the technology is the easy part. The hard part is convincing domain experts to curate the test set, establishing feedback loops with end users, and building organizational trust in AI-assisted search. The teams that invest in the human process alongside the technical architecture are the ones who get to 94% accuracy. The ones who only focus on the technology plateau at 75%.