Most business AI questions are not "write me a poem." They're "what's the renewal clause in the Henderson contract" and "which invoices from Q2 are still unpaid." A general-purpose model cannot answer either one, because it has never seen your contract or your invoice ledger. Ask anyway and you get a confident, plausible, wrong answer.
A RAG pipeline fixes that. It's the architecture behind every "chat with your documents" tool you've used, and it's the default way businesses put AI on top of their own knowledge. I built one from scratch — Nexus, open on GitHub — specifically so I could see where the theory breaks in practice. This is what each stage does, where it fails, and how to tell whether you need one.
What Is a RAG Pipeline in AI?
Retrieval-Augmented Generation was introduced in a 2020 Facebook AI research paper as a way to combine a language model with a searchable memory. The idea is simple: don't ask the model to recall a fact, hand it the fact and ask it to explain.
In practice a RAG pipeline is a search engine bolted to the front of an LLM. When a question comes in, the system searches your document store, pulls the handful of passages most likely to contain the answer, and builds a prompt that says, roughly: here are five paragraphs from the customer's files — answer using only these.
The model's job shrinks from "know everything" to "read this and summarise." That's a far easier job, and a far more reliable one.
RAG vs. Fine-Tuning vs. Plain Prompting
The three approaches solve different problems, and picking the wrong one is the most common expensive mistake I see.
| Approach | What it changes | Update cost | Best for | Sources attached? |
|---|---|---|---|---|
| Plain prompting | Nothing | None | General questions, drafting | No |
| RAG pipeline | What the model sees | Upload a file | Answering from your documents | Yes |
| Fine-tuning | The model's weights | Hours to days, per update | Enforcing tone, format, domain style | No |
| Long-context prompt | What the model sees | Paste the whole corpus | Small, stable corpora (under 500 pages) | Partially |
Fine-tuning teaches a model how to answer. RAG teaches it what to answer from. If your requirement contains the word "our" — our policies, our contracts, our product catalogue — you need RAG, not fine-tuning.
The Five Stages of a RAG Pipeline
Every RAG system, from a weekend project to an enterprise deployment, runs the same five stages.
1. Ingest. Extract raw text from the source file. This is duller and harder than it sounds — a PDF, a Word doc with tables, an Excel sheet with 40 tabs, and a PowerPoint deck each need a different parser. In Nexus I use pypdf, python-docx, openpyxl, and python-pptx respectively, because no single library handles all four well.
2. Chunk. Split the text into overlapping pieces. I use roughly 1,000 characters with 200 characters of overlap. The overlap matters: without it, a sentence that straddles a boundary gets cut in half and neither fragment answers the question.
3. Embed. Convert each chunk into a vector — a long list of numbers representing meaning, not keywords. Chunks about "termination notice period" and "how to end the agreement early" land near each other in vector space even with zero shared words. Nexus stores these in Supabase's pgvector.
4. Retrieve. Embed the incoming question with the same model, then find the nearest chunks by cosine similarity. Nexus takes the top 5. Anthropic's testing found that retrieving the top 20 chunks outperformed top-5 and top-10 across domains — more context beats tighter filtering, up to a point.
5. Generate. Build a prompt containing the question plus the retrieved chunks, with an instruction to answer only from that text and to say "not in the document" otherwise. That last instruction is the difference between a grounded system and a confident liar.
Where RAG Pipelines Actually Break
Nothing on that list fails in a demo. All of it fails in production. Four failure modes account for nearly everything I've had to debug:
- Bad chunking. A table split across two chunks becomes two piles of meaningless numbers. Chunk size, boundary, and overlap have more effect on answer quality than which LLM you pick.
- Retrieval misses. If the right passage isn't in the top-k, the model cannot answer no matter how good it is. Anthropic's benchmarks show a 5.7% baseline retrieval failure rate, dropping to 3.7% with contextual embeddings, 2.9% adding contextual BM25, and 1.9% with reranking — a 67% reduction from the same corpus and the same model.
- No confidence floor. Vector search always returns something, even for an unrelated question. Nexus sets a similarity threshold of 0.5: below it, the query is routed to a general chat model with no document context instead of forcing an answer out of irrelevant chunks.
- Provider failure. Rate limits and outages are not edge cases. Nexus falls back to a self-hosted model on an HTTP 429, reusing the same retrieved chunks so the answer stays grounded rather than degrading to a guess.
That third point is the one most tutorials skip, and it's the one that determines whether users trust the system after week two.
Does RAG Stop Hallucinations?
It reduces them a lot. It does not stop them.
Stanford HAI's 2026 AI Index Report found hallucination rates ranging from 22% to 94% across 26 leading models on a benchmark testing belief attribution — the same models that look flawless on standard evaluations. Grounding the answer in retrieved text removes most of the need to invent, but the model can still misread a passage or blend two chunks together.
The practical mitigation is transparency, not a better model. Nexus renders the retrieved source chunks under every answer in an expandable panel, so the person reading it can check the claim against the actual paragraph in about three seconds. That single UI decision does more for trust than any accuracy benchmark.
When You Shouldn't Build a RAG Pipeline
Anthropic's own guidance is refreshingly blunt: if your knowledge base is under roughly 200,000 tokens — about 500 pages — skip retrieval entirely and put the whole corpus in the prompt with caching enabled. No vector database, no chunking strategy, no embedding costs.
RAG earns its complexity when at least one of these is true: the corpus is too big for a context window, it changes frequently, different users are allowed to see different documents, or you need to cite which specific document an answer came from. If none apply, you're building infrastructure to solve a problem you don't have.
What This Costs to Run
The RAG market is projected to grow from $1.94B in 2025 to $9.86B by 2030, which tells you where the vendor pricing is heading — but a working system for a small business is not a six-figure line item.
| Cost component | Typical range | Notes |
|---|---|---|
| Document embedding | Cents per 100 pages | One-time, per document |
| Vector database | $0–$25/month | Supabase/pgvector free tier covers small corpora |
| LLM per question | $0.001–$0.03 | Depends on model and chunk count |
| Build | $4,000–$15,000 | The real cost; scales with format complexity |
The infrastructure is cheap. The engineering — parsing your actual messy files, tuning chunking, setting thresholds, handling failure — is what you're paying for.
How Smart AI Workspace Approaches This
I built Nexus in the open partly so clients can see the architecture before they commission one. It's model-agnostic by design: chat and embedding models route through OpenRouter, so the same pipeline can run on a free tier during testing and switch to Claude or GPT for production with a single config change. No rewrite, no lock-in.
That's how I scope client work too. Infrastructure goes in your name — your API keys, your Supabase project — and the model choice stays a config value rather than an architectural commitment, because the frontier model that's best today won't be in eighteen months. If you're deciding between a document chatbot and something that takes action on what it finds, the difference is covered in specialized agents vs. chatbots, and the wiring behind agentic systems is in how I build agentic workflows with Claude Code.
RAG Pipelines: FAQ
What is a RAG pipeline in AI?
A RAG pipeline searches your own documents for the passages most relevant to a question, then hands only those passages to a language model and asks it to answer from that text alone. Five stages: ingest, chunk, embed, retrieve, generate. The output is an answer grounded in your files with the sources attached.
How is RAG different from fine-tuning?
Fine-tuning alters the model's weights to change how it responds and must be redone whenever your knowledge changes. RAG alters what the model sees at question time, so updating knowledge means uploading a file. Use fine-tuning for style and format; use RAG for facts.
What is chunking and why does chunk size matter?
Chunking splits a document into pieces small enough to embed and retrieve individually. Chunk size, boundaries, and overlap determine whether a complete idea survives the split. Around 1,000 characters with 200 characters of overlap is a solid default; tables and structured data usually need custom handling.
Which vector database should I use for RAG?
For most small and mid-sized deployments, PostgreSQL with the pgvector extension — via Supabase or any managed Postgres — is enough, and it keeps your vectors next to your application data. Dedicated vector databases like Pinecone or Weaviate make sense at millions of chunks or with heavy filtering requirements.
Can a RAG chatbot work with Excel and PowerPoint files, not just PDFs?
Yes, but each format needs its own extraction path. Nexus handles PDF, DOCX, XLSX, and PPTX with four separate parsers, reading Excel sheets row by row and PowerPoint slides including table content. Format coverage is usually the largest hidden cost in a document chatbot build.
Want AI That Answers From Your Documents?
If your team keeps re-reading the same contracts, policies, or spec sheets to answer questions, that's a RAG problem with a measurable payback. Contact me for a free audit — I'll tell you whether retrieval is worth building or whether a simpler approach covers it. See the full scope of what I build, current pricing, or check verified work history on my Upwork profile.
Sources: Anthropic — Introducing Contextual Retrieval · Stanford HAI — 2026 AI Index Report · Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · MarketsandMarkets — Retrieval-Augmented Generation Market · AWS — What is RAG? · Nexus RAG Chatbot — source on GitHub
