RAG, short for retrieval-augmented generation, is a way of building an AI assistant that first looks up relevant passages in your own documents and then writes an answer from what it found, usually showing the source. Because the answer comes from your current content rather than the model's memory, a person can check it and you can keep it up to date.
This guide explains how RAG works in plain terms, where it fits, why projects fail, and what a first build involves. If you are weighing RAG against training a model on your data, read RAG vs fine-tuning as well.
How does RAG work?
Every time someone asks a question, a RAG system does four things:
- It finds the right passages. The question is matched against an index of your documents, using meaning as well as keywords.
- It ranks the results. The best passages go first, because the model can only use what it is given.
- It writes the answer. A language model composes a reply using only those passages.
- It shows its sources. The answer links back to the documents, and says "not found" when the passages do not contain the answer.
Before any of that happens, an ingestion step loads your files, such as PDFs, Word documents, web pages, wikis and tickets, splits them into sections that keep their meaning, and stores them in a searchable index.
What is RAG good for?
- Staff questions. Policies, manuals, wikis and past projects, answered with a link to the source.
- Customer support. Answers from your help centre and product data, with a handover to a person when the assistant cannot answer.
- Document search and Q&A. Questions across contracts, reports and technical documents instead of searching them one by one.
- Multilingual assistants. The same knowledge for English, French, Japanese or Arabic users, provided it is tested on real content from each language.
RAG is the wrong tool when the answer needs a calculation across live data, such as "what were our sales last quarter". That needs a reporting tool, or an AI agent that queries your database.
Why do RAG projects fail?
Most failures are in retrieval, not in the language model. The same model gives a poor answer when it is handed the wrong passages and a good one when it is handed the right ones.
- Missing or outdated content. RAG cannot answer from a document that does not exist. If a policy changed and the old PDF is still indexed, the assistant will quote the old policy.
- Poor chunking. Splitting a document in the wrong places separates a rule from its exceptions.
- One kind of search only. Keyword search and meaning-based search each miss things the other finds, so hybrid search with re-ranking usually works better.
- No access control. People should only get answers from documents they are allowed to see.
- No testing. Without a set of real questions with known answers, nobody can tell whether a change made things better or worse.
How do you know a RAG assistant is accurate?
Build a test set before launch: real questions from your staff or customers, each with the correct answer and the document it should come from. Score two things separately: did retrieval find the right passage, and did the answer use it correctly. Then re-run the set after every change to the content, the index or the model.
This is the step most often skipped, and it is what makes an assistant trustworthy. At Xcrino we build test sets from real questions, check accuracy and citations before launch, and monitor quality afterwards.
What does a first RAG build cost, and how long does it take?
At Xcrino, RAG projects start with a fixed-price AI proof of concept from $3,000 (about ₹2.7 lakh). It takes 2–4 weeks and shows how accurate the assistant is on your real documents. If the results are good, a first production release typically takes 1–2 months and starts from $5,000 (about ₹4.5 lakh). The final cost depends on the number and format of your sources, access rules, languages and integrations.
Before the build starts, our founder and AI lead, Dr Sunil Gupta, reviews whether your data can support the use case. You can read more about our RAG development services.
RAG is the knowledge part of an assistant; an AI agent adds the action part. Our guide to AI agent vs chatbot shows how chatbots, RAG and agents fit together.
Where did RAG come from?
The term comes from a 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and colleagues. They combined a pre-trained language model with a searchable index of documents, and showed that the index can be swapped to update what the system knows as the world changes. That idea, keeping knowledge in an index you control and not inside the model, is why RAG suits company content that changes.
What are the main parts of a RAG system?
- Ingestion: loading PDFs, Word files, web pages, wikis and tickets, with text recognition for scans.
- Chunking: splitting content into sections that keep their meaning, so a rule stays with its exceptions.
- Indexing: storing the sections in a search index, often combining keyword and meaning-based search.
- Retrieval and ranking: finding the best passages for a question and putting the strongest first.
- Generation: the language model writes the answer from those passages and cites them.
- Access control: users only get answers from documents they are allowed to see.
- Evaluation: a test set of real questions that scores retrieval and answers.
The diagram below shows the flow: preparation happens once and whenever documents change, and the answer path runs for every question.
How do you measure RAG quality?
Measure retrieval and answers separately, because a wrong answer can come from either. Useful measures are:
- Retrieval recall: did the right passage come back at all?
- Context relevance: how much of what was retrieved was useful?
- Groundedness: is every claim in the answer supported by the retrieved passages?
- Answer correctness: does the answer match the known right answer?
- Citation accuracy: do the cited sources really contain the claim?
Score them on a fixed set of real questions before launch and after every change to the content, the index or the model.
Which teams use RAG assistants?
- Customer support: answers from the help centre and product data, with a handover to a person.
- HR and operations: policies, procedures and onboarding questions for staff.
- Sales and pre-sales: product facts, past proposals and security answers for RFPs.
- Legal and compliance: questions across contracts and policy documents, with sources shown.
- Engineering: answers across internal wikis, runbooks and past incidents.
How is RAG different from search, a chatbot or fine-tuning?
- Search returns a list of documents; RAG reads the best passages and writes a direct answer.
- A plain chatbot answers from what the model learned in training; RAG answers from your current content.
- Fine-tuning changes how a model behaves; RAG changes what it can look up. See RAG vs fine-tuning for how to choose.
If you want an assistant that can also take actions, see AI agent vs chatbot and how to build an AI agent.
RAG is one building block of an AI agent. See AI agents for business: the complete guide for how the pieces fit together.
Questions we are often asked
Is RAG the same as ChatGPT?
No. ChatGPT is a general-purpose chatbot that answers from what the model learned in training. A RAG assistant answers from your own content, shows the source of each answer and can respect who is allowed to see what.
Do we need to train a model on our data?
Usually not. RAG gives the model your content at the moment of the question, so nothing has to be retrained when a document changes. Fine-tuning changes how a model writes, not what it knows.
Can RAG work with scanned documents?
Yes, using OCR (text recognition) when the files are loaded. The quality depends on the scans, so we test it on a sample first.
Will our data stay private?
That depends on how the system is deployed, so we agree where data is stored and processed before the build starts. We sign an NDA whenever you ask, and source code and assets transfer to you on full payment.