A general-purpose large language model knows a great deal about the world and nothing about your business. It has never read your pricing policy, your product documentation, or last quarter's contracts. Ask it about them and it will either refuse or, worse, produce a confident answer that sounds right and is not.
There are two main ways to fix that: retrieval-augmented generation (RAG) and fine-tuning. They are often presented as alternatives, but they change different things, and most production systems that need both use both.
What is retrieval-augmented generation (RAG)?
RAG gives the model the right information at the moment it answers. The model itself does not change.
A RAG system has two halves.
Indexing, done ahead of time. Your documents are cleaned, split into passages ("chunks"), and converted into embeddings: lists of numbers that capture what each passage means. The embeddings are stored in a vector database such as Pinecone, Chroma, or Postgres with pgvector.
Answering, done per question. When a question arrives, it is embedded the same way, and the database returns the passages whose meaning is closest. Those passages are placed in the model's prompt with an instruction: answer from this context, cite it, and say so if the answer is not there.
The result is an assistant that answers from your current documents, can point to the exact source of each claim, and stays up to date as soon as you update the documents, with no retraining.
What is fine-tuning?
Fine-tuning changes the model itself. You train an existing model further on examples of the inputs and outputs you want, and it adjusts its behaviour to match.
Fine-tuning is good at teaching behaviour: a consistent output format, a house style or tone, a specialised classification task, or domain phrasing the base model handles poorly. It is a weak way to teach facts. Knowledge baked in by fine-tuning cannot be cited, goes stale the moment your information changes, and can only be updated by training again.
RAG vs. fine-tuning at a glance
| RAG | Fine-tuning | |
|---|---|---|
| Changes | What the model knows when it answers | How the model behaves and writes |
| Keeping current | Update the documents | Retrain the model |
| Citations | Can cite the source passage | Cannot point to a source |
| Best for | Knowledge that changes; answers you must trace | Format, tone, specialised tasks |
Which should you use?
For most business use cases, start with RAG. Internal knowledge assistants, customer support answers, document Q&A for legal or finance teams, and product copilots all depend on specific, changing facts that must be traceable. RAG is built for that.
Add fine-tuning when behaviour is the problem, not knowledge. If the model answers from the right passages but in the wrong format, misreads your domain's terminology, or needs to perform a narrow task very consistently, fine-tuning on good examples closes the gap.
Many mature systems combine them: a fine-tuned model that follows your format and terminology, answering from passages retrieved by RAG.
What makes a RAG system work in production
A demo RAG system can be built in an afternoon. A reliable one takes engineering in five places.
- Chunking. Split documents where the meaning splits, keep headings and metadata with each chunk, and avoid cutting tables and lists in half. Bad chunking is the most common reason the right answer never reaches the model.
- Retrieval quality. Test retrieval on its own, against real questions, before judging the answers. Combining semantic search with keyword search helps with product codes, names, and exact terms that embeddings handle poorly.
- Grounded generation. Instruct the model to answer only from the retrieved context, cite it, and say plainly when the context does not contain the answer.
- Evaluation. Build a set of real questions with known good answers and measure retrieval hit rate, answer accuracy, and how often the model invents things. Run it before launch and after every significant change.
- Access control and freshness. Mirror your document permissions so people only retrieve what they are allowed to see, and re-index automatically when documents change.
Common mistakes
- Fine-tuning to add knowledge. It is expensive, cannot cite sources, and goes stale. Use retrieval.
- Judging the system by the model alone. Most bad answers are retrieval failures: the right passage was never found.
- Skipping evaluation. Without a test set, every change is a guess.
- Ignoring languages. If your documents or users are multilingual, choose embedding models and test sets that cover every language you need. We build pipelines that include Arabic and Urdu for exactly this reason.
Getting started
List the questions your team or customers ask most, gather the documents that answer them, and check how current and well organised those documents are. That tells you most of what you need to know about scope.
Our LLM & RAG development page describes how we build these systems end to end, and our chatbots and AI assistants page shows where they most often end up: in front of customers and staff, answering with sources.