For an assistant that answers questions about changing business documents, start by testing retrieval-augmented generation, or RAG. Retrieval supplies relevant source material at request time. Fine-tuning changes model behavior through training examples. They solve different problems and can be combined.
Neither option guarantees a correct answer. The useful decision is which part of your system is failing: finding the evidence, interpreting it, or producing the desired response format.
Separate knowledge from behavior
Imagine a hypothetical internal assistant answering questions about delivery policies. Shipping regions, eligibility rules, and effective dates change. Those facts belong in a maintained source that can be updated and inspected.
The desired behavior is different: answer concisely, cite the policy, ask for missing order details, and hand off ambiguous cases. Start by expressing those rules in the prompt and testing them. Consider fine-tuning only if you can demonstrate a persistent behavioral problem and have suitable examples to train and evaluate it.
OpenAI describes supervised fine-tuning as learning from examples of desired inputs and outputs. Its documentation recommends establishing evaluations first. Read the supervised fine-tuning guide.
A model that writes in your brand voice may still lack the current policy. A retrieval system that finds the policy may still produce an unclear response. Diagnose these separately.
Try direct context before building an index
If the approved material is small enough, supplying it directly can be a useful baseline. This removes retrieval as a source of failure and gives you something to compare against.
For example, test ten questions against a short policy document included with each request. Check whether the model answers from that document, distinguishes exceptions, and declines questions the document does not cover. These are illustrative test cases, not a recommended universal dataset size.
Direct context becomes less attractive as documents grow, permissions differ, and irrelevant material consumes more of the request. Measure latency and usage for your actual inputs before choosing a more complex architecture.
What a RAG system needs beyond a vector database
A practical retrieval pipeline has several decisions: which documents enter the index, how they are split, how a query finds candidates, and how those candidates reach the model.
Preserve metadata such as source URL, document title, revision date, tenant, and access policy. Keep headings or section identifiers with each passage. A paragraph that says “this exception applies” is much less useful if its qualifying heading was discarded.
Semantic search can find text related in meaning even when the wording differs. Exact identifiers, product codes, and policy names may also benefit from keyword matching. OpenAI’s retrieval documentation covers vector stores, attribute filtering, and hybrid search settings. See the retrieval guide.
Enforce access before restricted text enters the model’s context. Asking the model not to reveal a document is not a substitute for preventing unauthorized retrieval. Also define how edits and deletions propagate: an outdated passage should not remain the assistant’s answer after the source changes.
Evaluate retrieval and answers separately
For each question in your test set, identify the passage a reviewer would use. First ask whether retrieval returned that evidence. Then ask whether the generated answer accurately used it.
This separation produces actionable diagnoses:
- The right document never appears: investigate indexing, permissions, query matching, and ranking.
- The right passage appears without its exception: investigate chunk boundaries and surrounding context.
- The evidence is present but the answer contradicts it: investigate instructions and answer generation.
- The answer is correct but the citation is unrelated: investigate how citations map to retrieved passages.
Do not score an answer as grounded merely because it contains a link. The linked source must support the specific claim. Include “no supported answer” cases so guessing is not rewarded.
When fine-tuning becomes worth testing
Fine-tuning can be a candidate when you need consistent formatting, classification, or response behavior across many representative examples and prompting has not met the requirement. Compare it against the best simpler baseline using held-out cases.
Budget for preparing examples, checking their quality, running evaluations, and maintaining the trained configuration. If the real defect is stale knowledge or missing documents, improving retrieval is usually the more direct experiment.
A combined system can retrieve current evidence and use a tuned model to produce a particular response style. That adds two components to maintain. Require evidence that the improvement justifies the extra complexity.
A decision you can test this week
Choose one narrow use case and an approved document set. Build a direct-context baseline. Add retrieval only when the baseline’s size, access, or relevance constraints warrant it. Compare source coverage, answer correctness, latency, and cost using the same questions.
Keep a record of source revisions alongside prompt and model versions. Then apply the AI deployment readiness checklist before giving the system real tools or customer traffic. The goal is a system you can inspect and improve, not a particular architecture label.
FAQ
Is RAG the same as training a model?
No. RAG retrieves information for a request; fine-tuning changes model behavior through training. Indexing documents for retrieval is not the same as training those documents into a model’s weights.
Can RAG prevent hallucinations?
No. It provides evidence, but retrieval can miss the right material and generation can misinterpret it. Test unsupported questions, citation accuracy, and fallback behavior.
Should private documents go in one shared index?
Only if the system reliably enforces the required access boundaries. Tenant and document permissions must be applied before retrieved text reaches the model, and tested with cross-account cases.
