Fine-Tuning Can't Teach a Model New Facts. That's Why Most "Train It on Our Data" Projects Fail.

The request we actually get

"Can you train the model on our documents" is one of the most common ways a prospective client describes an AI project to us. It's a reasonable thing to want and almost always the wrong description of what needs to happen. Fine-tuning and retrieval-augmented generation (RAG) solve different problems, and the mismatch between which one a team asks for and which one they actually need is where a lot of AI budgets go to die.

The distinction that matters isn't RAG vs. fine-tuning as competing technologies. It's knowledge vs. behavior as competing problems. Get that sorted first and the tooling choice mostly falls out on its own.

What fine-tuning actually changes

Fine-tuning adjusts the model's weights by training it further on examples you provide. What that buys you is a model that behaves differently by default: it adopts a tone, follows a rigid output format more reliably, gets better at a narrow task pattern, or stops needing a long instruction prompt to act a certain way every time.

What it does not reliably buy you is a model that knows new facts. Training a model on a stack of policy documents or product specs doesn't make it "know" those documents the way retrieval does — it nudges the model's general behavior in the statistical direction of that training data, which is a different thing from being able to recall and cite a specific clause on demand. Ask a fine-tuned model a pointed factual question about something in its training set and you'll often get a plausible-sounding answer that's subtly wrong, because the model learned a style and a vibe, not an index it can look up against.

This is also why fine-tuned knowledge goes stale in a way that's expensive to fix. If your product catalog, pricing, or policy changes, a RAG system updates by re-indexing a document. A fine-tuned model updates by re-running training — a slower, costlier cycle you'll be repeating every time the underlying facts move.

What RAG actually changes

RAG leaves the model's weights alone and instead changes what the model sees at request time: it retrieves relevant chunks of your data and inserts them into the prompt before the model answers. The model is reasoning over source material you handed it in that specific request, not over a lossy statistical compression of training examples.

That's why RAG is the right default when the problem is "the model doesn't know our stuff" — current pricing, internal policy, a knowledge base, support history, case law, whatever. It's also why RAG systems can cite their sources and fine-tuned models generally can't: the retrieved text is sitting right there in context, traceable back to a document.

The tradeoff is architectural, not magical. Retrieval quality is a search problem, and search problems fail in boring, specific ways — bad chunking, bad embeddings, queries that don't match how the documents are phrased, retrieval that returns technically-relevant-but-useless chunks. A RAG system that returns the wrong three paragraphs will produce a confident, fluent, wrong answer, because the model is good at sounding right given whatever context it's handed. This is a separate failure mode from fine-tuning's and it's the one we see teams underestimate most.

Why "train it on our documents" is usually the wrong ask

When a client says this, they're almost always describing a knowledge problem — they want the system to answer questions using their specific content. Fine-tuning is the wrong tool for that job, not because it's bad technology, but because it optimizes behavior, and behavior isn't what's missing. The actual fix is a retrieval layer, and the fine-tuning instinct usually comes from a reasonable but mistaken mental model: that an LLM "learns" documents the way a person does by reading them once. It doesn't. It learns statistical patterns across many examples, which is a fit for teaching a model to always respond in a certain format or voice, and a poor fit for teaching it a specific set of facts you need recalled accurately.

The flip side also happens, less often: a team builds a RAG pipeline to fix an output that's technically accurate but reads wrong — off-brand tone, inconsistent formatting, missing the specific structure a downstream system expects to parse. More retrieval won't fix that. That's a behavior gap, and fine-tuning (or often just a better system prompt, which is cheaper and should be tried first) is the actual lever.

Where combining them makes sense

The two aren't mutually exclusive, and the strongest production setups we've built use both for what each is good at: a lightly fine-tuned model for consistent tone, format discipline, or a narrow task pattern, sitting on top of a retrieval layer that supplies the current, citable facts. The fine-tuning handles "how it answers," the retrieval handles "what it knows." Treating them as a single either/or decision is how teams end up fine-tuning their way around a retrieval problem, or bolting retrieval onto a model that was never going to behave correctly no matter what context you feed it.

The cost shape worth knowing before you commit

The specific dollar figures vendors publish for either approach vary too much by provider, model size, and data volume to treat any single number as a reliable benchmark — be skeptical of anyone quoting you a precise figure without having seen your workload. What's stable across providers is the shape of the cost, not the number: fine-tuning concentrates cost upfront, in data preparation and training runs, and in the ongoing work of retraining when your underlying facts change. RAG concentrates cost in the retrieval infrastructure and in the extra tokens you pay for on every single request, since retrieved context gets billed every time. Low request volume with frequently changing source material favors RAG's cost shape. Very high, stable request volume on a narrow, unchanging task is where fine-tuning's upfront cost starts to pay for itself.

The takeaway

Before anyone scopes a RAG build or a fine-tuning job, answer one question: is the system getting facts wrong, or is it getting style and format wrong? Facts wrong means a retrieval problem — fix the search, not the weights. Style or format wrong means a behavior problem, and a better prompt is worth trying before either one. Skip this question and you'll end up fine-tuning your way around a search bug, which is the single most common way we've seen an AI budget get spent on the wrong fix.