Your RAG Pipeline Is Missing Two Cheap Fixes: Hybrid Search and Reranking

The pipeline most teams actually ship

Ask most engineering teams what their RAG system does and you'll hear some version of: embed the documents, embed the query, pull back the top-k nearest chunks by cosine similarity, stuff them in the prompt. That's a real system — it demos well, it answers the obvious questions, and it's where almost everyone stops.

It's also missing two additions that are cheap, well-evidenced, and routinely skip the list because "the demo already works." The cost of skipping them doesn't show up in the demo. It shows up three months later, in the support ticket where the model confidently answers from the wrong chunk, or doesn't answer at all because the right chunk never made it into the top-k.

Why vector-only retrieval misses obvious matches

Dense vector search is good at semantic similarity and bad at exact matching. If a user asks about "invoice 4471-B" or a specific SKU, part number, or error code, a pure embedding search can rank semantically-related-but-wrong chunks above the one with the literal string match, because embeddings compress text into a representation that doesn't preserve rare tokens well. This isn't a tuning problem — it's a structural property of how dense retrieval works.

The fix isn't new. It's combining dense vector search with sparse keyword search (BM25) and fusing the two rankings — usually with Reciprocal Rank Fusion. Vendor benchmarks from Weaviate put hybrid search ahead of the best single method by roughly 2.6 to 13 points of nDCG@10, depending on dataset. A separate, more conservative benchmark (WANDS, an e-commerce retrieval set) showed BM25 alone and vector alone landing within half a point of each other, with untuned hybrid fusion beating both by about 1.2%. The range is wide because "how much hybrid search helps" depends heavily on how much of your query traffic is exact-match-sensitive — product catalogs and internal documentation with IDs and codenames benefit more than general-knowledge Q&A. But the direction is consistent across every benchmark we looked at: hybrid beats either method alone. It doesn't beat it by a vendor-slide margin every time, but it doesn't lose.

Most vector database platforms — Weaviate, Elasticsearch, OpenSearch, pgvector with an extension — support hybrid search natively now. The implementation cost is closer to a config change than a rebuild.

Reranking: the highest-leverage fix most pipelines skip entirely

Retrieval and reranking solve different problems. Retrieval has to be fast across millions of chunks, so it uses cheap approximate methods. Reranking runs a slower, more accurate cross-encoder model over just the top 20-100 candidates retrieval already found, and reorders them by actually reading query and chunk together instead of comparing pre-computed vectors.

The best-documented numbers here come from Anthropic's published work on contextual retrieval: adding a reranking step on top of contextually-embedded chunks plus BM25 took retrieval failure rates from 5.7% (embeddings alone) down to roughly 1.9% — about a two-thirds relative reduction. Cohere's Rerank product claims similar territory, with third-party testing reporting hit-rate improvements in the range of 25-30 points on hard retrieval tasks. Treat the exact percentages as vendor-reported and directional rather than universal, but the pattern — reranking catches real errors that pure retrieval misses — shows up independently across every source we checked, which is more than can be said for most AI tooling claims.

The practical case for reranking is that it's one more model call on a small candidate set, not an architecture change. If your pipeline retrieves 50 chunks and reranks down to the 5 that go in the prompt, you've added latency in the tens to low hundreds of milliseconds and a marginal per-query cost — in exchange for a measurable drop in wrong-chunk answers. For anything where a wrong answer has a real cost (customer support citing policy, internal tools answering compliance questions), that trade is usually worth it on the first pass, not something to defer until retrieval quality becomes a visible problem.

Chunking strategy: the one that's genuinely contested

This is where we'd expect to tell you "switch to semantic chunking," and the research doesn't support it cleanly. Semantic chunking — splitting text at points of topic change rather than fixed token counts — sounds obviously better and costs more to compute (it requires an embedding pass just to decide where to cut). Multiple 2025 studies, including one presented at NAACL, found no systematic evidence that semantic chunking beats simple fixed-size or recursive character splitting on real retrieval tasks. One benchmark found semantic chunking winning on raw retrieval recall but losing on end-to-end answer accuracy, because it tended to over-fragment into chunks too small to carry context.

Our read: start with recursive splitting at 256-512 tokens with overlap, which is cheap, predictable, and competitive with fancier methods in most of the published comparisons. Don't invest in semantic chunking speculatively — profile your actual failure cases first, and if the chunk boundaries are visibly cutting through the right answer, that's a signal worth acting on. If you want the more defensible upgrade, Anthropic's contextual retrieval approach — prepending a short LLM-generated summary of where a chunk sits in the document before embedding it — has real published numbers behind it and doesn't require reworking your chunking logic at all.

Where this actually sits on the priority list

If you're building or inheriting a RAG system and it's underperforming, the order of operations that matches the evidence is: add hybrid search first (it's close to a config change on most platforms), add reranking second (it's one more model call, not an architecture change), and treat chunking strategy and embedding model choice as tuning problems to revisit only after you've profiled actual failure cases — not places to guess upfront. Embedding model choice matters less than people assume once you're past the basics; the MTEB leaderboard changes month to month, and chasing the top spot is rarely where your retrieval errors are actually coming from.

The takeaway: vector search alone is not a retrieval strategy, it's a first draft. Hybrid search and reranking are both well-evidenced, inexpensive to add on top of an existing pipeline, and routinely skipped because the demo worked without them. If your RAG system is underperforming in production, check whether you've actually added either before reaching for a bigger model or a different vector database.