Getting Bad RAG Answers? Check Retrieval Before Changing the Model

If search results miss the mark, check how documents are read and split before adding a larger model. When table columns get scrambled or headings become detached from their text, retrieval and generation may struggle to correct the error. RAG quality depends on a process that runs from parsing through evaluation.

This is the second post in the enterprise RAG series. Part 1 covered Markdown knowledge bases and tool selection; here, we look at parsing, chunking, and search. The final post covers permissions, memory, security, and operating costs.

Preserve document structure during parsing

Extracting only text from a PDF can change reading order or break the rows and columns of a table. Clause numbers in a contract, headings in a manual, and table column labels can matter as much as the text itself. A parser extracts text and layout from a file, then converts that structure into a form a search system can use.

Tools such as Docling offer document-layout analysis and table processing, but results from a public benchmark do not guarantee how your documents will fare. You can also compare document-processing tools such as Unstructured or managed parsing services such as LlamaParse. A public library and a paid API may differ in features and in how they process data. Try representative scanned PDFs, multi-column documents, and tables with merged cells, then compare the extracted output with the originals. Check whether table titles and row-column relationships survive, and whether repeated page headers are mistakenly added to the body.

Chunks that keep the context a question needs

Searching an entire document at once can make it hard to find the passage relevant to a question. Split it too finely, and conditions needed to understand the answer may get separated. Start by looking at Markdown headings and paragraph breaks, but do not assume one splitting rule fits every document. Tables, lists, FAQs, and policies may each have different boundaries.

For example, the question “What is the meal allowance for business travel?” needs more than a chunk containing only the amount; the applicable people and exceptions may matter too. You can try adding the heading path to a chunk or including a neighboring paragraph. There is no universally correct chunk size or overlap length. Adjust them by checking whether evidence gets cut off in real questions. A tool such as LangChain’s MarkdownHeaderTextSplitter can split by heading and then further divide long sections. Keep the source document ID, heading path, version, and source location with each chunk so you can trace the evidence behind an answer.

Why combine lexical and semantic search

Vector search can help find documents with similar meaning even when they use different wording. By contrast, lexical search such as BM25 may do a better job when an exact match matters, as with an internal product code, error identifier, or clause number. Hybrid search combines the two as a way to complement different retrieval signals. It does not improve performance for every corpus and question, so evaluate whether lexical search alone is sufficient too.

When the two sets of scores use different ranges, Reciprocal Rank Fusion (RRF) can combine results by rank. Instead of trying to align each search engine’s score scale, it considers where each result appeared in the ranking. Which combination works better depends on the relevant documents for each question and the number of candidates retrieved. Build an evaluation set with both common and difficult questions rather than drawing conclusions from a handful of examples.

How to evaluate embeddings for Korean documents

Korean documents can contain particles and verb endings, variations in spacing, English abbreviations, and product names. Public multilingual models such as BGE-M3 and multilingual E5 are available, as are embeddings offered through APIs. A model card’s language-support list or output dimensions can help narrow down candidates, but they do not rank search performance on your internal documents. Compare commercial APIs and locally deployed models for cost, data handling, and operational effort as well.

Collect questions paired with the documents that contain the correct answers and compare retrieval metrics such as Recall@k and nDCG to see whether relevant documents appear near the top. Recall@k measures how much of the correct evidence appears in the top k results; nDCG measures whether more relevant results rank earlier. Add latency and request cost to understand the operating tradeoffs. Testing differently worded versions of the same question, internal abbreviations, and specific document numbers from real search logs can reveal differences that model cards do not show.

Apply Korean text preprocessing carefully. Keeping only nouns or removing stop words in bulk can erase negation, conditions, or time references and change the meaning. Morphological analysis may help with some corpora, but compare it with a baseline that preserves the original text before adopting it.

When to use reranking and freshness weighting

If the first search retrieves a broad set of relevant documents, a reranker can look at each document alongside the question and reorder the results. This can help when there are many candidates or when similar documents need to be prioritized, but it adds model calls and computation. Compare the position of relevant documents and answer quality before and after reranking to see whether it helps in practice. Candidate counts and response times are not the same across all systems.

Freshness should not be determined by date alone. In some workflows, an effective date, revision status, or whether a document has been retired matters more than its announcement date. You can store metadata such as creation date, effective start and end dates, and version, then apply it to search results according to the work rules. Include examples where an older document is still valid in your evaluation set. Also distinguish questions about current policy from questions about policy at a past point in time. Decide whether to filter candidates first by retired status and validity period or apply a freshness weight to the remaining candidates; this distinction helps avoid breaking searches for historical information.

Evaluate retrieval failures separately from generation failures

When a RAG answer is wrong, separating possible causes helps you improve it. If the correct evidence never appeared among the candidates, the problem may be parsing, chunking, or retrieval. If it appeared but the answer did not use it, the issue may lie in context construction or generation. Record the relevant document and evidence passage needed for each answer, then inspect what each stage selected.

Rather than deciding based on one performance number, check whether the system finds evidence for real work questions and whether people can spot incorrect answers. Once documents have user-specific permissions or conversations add memory, retrieval design meets security design. The final post in the series looks at that boundary.