Preparing documents for RAG: fix the source before changing the prompt
Collect documents from a connected knowledge base or public help site, then check extraction, versions, and chunk boundaries before indexing them for RAG.
Your knowledge-base assistant keeps quoting an old policy or leaving out part of a table. Beeuni can read the source documents, find outdated copies and extraction errors, and prepare corrected text for your search index. This walkthrough uses a shipping guide to show what to check and what to export.
Retrieval-augmented generation, or RAG, finds relevant passages in your documents and gives them to the model when it answers a question. These passages are often called chunks: smaller sections prepared for search. A Beeuni agent can collect documents from your connected knowledge base or public help site, extract their text, and run scripts to prepare those passages. Your application then indexes and searches them with the tools you choose.
Collect the knowledge base where it lives
For a public help site, give the agent a starting URL and the sections to include. Ask it to follow the relevant links, save the article text, and report pages it could not open. For private documentation, connect the service and grant access to the intended collection. A private URL alone does not make the page readable.
Keep a collection manifest with each page or document reference, title, retrieval time, and version information where the source provides it. A repository commit can identify a documentation snapshot; a live help page may need its own saved copy. The agent can compare this inventory with the previous import to identify changed, removed, and newly discovered pages before preparing another export.
Start with a question the application gets wrong
Find the passage that should answer it in the original document. Compare that page with the extracted text and the chunk returned by retrieval. This separates an extraction problem from a retrieval problem. If the correct passage reached the model, inspect how the answer used it.
| What you see | What to inspect | What to change |
|---|---|---|
| A value without its unit | Table headings and adjacent cells | Keep headers and units with the relevant rows |
| An answer based on an old rule | Document versions and effective dates | Remove superseded text or make version filtering explicit |
| A sentence cut off halfway | Chunk boundaries in the exported text | Split at section or paragraph boundaries where possible |
| Repeated navigation in results | Headers, footers, and page menus | Remove repeated boilerplate without deleting substantive notes |
| No matching passage | Extraction coverage and retrieval results | Recover missing text, then retest indexing and retrieval |
A table header can change the answer
Imagine a shipping guide with a table headed “Delivery time, business days after dispatch.” Its row for the standard service contains the value “3–5.” An extraction that keeps only “Standard: 3–5” has lost both the unit and the starting event. The model now has to guess whether the number means days, a price, or something else.
Example of a passage with its context preserved:
Document: Shipping guide
Section: Delivery estimates
Service: Standard
Delivery time: 3–5 business days after dispatch
Source: sample-shipping-guide.pdf, page 2That passage can support “How long does standard delivery take after dispatch?” It cannot answer “Will my order arrive by Friday?” without the dispatch date and relevant calendar. Include both questions in review: the first checks whether the table survived extraction, and the second checks whether the application recognizes missing information.
What should travel with each passage?
Keep a stable document ID, title, section, page reference where available, and version or effective date. Preserve the source URL for public pages. Those fields let your application show a citation and distinguish two versions with similar wording.
There is no chunk size that fits every collection. A short FAQ answer can stay whole. A long procedure may need several chunks, each carrying its section heading. Ask the agent to export a sample, then read the chunks without opening the source. Check whether the terms, units, and exceptions still make sense.
Can the agent write evaluation questions too?
It can draft candidates from the documents. Review the expected answers and source passages before using them to judge your application. Include questions whose answers cross sections, questions about outdated rules, and questions the collection cannot answer. Keep this evaluation set separate from examples used to tune the system.
Follow one failure through the retrieval results
Return to the fictional shipping guide. Suppose the current guide says standard delivery takes 3–5 business days after dispatch, while an older guide says 2–3. Your application answers “2–3 days.” Save the question, the retrieved passages in their returned order, their metadata, and the answer. You can now investigate where the outdated rule entered the response.
If only the old guide appears, check whether the replacement was imported and whether a version filter excluded it. If both versions appear, inspect how the application chooses between them. If the correct passage arrives without its heading, repair the extraction or chunk boundary. If the complete, current passage reaches the model and the answer still loses “business days after dispatch,” investigate answer generation. Re-extracting the collection would not address that last failure.
Keep this trace with the evaluation case. After a change, compare the passages as well as the answer. A plausible answer can hide a retrieval miss when the model already knows a similar fact. It may fail on the next question about a rule specific to your documents.
Test chunk size with your evaluation questions
Larger chunks can preserve a procedure and its exceptions, but they also bring unrelated text into the prompt. Smaller chunks can make a fact easier to retrieve while separating it from a condition that makes it true. Overlap may help at boundaries, but it can fill several result slots with nearly the same passage. Inspect returned result sets before changing these settings across the collection.
For each reviewed question, mark which passages are sufficient to support an answer. Check whether they appear among the results your application sends to the model. Record irrelevant and duplicate results too. Evaluate final answers separately for factual support, citations, and recognition of missing information. A single overall score makes it harder to see which part changed.
Keep the test questions and retrieval settings with each export. When testing cleaner source text, use the same retrieval configuration first. When testing chunk size or ranking, hold the source version fixed. Changing both at once may improve the answer, but leaves you unable to tell which change helped.
What should you export from Beeuni?
Ask for the cleaned text, passage records with metadata, a source manifest, and a list of extraction failures. Import those files into your indexing pipeline.
The dataset preparation guide covers review records and data splits. If you need to trigger preparation from your application, see using Beeuni as an agent backend.
More articles
Infrastructure for RSI research: running and evaluating self-improving agents
How Beeuni helps researchers run repeated agent-improvement experiments: edit code and instructions, execute tests, investigate failures, and compare versions.
Use Beeuni to run and evaluate a model fine-tuning experiment
Have an agent prepare training data, run a fine-tuning job on your connected training computer, inspect failures, and compare model outputs.