Build an AI training dataset from your existing documents

Your support history could help train a request classifier, but it is scattered across tickets and replies. Beeuni can collect the selected conversations, turn them into labeled examples, and run checks for missing fields and duplicates. You review uncertain labels and get a JSONL dataset, with one JSON record per line and links back to the original tickets.

Choose the behavior before the file format

Suppose you want a model to classify incoming requests. Each example needs a request and an approved category. If you want answers grounded in a handbook, an example needs a question, supporting text, and a checked answer. Converting every paragraph into a question does not establish that the answers are correct or useful.

Connect the service that holds the material, or give the agent access to the documentation repository. Start with a named collection or date range. For public documentation, specify the site and sections to include. Ask the agent to list the sources it collected and any pages it could not read before reviewing the first examples.

Work through a small labeling problem first

Consider a fictional support team preparing a request classifier. It wants four labels: billing, account access, product help, and needs review. Through an authorized support connector, the agent reads closed tickets from an agreed date range. Collect only the fields needed for labeling and source tracking, leaving out contact details and unrelated attachments. The queue each ticket reached is not necessarily its correct label: a billing agent may have answered a forwarded login question.

Save a source copy in the Computer with the ticket references, selection rule, and collection time. Run the conversion against that copy so a ticket edited halfway through review does not silently change the examples. The agent can write and execute the extraction and validation scripts, then return the dataset, scripts, and review report together.

Start with requests that include short messages, long exchanges, and more than one issue. Decide what the input will contain at prediction time. If the model will see only the first customer message, do not put the eventual resolution into the training input. Otherwise it learns from information your application will not have when a new request arrives.

Write the rule for the third row before asking the agent to label the rest. You might route according to the immediate blocker, or send any request with competing intents to review. Both policies can be reasonable. Mixing them across annotators produces inconsistent examples for the model to learn from.

Use disagreements to improve the labeling guide

Have two reviewers label the same small batch independently. Keep their original labels and the final decision. When they disagree, record the reason: missing context, overlapping categories, or a labeling mistake. An agent can collect these cases into a review file, but the person responsible for the workflow should settle the routing policy.

If a decision changes the labeling guide, revisit earlier examples affected by that rule. For the fictional classifier, choosing “immediate blocker” puts the cancellation request under account access. Other messages combining sign-in trouble with payments now need the same treatment. Correcting only the disputed row leaves the older inconsistency in place.

Keep source messages separate from synthetic variants. Rewording one request ten times may help cover phrasing, but it does not add ten independent situations. If you generate variants, record their parent example and review their meaning. A small change such as “was charged” becoming “will be charged” can change the intended label.

Keep a review record before producing training records

Use an intermediate schema that keeps the source beside each example. For a question-and-answer dataset, a made-up review record might look like this. The question and answer in your dataset must come from your own checked material.

{"id":"sample-001","source":"sample-handbook.txt","section":"Opening hours","question":"When does the desk open?","answer":"The desk opens at 9 a.m.","review_status":"pending","split":"unassigned"}

This is a working schema, not a training API payload. After review, convert approved records to the format required by your training tool. Keep the source references and review decisions in a companion file if that tool accepts only messages or input-output pairs.

Give the agent checks it can run

• Structure: parse every JSONL line and require the expected fields and value types.
• Sources: verify that each example points to an existing source and section.
• Duplicates: compare normalized text, then review near-duplicates separately. Similar wording can still describe different facts.
• Privacy: scan for personal information and credentials, then review flagged records before export.
• Coverage: count examples per category and list categories with little or no reviewed material.

A script can check that a source reference exists. It cannot prove that the answer follows from that source. Inspect the answers, including examples with missing context or contradictory passages. Keep generated candidates separate from approved records.

Keep the same case out of both training and testing

Decide what the evaluation is meant to measure. To test performance on unseen documents, keep all examples from one document in the same split. Where several files describe the same case, group by case instead. Randomly splitting rows can put a near-copy of an evaluation answer into training.

Ask for train.jsonl, validation.jsonl, a source manifest, and a validation report. The report should include record counts, rejected records, duplicate groups, and the split rule. Once the dataset passes these checks, you can have Beeuni run a fine-tuning experiment on your connected training computer.

For the classifier, a usable delivery includes the label definitions, the input fields, and the rule for ambiguous requests alongside the JSONL files. Include the preparation script and its settings so another developer can reproduce the conversion. Record which source batch the export came from; a file named final.jsonl does not tell a reviewer what changed since the last export.

If the source starts as a spreadsheet, the CSV cleanup guide covers identifiers and missing values. To put dataset preparation behind your own interface, start with the SDK guide.