Use Beeuni to run and evaluate a model fine-tuning experiment

The training script exits overnight. The output folder contains a checkpoint, a saved model state from a point in training, but the log ends before evaluation. Did training finish? Can that saved model be loaded? Which settings produced it? A Beeuni agent can inspect the files and logs, try loading the checkpoint, and report what finished before you decide whether to run it again.

With access to your training computer, the agent can inspect the code and data, run commands, read logs, and collect evaluation results. You choose the model and what counts as an improvement. Beeuni uses the training software and compute available on that computer to carry out the work.

Start with one behavior you want to change

Consider the support-request classifier from the dataset guide. It assigns requests to billing, account access, product help, or review. You want to test whether a fine-tuned model puts requests in the right category more consistently than the base model with instructions. Keep examples involving both access and billing: they will show whether the model learned which category takes priority when a request contains several problems.

Give the agent the label definitions, reviewed training records, and a separate evaluation set. Have it run the base model on that evaluation set before changing weights. Save the model identifier, instructions, generation settings, and predictions. Without that baseline, a successful training command tells you very little about whether the extra work helped.

Set up access to the training computer

Connect the computer that will run the experiment through Computer Tunnel and select it in the active profile. Allow filesystem and shell access to the training directory. Point the agent to the repository, dataset location, and Python environment. For a GPU training run, ask it to check the available device, memory, drivers, and framework compatibility before choosing settings.

The base model and training method determine how much memory and compute you need. With an adapter method, you train a small set of additional parameters while keeping the base model weights fixed. Updating every weight usually needs more resources. Ask the Beeuni agent to inspect your training script and available hardware, then explain whether the planned settings fit. Review any proposed package installation, model download, or environment change before starting.

Use the connected training computer and repository I specify.
Inspect the training script, model access, Python environment, and available devices.
Validate the reviewed dataset and keep the evaluation set separate.
Run the base-model evaluation and save its predictions.
Propose a small fine-tuning run with the settings and output directory shown.
Wait for my approval before starting training or changing the environment.
Save the command, configuration, logs, and evaluation results for each run.

Make the first run small enough to inspect

Before a full run, use a small training subset to check the pipeline. Can the tokenizer read the records? Are the labels represented consistently? Is the model being trained to predict the intended text? Can the script save a checkpoint and load it to answer a request? These checks catch a broken setup before you spend time training on the whole dataset.

Ask the Beeuni agent to show one example exactly as the training script will read it. Have it mark the text the model is being trained to predict and the text supplied only as context. The training loss measures errors on those predictions. If the script includes the wrong fields, training can run normally while teaching the wrong behavior. Inspect this example before relying on a loss chart.

Keep each experiment in a separate output directory. Record the code revision, dataset version, base model, training method, seed, and settings. If the method saves only an adapter, record the base model it requires. A folder of weights should come with enough information to load the model again.

Check training progress without starting the job again

Use the training computer’s job runner or a persistent terminal session, with logs written to disk. Have the agent record how to inspect and stop that job. A command launched during a tool call needs its own process management; the chat connection should not be the only record that it started.

When you ask for an update, the agent can check the recorded job, read the latest log, and inspect saved checkpoints. If the connection drops, check whether that same job is still running before launching another. Resuming a checkpoint depends on what the training framework saved, including optimizer and scheduler state when those are needed.

For an out-of-memory failure, have the agent preserve the failed configuration and read the error before proposing a change. Reducing batch size, shortening sequences, or changing the training method can affect both memory use and the experiment. Record the revised settings as another run so the comparison remains understandable.

Compare the mistakes, then decide what to keep

Run the tuned model on the same evaluation examples used for the base model. Keep these examples out of the training data, and keep prediction settings and scoring rules fixed. For the support classifier, compare results for each category and inspect the requests that changed category. An overall accuracy increase can hide worse handling of the rare cases you care about.

Ask Beeuni to write a report that links changed predictions to the source examples and run settings. If you revise the dataset after inspecting these errors, use a separate test set for the final decision. Repeatedly tuning against the same examples can make the evaluation look better without improving performance on new requests.

Keep the selected weights or adapter, loading instructions, environment details, and evaluation report together. Test loading the result in a fresh process. Deployment can follow once you accept the comparison; until then, the existing model can keep serving your application while the experiment stays separate.