Evals
Run agents on repeatable tasks and compare their scores.
An evaluation runs an agent on tasks whose verifiers score the result. Evals is experimental and off by default. Enable evals under Settings → Feature flags, then open Evals in your project. Project file storage must be available.
A first run
- Run
beeuni evals initin your linked project to create a Harbor-format example task. - Add the suite below to
beeuni.yaml. Replaceworkerwith an agent in your project. - Publish the project changes with
beeuni ship, then start the suite.
evals:
smoke:
tasks: evals/tasks
agents: [worker]
beeuni evals init
beeuni evals run smoke --attempts 3 --wait
beeuni evals show <job-id>
beeuni evals compare <baseline-job-id> <new-job-id>
A single-step task has a task.toml, an instruction.md, and a verifier at tests/test.sh. For multi-step tasks, each step has its own instructions and verifier under steps/<name>/. The verifier is copied into the trial environment after the agent stops and writes /logs/verifier/reward.txt or reward.json.
Each trial creates its own session and sandbox. Trials consume credits like other sessions. Start with a small task set and few attempts.
All steps of a trial use the same workspace. Trials use the project workspace by default. To give each trial its own workspace, set workspace = "fresh" under [metadata.beeuni] in task.toml.
Results and controls
Read the job and trial pages for scores, errors and verifier output. Result files are also available under /Evals/<job>/ in Files. Compare runs with the same tasks and judge model before deciding a change improved an agent.
| Command | Effect |
|---|---|
beeuni evals cancel <job-id> |
Cancel the job and stop running trials. |
The SDK provides planProjectEvalJob to check a run without starting trials, startProjectEvalJob to start it, and getProjectEvalJob to read progress. Reuse an idempotencyKey for retries of the same start request. Run beeuni evals --help for datasets, trial actions and model-judge commands.