Skip to content
Beeuni
English
Esc
↑↓navigate↵open⌘Jpreview
On this page

Evals

Run agents on repeatable tasks and compare their scores.

An evaluation runs an agent on tasks whose verifiers score the result. Evals is experimental and off by default. Enable evals under Settings → Feature flags, then open Evals in your project. Project file storage must be available.

A first run

  1. Run beeuni evals init in your linked project to create a Harbor-format example task.
  2. Add the suite below to beeuni.yaml. Replace worker with an agent in your project.
  3. Publish the project changes with beeuni ship, then start the suite.
evals:
  smoke:
    tasks: evals/tasks
    agents: [worker]
beeuni evals init
beeuni evals run smoke --attempts 3 --wait
beeuni evals show <job-id>
beeuni evals compare <baseline-job-id> <new-job-id>

A single-step task has a task.toml, an instruction.md, and a verifier at tests/test.sh. For multi-step tasks, each step has its own instructions and verifier under steps/<name>/. The verifier is copied into the trial environment after the agent stops and writes /logs/verifier/reward.txt or reward.json.

Each trial creates its own session and sandbox. Trials consume credits like other sessions. Start with a small task set and few attempts.

All steps of a trial use the same workspace. Trials use the project workspace by default. To give each trial its own workspace, set workspace = "fresh" under [metadata.beeuni] in task.toml.

Results and controls

Read the job and trial pages for scores, errors and verifier output. Result files are also available under /Evals/<job>/ in Files. Compare runs with the same tasks and judge model before deciding a change improved an agent.

Command Effect
beeuni evals cancel <job-id> Cancel the job and stop running trials.

The SDK provides planProjectEvalJob to check a run without starting trials, startProjectEvalJob to start it, and getProjectEvalJob to read progress. Reuse an idempotencyKey for retries of the same start request. Run beeuni evals --help for datasets, trial actions and model-judge commands.

Was this page helpful?