Agent evals

DaloyJS ships instructions for AI coding agents: an AGENTS.md and a daloyjs-best-practices skill in every scaffolded project, the express-to-daloyjs skill, and a docs MCP server. Those files are prompts, and a prompt cannot be unit-tested. You can only find out whether it works by running agents against realistic tasks many times and measuring the result. That is what an eval is.

The repo's evals/ harness answers four questions with numbers instead of impressions:

• Can an agent complete common DaloyJS tasks from the shipped instructions alone?
• Under pressure (a failing test, a 413, a 429), does it keep the secure defaults, or delete them to make the test pass?
• How much does each instruction surface (AGENTS.md, the skill) actually help?
• Does the docs MCP search return the right page for the questions agents ask?

Evals versus tests

The framework itself is deterministic code, so it is covered by ordinary tests, coverage gates, and the red-team suites. Evals are for the parts whose behaviour depends on a model:

What gets measured

The four guardrail tasks are the reason the harness exists. They recreate the failure mode from Supabase and Aikido's secure-by-default write-up: an agent told to "make it work" removes the check that blocks it.

Graders

No LLM judge decides pass or fail. Every grader is deterministic:

Run the evals

1. Build the framework

   Each workspace gets a copy of dist/, so the agent codes against the current source.

   bashCopy

   pnpm build

2. Calibrate the graders

   Calibration needs no AI and no network. For every task it proves that the reference solution passes every grader, doing nothing fails, and each known-unsafe shortcut is caught by the graders it names.

   bashCopy

   pnpm eval:calibrate

3. Run a live agent in a sandbox

   Live agents get broad tool permissions inside the workspace, so the runner refuses them unless you pass --sandbox-ack. Pass it only inside a container or VM that holds no secrets except the agent's own API key.

   bashCopy

   pnpm eval --agent codex --sandbox-ack --suite guardrail --trials 5
   pnpm eval --agent claude --sandbox-ack --ablation full,bare --trials 3
   DALOY_EVAL_AGENT_CMD='["my-agent","--prompt","{prompt}"]' pnpm eval --agent command --sandbox-ack

Useful flags: --task <ids>, --model <id>, --k <n>, --timeout-min <n>, --fail-under <0..1> (exit 1 below that pass rate), --keep (keep workspaces for inspection), and --list. Results land in evals/results/ as JSON (every trial, its grades, and the agent's diff) plus a Markdown summary, which is also appended to the GitHub Actions job summary.

Read the results

Agents give different results on each run, so every task runs n trials and the report shows:

• pass@1: the plain success rate.
• pass@k: the chance that at least one of k attempts passes ("can it do it at all?").
• pass^k: the chance that all k attempts pass ("can we rely on it?"). For guardrail tasks this is the number that matters. An agent that strips secureHeaders() one run in five is not safe.
• Failing graders: which check failed how often. guardrails×3 points at a different problem than hidden×3.

Run the same tasks under --ablation full,no-agents-md,no-skill,bare to measure what each instruction surface contributes. If bare scores the same as full, the instructions are not doing their job.

Docs retrieval eval

The website has its own deterministic eval: website/tests/evals/docs-retrieval.test.ts runs realistic agent questions through the same rankDocPages() that powers the MCP search_docs tool and reports recall@1, recall@3, and mean reciprocal rank over the real docs corpus. The thresholds are regression floors set from the measured baseline, so a ranking or content change that sends agents to the wrong page fails cd website && pnpm test. When a real search misses, add the query to docs-retrieval.dataset.ts with the page that should have won, then fix ranking or content. Never edit the expectation to match the current result.

Add a task

evals/tasks/<task-id>/
├── task.ts            # prompt, setup, graders, reference, unsafe variants
└── hidden/
    └── *.test.ts      # behaviour the agent never sees

1. Write the prompt like a real user request. Never mention what the graders check.
2. Build the scenario in setup with the helpers in evals/tasks/helpers.ts. They throw if a template anchor moves, so template drift shows up as a calibration failure.
3. Write a reference solution and at least one unsafe variant that simulates the shortcut you expect, with the graders that must catch it in expectFail.
4. Register the task in evals/tasks/index.ts, then run pnpm eval:calibrate --task <task-id> and pnpm test.

Grow the task set from real failures. Each time an agent ships something wrong in a DaloyJS project, turn that situation into a task.

In CI

.github/workflows/evals.yml runs calibration whenever the harness, the templates, or src/ change, and weekly. Live agent runs are manual (workflow_dispatch): pick the agent, an exact CLI version, the model, suites, ablations, and trial count. The run uses the protected agent-evals environment for the agent's API key and uploads the JSON results as an artifact. Nothing agent-related runs on pull requests.

Use the pattern in your own app

The same ideas work for a team checking its own agents on its own DaloyJS service:

• Keep a few hidden app.request() tests per task, including the unhappy paths an agent is likely to skip.
• Diff the secure defaults before and after, and fail on any weakening. See the secure-by-default guide.
• Run every task several times and track pass^k, not a single lucky run.