Agents change under you. Someone moves one to a cheaper model to save money. Someone else adds a paragraph to the prompt after a bad run. Each change looks reasonable, and nobody can tell whether the agent got better or worse.
Machine learning teams handle this with hill-climbing. You make one change, score it against a fixed set of real tasks, and keep it only if the score goes up. Then you do it again. Runlayer Agents now have the tools for each step, starting from runs your agents have already finished.
Step 1: Read the failures first
Open a run, mark it Pass or Fail, and write one line about what went wrong, like “used last month instead of last week.”
After a few dozen notes, click Analyze failures on the agent’s Evals tab. Runlayer groups the failing notes into named categories, with a count on each. Hamel Husain and Shreya Shankar call this error analysis and rank it first among eval activities. If you write scorers before reading runs, you end up testing for failures you imagined.
Step 2: Turn real runs into a test set
Each failure category has a Create eval button that starts an eval from its runs. You can also pick up to 20 finished runs from the Monitor tab. Scheduled agents are the easiest place to start. They do the same task every day or week, so you already have plenty of comparable runs to replay.
The test set is work the agent has already done, so there’s no synthetic dataset to write.
Step 3: Change one thing
An eval tests a single change: a different model, an edited system prompt, or, where your workspace allows it, a different harness.
For every run you pick, Runlayer starts two fresh replays. The control replay uses the model and prompt captured from the original run. The candidate replay uses the same setup with your change applied. The two replays differ only in that change. The original output stays on the page for reference.
Replays stay out of production
An eval of an agent that posts to Slack or writes to a CRM shouldn’t post or write anything. By default, replays never call your real connectors. An LLM simulates each tool result, using the calls recorded in the original run as reference. Later results stay consistent with earlier ones in the same replay, so a release list doesn’t change between two calls. Memory writes are off. Commands inside the agent’s sandbox still run.
You can let some tools run for real on a per-eval basis, for example simulating only tools that write. The dialog warns you that those replays can cause real side effects.
Step 4: Score what you care about
Three built-in LLM scorers cover the common questions. Baseline battle compares candidate to control and returns improved, similar, or regressed. Goal completion rates how much of the task the replay finished. Instruction adherence checks the replay against the agent’s instructions.
Each failure category from Step 1 should get its own scorer. Click Author scorer with AI, describe the check in a sentence (“score 1 when the report covers exactly the last seven days”), and Runlayer writes a TypeScript scorer you can read and edit before you save it. Give each scorer one failure and a pass or fail answer. Scorers are frozen into an eval when it starts, so editing one later doesn’t rewrite old results.
Step 5: Check the judge
An LLM judge can be wrong. Open a case to see the original, control, and candidate outputs next to each other, read every scorer’s reasoning, and record your own verdict.
Once you’ve labeled a few cases, each comparative scorer shows Agreement with you: how often its verdict matched yours, with a confusion matrix. Below 10 labels, Runlayer warns that the number is too noisy to trust. If the judge often disagrees with you, fix the scorer before you act on its results.
Step 6: Read cost and quality together
We ran this on one of our own agents. Every week it checks new releases, finds the owner of each feature, and nudges them in Slack to tell the customers who asked for it. We tried moving it to Claude Haiku 4.5 and replayed five past runs.
Across the five replays, the candidate cost $0.97 and control cost $5.84. The judge marked all four completed cases as regressed, and one replay failed. The judge’s reasoning explained why: the agent’s Slack nudges couldn’t be delivered. The control replay caught that and reported it. The Haiku replay didn’t.
Going by cost alone, you’d ship it. With the verdicts next to the cost, you keep the bigger model for this agent.
On a simpler agent the same eval can come out the other way, and then you have the numbers to back the switch.
Step 7: Keep the winner and guard it
When a candidate wins, apply the change and start the next round from there.
A model or runtime can change under an agent that nobody has touched. Run an eval with no override and it becomes a regression test: each run replays once under its original setup and is scored against the original output. Run one when your provider ships a new model version.
A climb only improves what your scorers measure. Go back to labeling runs every few weeks so new failures get scorers of their own.
Let an agent do the climbing
Evals are also available as Runlayer MCP tools, including list_agent_runs, create_agent_eval, and get_agent_eval, so an agent can run the loop. An optimizer agent on a weekly schedule can collect last week’s failing runs, draft a prompt edit, run the eval, and post the verdict counts and cost change to Slack. A person reads the result and decides whether the change ships.
Skills give you another lever. In my talk at AI Engineer World’s Fair this year, I showed an agent go from 9 successful runs out of 25 to 25 out of 25 after turning on self-improvement, which lets an agent distill its own skills from earlier runs. An eval tells you whether a change like that worked on your tasks.
Governed like every other run
Replays appear in the agent’s run history and bill to your configured AI provider. Only people who can edit the agent can create evals, record verdicts, or manage scorers. If your workspace redacts run content, a redacted run has to be unredacted before it can be replayed. Testing a change in a replay becomes the golden path for changing an agent, under the same controls as every other run.
Start with one agent
Pick the agent your team spends the most on, or the one people complain about most. Label 20 of its runs this week, run one eval with a cheaper model, and compare the verdicts to the cost.
Runlayer is the leading AI control plane, with AI enablement, security, and control in one platform. AI-native companies including PagerDuty, dbt Labs, and Baseten run their agent strategy on Runlayer. Offline evals are available now for Runlayer Agents.