A common question for an LLM user (company or individual) in a multipolar LLM world is, “How would this new/cheap/fast model behave on my workload?”
One way to do this is rely on public benchmarks, i.e., if majority of the workload looks like SWE Bench or ProgramBench, just see what are the scores on the benchmarks. This is unsatisfactory for a few reasons:
- Models are known to be “Benchmaxxed” on public benchmarks.
- Your workload doesn’t necessarily look similar to benchmarks.
- You or your engineers use the models/agents in a different way. Different skills, maybe, many might brainstorm a bit, collect context, etc. before asking the agent to implement something.
Given all that, a second option is to work towards creating a custom benchmark. Carefully design a set of problems and evaluation environments which represent the kind of work you do at the company, are different from existing popular evaluation datasets, and models are not yet perfectly capable of doing them.
In an ideal world, this is the right direction, but it takes:
- Someone who understands evaluation environments, harnesses, and the current state of the art in terms of what are popular evaluation benchmarks.
- Also understands different work happening inside your company: what people tried agents for, what failed, and, more importantly, what failed because of model capability rather than because of not having provided the right access, tools, or context.
- This is not a one-and-done thing. One needs to keep curating more evaluation traces. Most companies do this once and don’t want to continue investing in this. Also, it is somewhat difficult to decide if this falls under cost optimization, AI-forward strategy, engineering, or platform teams’ responsibility.
Having been part of such an endeavor before, I think the best way to do this is what I call “Replay Evals.”
Replay Evals
First, convince yourself and your CTO/Director of IT that you need to log AI-agent traces. Figure out how you want to do it: use Logfire, Braintrust, or LangSmith, or roll out your own tracing—add hooks to all harnesses, decide on a shared schema, dump to a DB.
Once done, you need to run. Consider what is run.
Categorization of use cases you value the most to be evaluated with/against. For example: is it document understanding, market research, customer research, coding? If so, what kind of coding: frontend, backend, webservice, database, etc.?
Classification
Once done, you need to understand this “workload” that LLMs/agents are executing for/with your employees. Best way is to add a post-classification hook. At the end of a session, create a classification task.
“Given this interaction with the user, classify the trace in the following categories:
- Work type
- Subcategory
- External tools used
- User intervention needed
- Difficulty level.”
Now we come to the Replay part. What we need to ask is: if this work/task was subjected to a different LLM/agent/harness, would we have gotten the same result? If so, would it have taken less time/tokens/$? If not, would the user drive it in the right direction? Would it succeed, or is it just hopeless?
In order to do that, we use something called a meta-harness, i.e., an agent harness that uses an LLM to drive another LLM in some harness. E.g., Opus-4-8 driving GPT-5.6/Claude in Codex or Kimi K3 in Pi.
What we are trying to do is give the meta-harness a summary of the ground truth trace (the one you logged) and let it act as a user trying to get this task done using a candidate model (the one you want to assess).
At the end, you want to know:
- Completion percentage—how much of the task was able to be done, 0–100%.
- Amount of tokens.
- Cost.
- “User” intervention relative to base trace—how many “nudges” it took the driver model to get the candidate model to do this.
Evaluation
So now we have:
- A task—with its classification profile.
- One base trajectory and the model/harness it used.
- One candidate trajectory.
- Quantitative numbers to compare the two.
So now you can answer questions like:
“If my frontend team used GPT-5.6 instead of Claude-Opus-5, would it even be able to do the task? Cost less? Require less/more human intervention/help?”
Do this for a sample of agent traffic you log and you now have a good way of answering which new models should I pay attention to? Which tasks can be migrated to a cheaper model safely and save money!