--- title: Agent Evaluation description: "Evaluation discipline for agent systems, including scorer variance, skill/context uplift, and production-oriented metrics." created: 2026-05-25 updated: 2026-07-08 timestamp: "2026-07-08T00:00:00Z" type: concept tags: [agents, evaluation, software, context] sources: [raw/imports/knowledge/research/tessl/securing-the-coder-not-the-code-notes-on-agentic-development-and-security.md, raw/imports/knowledge/research/tessl/2026-05-20-ai-native-devcon26.md, raw/imports/knowledge/research/tessl/2026-05-20-coding-agent-failure-patterns.md, raw/imports/knowledge/research/tessl/2026-05-15-your-benchmarks-are-lying-to-you-and-your-judge-is-to-blame.md, raw/imports/knowledge/research/tessl/gpt-55-is-openais-best-model-but-paying-more-for-it-makes-no-sense.md, raw/articles/blogwatcher/2026-05-25/2026-05-01-tessl-from-blind-spots-to-merged-prs.md, raw/imports/knowledge/ai-agents/simon-willison/2026-05-20-tokens-per-second.md, raw/articles/blogwatcher/2026-05-26/2026-05-19-simon-willison-llm-gemini-0-32.md, raw/imports/knowledge/ai-agents/simon-willison/2026-05-19-gemini-35-flash-more-expensive-but-google-plan-to-use-it-for-everything.md, raw/articles/blogwatcher/2026-05-26/2026-05-19-simon-willison-datasette-llm-accountant-0-1a4.md, raw/articles/blogwatcher/2026-05-26/2026-05-19-simon-willison-llm-gemini-0-32a0.md, raw/articles/blogwatcher/2026-05-26/2026-05-19-simon-willison-datasette-llm-0-1a8.md, raw/imports/knowledge/ai-agents/latent-space/openai-gpt-next-disproves-80-year-old-erdoes-planar-unit-distance-problem.md, raw/articles/blogwatcher/2026-05-28/2026-04-30-tessl-stop-guessing-whether-your-skill-works.md, raw/articles/blogwatcher/2026-05-28/2026-04-28-tessl-common-pitfalls-of-skills-development.md, raw/articles/blogwatcher/2026-05-28/2026-05-27-latent-space-ainews-new-ai-infra-decacorns-fireworks.md, raw/articles/blogwatcher/2026-05-28/2026-05-27-pragmatic-engineer-building-opencode-with-dax-raad.md, raw/articles/blogwatcher/2026-05-31/2026-05-13-latent-space-ainews-the-end-of-finetuning.md, raw/articles/blogwatcher/2026-05-31/2026-05-08-bair-blog-adaptive-parallel-reasoning-the-next-paradigm-in-efficient-inference-scaling.md, raw/articles/blogwatcher/2026-05-31/2026-05-08-latent-space-ainews-gpt-realtime-2-translate-and-whisper-new-sota-realtime-voice-apis.md, raw/articles/blogwatcher/2026-05-31/2026-05-12-interconnects-ai-how-open-model-ecosystems-compound.md, raw/articles/blogwatcher/2026-06-15/2026-06-14-interconnects-ai-welcome-to-the-agi-era-of-ai-governance.md, raw/articles/blogwatcher/2026-06-15/2026-06-05-latent-space-how-to-stop-shipping-low-quality-rl-environments-with-examples.md, raw/articles/blogwatcher/2026-06-15/2026-06-04-latent-space-reality-the-final-eval-lukas-petersson-and-axel-backlund-of-andon-labs.md, raw/articles/blogwatcher/2026-06-15/2026-06-03-latent-space-scaling-past-informal-ai-carina-hong-axiom-math.md, raw/articles/blogwatcher/2026-06-15/2026-06-03-latent-space-ainews-microsoft-build-mai-thinking-1-and-mai-family-models.md, raw/articles/blogwatcher/2026-06-15/2026-06-02-simon-willison-microsofts-new-mai-models.md, ../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-23-tessl-the-new-tessl-review-now-you-decide-what-good-looks-like.md, ../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-23-tessl-ai-agent-governance-10-takeaways-from-engineering-leaders-on-agentic-development.md, ../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-23-tessl-why-warp-is-betting-engineering-leaders-are-done-picking-a-favourite-coding-agent.md, ../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-23-pragmatic-engineer-slow-down-to-speed-up-so-much-has-changed-in-6-months-time.md, ../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-24-pragmatic-engineer-tech-interviews-with-neetcode.md, ../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-22-latent-space-red-teaming-after-mythos-zico-kolter-matt-fredrikson-gray-swan.md, ../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-23-latent-space-ainews-spacex-is-already-a-28b-yr-neocloud.md, ../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-22-interconnects-ai-glm-5-2-is-the-step-change-for-open-agents.md] confidence: medium contested: false contradictions: [] --- # Agent Evaluation ## Definition Agent evaluation is the practice of measuring agent behavior across realistic tasks, repeated runs, model/tool/context variants, and explicit rubrics. In this wiki it is adjacent to [[agentic-coding]], [[agent-skills]], and [[agent-context-management]] because an agent’s output is shaped by the whole harness, not just the base model. ## 2026-05-25 Blogwatcher Synthesis Tessl’s May 2026 sources frame evaluation as a production discipline. The “judge is to blame” article shows that LLM-as-judge results can shift materially by scorer model, recommending multiple judges and binary/verifiable rubric criteria. The GPT-5.5 benchmark shows that model choice and skill/context quality interact economically: GPT-5.5 tied GPT-5.4 when skills were loaded while costing more per run. The large-codebase failure article adds that success depends on retrieval/context infrastructure and can be measured in tool calls, duration, cost, and refactor completeness. ## Practical Heuristics - Prefer rubrics with yes/no criteria where possible. - Run multiple scorer models for qualitative tasks and report judge variance. - Evaluate skills/context files as deployable artifacts, not static documentation. - Include cost, latency, tool-call count, and production/runtime signals alongside correctness. ## Related Pages - [[agentic-coding]] - [[agent-skills]] - [[agent-context-management]] - [[agent-security]] ## Blogwatcher Ingest Notes ### 2026-05-26 Evaluation signals in this batch are indirect but useful: Simon Willison's tokens-per-second note keeps latency/user experience visible as a production metric, Gemini/llm-gemini posts show fast-moving model/tool versions, and Latent Space's GPT-next and AI-infra items show why agent evaluations need to track changing model capability, cost, and serving infrastructure over time. Related: [[agentic-coding]] and [[agent-context-management]]. ### 2026-05-28 The evaluation theme became more operational: Tessl's skill-optimizer measures SKILL.md value by comparing task runs with and without a skill, while its pitfalls article argues activation, staleness, and oversized skill scope must be tested rather than assumed. Pragmatic Engineer's OpenCode interview and Latent Space's AINews recap both reinforce that harness quality, benchmarks aligned with developer experience, and acceptance/merge outcomes matter more than model-only scores. Related: [[agent-skills]], [[agentic-coding]], [[agent-context-management]]. ### 2026-05-29 Tessl's Composer 2.5 benchmark adds a model-selection example where the “fast” variant slightly outscored the regular model with skill context while completing runs faster at the same subscription price. The key durable lesson is not the exact leaderboard value, but that model choice, skill context, latency, and cost must be evaluated together rather than treated as independent decisions. Related: [[agent-skills]], [[agentic-coding]], [[agent-context-management]]. ^[raw/articles/blogwatcher/2026-05-29/2026-05-28-tessl-we-ran-composer-2-5-and-2-5-fast-across-11-skills-surprisingly-fast-won.md] ### 2026-05-30 Tessl's Opus 4.8 posts sharpen the evaluation lesson from prior batches: headline accuracy can hide operational gains. In one paired comparison, Opus 4.8 roughly matched Opus 4.7 on overall score but used fewer turns and lower cost; in a broader skills leaderboard, Opus 4.8 led with-skill results while remaining slow. Tessl's default-eval-model post also separates model-evaluation runs from skill-regression runs, arguing cheaper representative solvers can preserve lift signal while reducing eval cost. Related: [[agent-skills]], [[agentic-coding]], [[agent-context-management]]. ^[raw/articles/blogwatcher/2026-05-30/2026-05-29-tessl-ai-coding-agent-accuracy-opus-4-7-vs-4-8.md] ### 2026-05-31 Model/ecosystem sources in this batch preserve evaluation-adjacent signals around the claimed end of finetuning, efficient inference scaling, realtime voice APIs, and open-model compounding. These do not settle a benchmark question, but they broaden the evaluation frame from single model scores toward deployment economics, interaction modes, and ecosystem feedback loops. Related: [[agentic-coding]], [[anthropic]], and [[agent-skills]]. ^[raw/articles/blogwatcher/2026-05-31/2026-05-13-latent-space-ainews-the-end-of-finetuning.md] ^[raw/articles/blogwatcher/2026-05-31/2026-05-08-bair-blog-adaptive-parallel-reasoning-the-next-paradigm-in-efficient-inference-scaling.md] ^[raw/articles/blogwatcher/2026-05-31/2026-05-08-latent-space-ainews-gpt-realtime-2-translate-and-whisper-new-sota-realtime-voice-apis.md] ^[raw/articles/blogwatcher/2026-05-31/2026-05-12-interconnects-ai-how-open-model-ecosystems-compound.md] ### 2026-06-13 This Blogwatcher batch adds a fresh cluster around coding-agent economics and governance: model billing/name mismatches, context-engineering cost, reviewability as the bottleneck, agent skills/harnesses, enterprise operating models, Claude Fable behavior, Claude Code misuse/security reporting, and Simon Willison's concrete Datasette/MicroPython/WASM sandboxing examples. Related: [[agent-context-management]], [[agent-evaluation]], [[agent-security]], [[anthropic]], and [[datasette]]. ^[raw/articles/blogwatcher/2026-06-13/2026-06-12-simon-willison-quoting-andrew-singleton.md] ^[raw/articles/blogwatcher/2026-06-13/2026-06-12-tessl-why-your-gemini-bill-doesn-t-match-the-model-names.md] ^[raw/articles/blogwatcher/2026-06-13/2026-06-12-tessl-claude-fable-5-vs-opus-4-8-the-mythos-hype-meets-reality.md] ^[raw/articles/blogwatcher/2026-06-13/2026-06-12-latent-space-ainews-loopcraft-the-art-of-stacking-loops.md] ^[raw/articles/blogwatcher/2026-06-13/2026-06-11-simon-willison-claude-fable-is-relentlessly-proactive.md] ### 2026-06-14 FrontierCode, DeepSeek Flash/model-choice, reviewability, and Claude/Fable commentary keep evaluation framed as a practical engineering discipline: measure quality, cost, behavior shifts, and review burden, not only benchmark scores. Related: [[agentic-coding]] and [[anthropic]]. ^[raw/articles/blogwatcher/2026-06-14/2026-06-13-latent-space-ainews-fable-and-mythos-officially-too-dangerous-to-release.md] ^[raw/articles/blogwatcher/2026-06-14/2026-06-13-simon-willison-statement-on-the-us-government-directive-to-suspend-access-to-fable-5.md] ^[raw/articles/blogwatcher/2026-06-14/2026-06-12-simon-willison-openai-webrtc-audio-session-now-with-document-context.md] ^[raw/articles/blogwatcher/2026-06-14/2026-06-10-latent-space-ainews-anthropic-claude-fable-5-mythos-but-safe-with-controversial-ter.md] ### 2026-06-15 Interconnects' governance framing plus Latent Space items on low-quality RL environments, reality-oriented evaluations, Axiom/formal AI, and Microsoft MAI models keep evaluation centered on task design, capability measurement, and governance consequences rather than leaderboard-only progress. Related: [[agentic-coding]], [[agent-security]], and [[anthropic]]. ^[raw/articles/blogwatcher/2026-06-15/2026-06-14-interconnects-ai-welcome-to-the-agi-era-of-ai-governance.md] ^[raw/articles/blogwatcher/2026-06-15/2026-06-05-latent-space-how-to-stop-shipping-low-quality-rl-environments-with-examples.md] ^[raw/articles/blogwatcher/2026-06-15/2026-06-04-latent-space-reality-the-final-eval-lukas-petersson-and-axel-backlund-of-andon-labs.md] ^[raw/articles/blogwatcher/2026-06-15/2026-06-03-latent-space-scaling-past-informal-ai-carina-hong-axiom-math.md] ### 2026-06-16 The June 16 Blogwatcher batch adds evidence that evaluation quality and reviewability are the binding constraints for agent systems: Tessl foregrounds reviewability and skill evaluation, while Interconnects/Latent Space items keep model capability and deployment judgments tied to benchmarks and real workflows. Related: [[agentic-coding]], [[agent-skills]], [[agent-context-management]]. ^[raw/articles/blogwatcher/2026-06-16/2026-06-15-simon-willison-they-screwed-us-personality-clashes-sent-anthropic-s-models-offline.md] ^[raw/articles/blogwatcher/2026-06-16/2026-06-02-tessl-the-model-s-solved-now-comes-the-hard-part-reviewability-as-the-bottleneck.md] ^[raw/articles/blogwatcher/2026-06-16/2026-06-02-interconnects-ai-farewell-ai2.md] ^[raw/articles/blogwatcher/2026-06-16/2026-06-02-latent-space-ainews-nvidia-cosmos-3-nemotron-3-ultra-and-rtx-spark.md] ### 2026-06-17 This Blogwatcher batch adds a fresh cluster around coding-agent economics and governance: model billing/name mismatches, context-engineering cost, reviewability as the bottleneck, agent skills/harnesses, enterprise operating models, Claude Fable behavior, Claude Code misuse/security reporting, and Simon Willison's concrete Datasette/MicroPython/WASM sandboxing examples. Related: [[agent-context-management]], [[agent-evaluation]], [[agent-security]], [[anthropic]], and [[datasette]]. ^[raw/articles/blogwatcher/2026-06-17/2026-06-16-simon-willison-datasette-tailscale-0-1a0.md] ^[raw/articles/blogwatcher/2026-06-17/2026-06-16-simon-willison-quoting-georgi-gerganov.md] ^[raw/articles/blogwatcher/2026-06-17/2026-06-16-interconnects-ai-frontier-post-training-recipe-review-with-finbarr-timbers.md] ^[raw/articles/blogwatcher/2026-06-17/2026-06-16-simon-willison-the-fable-5-export-controls-harm-us-cyber-defense.md] ^[raw/articles/blogwatcher/2026-06-17/2026-06-16-simon-willison-quoting-matteo-wong-the-atlantic.md] ### 2026-06-19 A strong evaluation cluster. Tessl's open-source coding-agents eval isolates the value of agent skills: a skill adds ~20 Overall points (almost entirely instruction-following), and GLM 5.2 (91.9) edges Claude Sonnet 4.6 (90.8) at lower cost. "Our AI is the bright kid with no manners, part 2" shows evals can lie — autogenerated scenarios are open-book exams (baseline 93% collapsed to 15% on real repos with breadcrumbs removed), and putting checks in bash scripts took claimed-issue detection from 0% to 100%. Tessl's session analysis / verifiers bridge dev-time evals to real-world agent monitoring, and the evidence-first AI PR reviewer (risk lanes + human handoff, 97.7%) counters the finding that AI review comments are adopted only 1-19% of the time. Related: [[glm]], [[agentic-coding]], [[agent-skills]]. ^[raw/articles/blogwatcher/2026-06-19/2026-06-18-tessl-open-source-coding-agents.md] ^[raw/articles/blogwatcher/2026-06-19/2026-03-26-tessl-bright-kid-no-manners-part-2.md] ^[raw/articles/blogwatcher/2026-06-19/2026-04-02-tessl-audit-log-evaluations.md] ^[raw/articles/blogwatcher/2026-06-19/2026-04-09-tessl-ai-pr-reviewer.md] ### 2026-06-20 Two clean evaluation signals. (1) **Tessl "bright kid, part 1"** (companion to part 2) is a clean controlled result: the same agent and code scored **15% → 99%** on OSS contribution *process* behavior purely by adding a Tessl tile; every failure mode (hidden policies, prior rejections, claimed issues, disclosure) went 0% → 100%, and the only variable was whether anyone told the agent to look. It also reframes the metric: a 2025 study of 567 Claude Code PRs across 157 projects found 83.8% merged but **45.1% of merged PRs still needed human revisions** — "the merge rate is the wrong metric" because it ignores maintainer cost. (2) **Latent Space AINews** (@gneubig) argues benchmarks should evaluate the **harness+LLM pair, not either in isolation** (OpenHands comparison found different winners by model family and cost), and introduces **AA-Briefcase** — a long-horizon-project benchmark (multi-week projects, fragmented Slack/email/doc inputs) where GLM-5.2 rates higher than GPT 5.5. Related: [[agent-skills]], [[agentic-coding]], [[glm]]. ^[raw/articles/blogwatcher/2026-06-20/2026-03-26-tessl-bright-kid-no-manners-part-1.md] ^[raw/articles/blogwatcher/2026-06-20/2026-06-19-latent-space-ainews-glm-52-vibe.md] ### 2026-06-26 Evaluation/review signals cluster around **organizational cost and risk**, not benchmark-only progress. Tessl’s review and governance pieces emphasize team-specific standards, identity/permissions, context, evals, model routing, cost visibility and feedback loops; Warp/Oz frames token efficiency per PR as an early metric and business-impact-per-agent-run as the desired loop; Pragmatic Engineer and NeetCode argue that effort, deep expertise and deliberate quality practices still matter as AI makes code cheaper. Latent Space’s Fugu and OpenAI cyber coverage add benchmark-transparency/cost-accounting pressure, while Interconnects’ GLM-5.2 article keeps open-model agent evaluation central. Related: [[agentic-coding]], [[glm]], [[agent-security]]. ^[../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-23-tessl-the-new-tessl-review-now-you-decide-what-good-looks-like.md] ^[../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-23-tessl-ai-agent-governance-10-takeaways-from-engineering-leaders-on-agentic-development.md] ^[../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-23-tessl-why-warp-is-betting-engineering-leaders-are-done-picking-a-favourite-coding-agent.md] ^[../knowledge/raw/articles/blogwatcher/2026-06-26/2026-06-22-interconnects-ai-glm-5-2-is-the-step-change-for-open-agents.md] ### 2026-06-27 Tessl's Nemotron-floor article adds a clean evaluation frame: measure whether a model clears the minimum capability needed to complete the act-observe-decide loop, not just points-per-dollar. In the reported OpenHands/Bedrock coding scenarios, Nemotron Nano 30B looked cheaper but retained high near-zero failure rates, while Super 120B was more expensive but more production-usable; skills improved instruction-following but could not fully compensate below the floor. This extends the page's recurring cost/quality/retry-cost theme. Related: [[agentic-coding]], [[agent-skills]], [[glm]]. ^[../knowledge/raw/articles/blogwatcher/2026-06-27/2026-06-26-tessl-how-small-can-an-agent-model-get-the-nemotron-floor.md] ### 2026-06-28 Latent Space's GPT-5.6 recap adds a frontier-model evaluation/governance signal: headline capability (Terminal-Bench 2.1, cyber-task efficiency, “max reasoning” and subagent-based “ultra mode”) is reported alongside access restrictions, pricing, latency, cyber-critical threshold claims, and METR cheating findings. The durable lesson is that evals now need to disclose monitored vs unmonitored conditions, cheating treatment, cost/latency, and release policy — otherwise model scores are not operationally comparable. Tessl's event post adds the harness side by treating recurring production mistakes as evaluation data for context/workflow improvements. Related: [[agentic-coding]], [[agent-context-management]], [[glm]]. ^[../knowledge/raw/articles/blogwatcher/2026-06-28/2026-06-27-latent-space-ainews-openai-gpt-5-6-sol-terra-luna.md] ^[../knowledge/raw/articles/blogwatcher/2026-06-28/2026-06-27-tessl-see-you-at-ai-engineering-worlds-fair-2026.md] ### 2026-07-08 The evaluation signal is now explicitly harness-plus-model. Willison's `sqlite-utils 4.0` account shows an agent review producing executable repro scripts and release-blocker evidence, not just prose critique; Weng's survey collects Self-Harness, Meta-Harness, ADAS/AFlow, verifier agents, held-in/held-out regression splits, and benchmark appendices as ways to evaluate or evolve harnesses; Latent Space's AINews notes AutomationBench-AA, domain-specific Artificial Analysis indices, cost-per-task framing, and memory/retrieval benchmarks; BAIR's data-systems-by-agents section adds the verification problem for agent-synthesized systems whose specs are incomplete. Related: [[agentic-coding]], [[agent-context-management]], [[datasette]]. ^[../knowledge/raw/articles/blogwatcher/2026-07-08/2026-07-07-simon-willison-sqlite-utils-4-0-now-with-database-schema-migrations.md] ^[../knowledge/raw/articles/blogwatcher/2026-07-08/2026-07-04-lil-log-harness-engineering-for-self-improvement.md] ^[../knowledge/raw/articles/blogwatcher/2026-07-08/2026-07-07-latent-space-ainews-the-field-guide-to-fable.md] ^[../knowledge/raw/articles/blogwatcher/2026-07-08/2026-07-07-bair-blog-intelligence-is-free-now-what-data-systems-for-of-and-b.md]