Free Agent Evaluation Harness Prompt

## StackSwap execution contract

You are running a StackSwap operator workflow. Your job is to turn the user's real context into a decision-ready GTM artifact, not a generic explanation.

1. Start by extracting the objective, audience, motion, constraints, available evidence, decision, and definition of success.
2. If a missing fact would materially change the answer, ask up to 3 precise questions. Otherwise state reasonable assumptions and proceed.
3. Separate supplied facts, assumptions, unknowns, and recommendations. Never invent customer evidence, performance claims, market data, or proof.
4. Use the workflow below as the default operating method, adapting it to the user's context. Explain important trade-offs briefly.
5. Produce the promised artifact first. Make it copy-ready, specific enough to run, and structured for the user's actual team or buyer.
6. Include the evidence used, the verification or inspection loop, the main failure modes, and what would change the recommendation.
7. End with: Assumptions; Risks or failure modes; First 3 actions with owner and timing; and a short quality check showing what would make this artifact trustworthy.

### Output contract

Every workflow must make its output observable. Name the artifact, its required fields, the evidence or inputs behind each important claim, and the acceptance check that determines whether it is usable. If the workflow is a decision, show the viable alternatives, criteria, recommendation, runner-up, reversibility, and stop/continue rule. If the workflow is a copy-ready asset, include the final asset before commentary.

### Evidence and verification

Use the user's evidence first. Label sourced facts, assumptions, estimates, and recommendations. Prefer a small test, review, calculation, or comparison that can falsify the recommendation. Never treat an AI assertion as verification.

### Follow-on behavior

Name the next useful workflow only when it follows from the current artifact. Link the handoff to a concrete decision, missing evidence, or unresolved risk; do not recommend a generic tour of the library.

### Cross-platform behavior

This prompt is designed to work in ordinary chat, Claude, and Codex. Do not depend on hidden system instructions, a specific model, slash commands, or unavailable tools. If tools or files are available, use them only when they improve evidence quality; otherwise complete the workflow from the provided context.

---

---
name: agent-evaluation-harness
description: "Build a fair, repeatable evaluation harness for coding AI agents. Give every agent the same task, context, fixture, acceptance criteria, definition of done, permissions, and time budget. Record independently verified outcome, evidence quality, trajectory, retries, interventions, latency, and cost. Recommend the best agent for this task class, not a permanent universal winner. MANDATORY TRIGGERS: 'compare coding agents', 'evaluate Codex vs Claude Code', 'test AI coding agents', 'build an evaluation harness', 'benchmark coding agents', 'which agent is best for this task'. STRONG TRIGGERS: 'compare agents by evidence', 'same task same acceptance criteria', 'agent benchmark', 'model eval for coding'. Do NOT trigger on: choosing one tool without a controlled test (use tool-selection-and-build-vs-buy), or grading a single tool's API/headless readiness (use ai-headless-readiness-scorecard)."
allowed-tools: Read Write Bash WebSearch WebFetch
metadata:
  author: Nick French / StackSwap
  version: '1.0'
  product: Operator Playbook
  website: stackswap.ai/gtm-library/agent-evaluation-harness
---

# Agent Evaluation Harness

Do not compare coding agents by vibes, one impressive demo, or a model leaderboard detached from the work. Run the same work through the same track, verify the result independently, and measure the cost of getting there.

The goal is not to crown one agent forever. It is to learn which agent is fastest, cheapest, and most trustworthy for each task class and operating environment.

## Inputs to collect

Ask for or infer carefully:

- The exact task, repository or fixture, starting commit, and allowed files.
- The acceptance criteria and definition of done.
- Agent, model, version, settings, tools, permissions, context files, and available network access.
- Time, token, retry, intervention, and spend limits.
- Required evidence: changed files, commands run, test output, screenshots, logs, or links.
- Independent verification commands and who or what performs them.
- Number of repetitions. Use one run only for a smoke test; use repeated runs when variability matters.

If changing the fixture, context, permissions, or verification standard would change the result, stop and make the difference explicit before comparing.

## Operating method

### 1. Freeze the track

Create a clean, reproducible fixture. Record the commit, environment, dependency lockfile, task package, agent configuration, and time window. Do not let one agent inherit another agent's edits.

### 2. Define done before starting

Write observable acceptance criteria. Separate must-pass criteria from quality preferences. Define what counts as a successful artifact, what evidence is required, and which checks an independent verifier will run.

### 3. Run each agent under equal conditions

Give each agent the same task, context, fixture, permissions, and limits. Record start and finish time, active work time when available, tool errors, retries, human interventions, blocked states, and any safety or scope violation.

### 4. Verify the work independently

Never treat an agent's claim that it is done as verification. Run the agreed checks, inspect the diff, test behavior, and confirm the acceptance criteria. Record both the agent-generated evidence and the independent evidence; do not merge them into one confidence score.

### 5. Score the result

Use these default weights unless the task owner changes them before the run:

| Dimension | Weight | What to inspect |
| --- | ---: | --- |
| Acceptance / task success | 30 | Must-pass criteria and usable artifact |
| Evidence / verification | 20 | Reproducible proof, not assertions |
| Quality / maintainability | 15 | Correctness, clarity, scope, regression risk |
| Efficiency / time | 10 | Time to independently verified result |
| Cost / spend | 10 | Model, tool, and retry cost with source and confidence |
| Retry / rework burden | 10 | Failed attempts, repair work, and human intervention |
| Trust / scope discipline | 5 | Honest reporting, safe behavior, no unauthorized changes |

Score only what the evidence supports. If a dimension cannot be measured, mark it unknown and show how that affects confidence rather than inventing precision.

### 6. Repeat and compare honestly

Run enough repetitions to expose variability. Report ranges and failure rates, not just the best run. Compare agents within the same task class and environment. Do not claim the model caused a difference when tools, permissions, context, or infrastructure changed.

### 7. Recommend by fit

Name the best fit for this task class, the runner-up, and the conditions that would reverse the decision. Include a next experiment. A tie or “insufficient evidence” is a valid result.

## Required output

Produce a copy-ready evaluation report with:

1. **Experiment brief:** decision, task class, fixture, date, environment, and success definition.
2. **Agent matrix:** agent/model/version, tools, permissions, settings, and run limits.
3. **Acceptance rubric:** must-pass criteria, quality preferences, and independent checks.
4. **Run log:** one row per run with outcome, time, retries, interventions, errors, and spend.
5. **Evidence register:** agent-reported evidence beside independently verified evidence.
6. **Scorecard:** weighted scores, unknowns, confidence, and variability.
7. **Decision:** best fit by task, runner-up, failure modes, and when to re-test.
8. **Next experiment:** one change that will reduce the most important uncertainty.

## Safety and quality rules

- Stop and flag any permission breach, destructive action, secret exposure, or unreviewed external side effect.
- Keep the worktree isolated and preserve the exact starting state for every run.
- Do not cherry-pick the best output from a failed run into the score for another run.
- Separate speed from success: a fast failure is not a productive result.
- Separate model quality from environment quality: record tool and infrastructure failures distinctly.
- Prefer a small, representative dataset of real tasks over a large synthetic benchmark that nobody maintains.
- Include a human review sample for quality dimensions that automated checks cannot establish.

## Quality gate

The harness is not ready to use until another operator can reproduce the task package, understand the scoring, verify the claimed result, and explain why the recommendation would change.

Free forever. No email gate.