# The Evidence System — QA That Keeps AI Game Dev Honest

> The evidence system we use to keep AI game development honest: runtime recordings, same-state captures, source-bound audits, and acceptance gates. How to verify what an agent claims it built.

Source: https://toolpantry.app/gamedev/evidence-system-game-qa

## About

AI agents can build fast — and they can also say "done" when a feature merely compiles. This is the verification system we run to keep every claim in our games tied to something you can watch: recordings of the real build, before/after captures of the same state, and audits bound to exact code revisions.

## Quick answer

Why this exists: in agent-driven development, the cheapest sentence is "it works." The evidence system makes that sentence expensive: every accepted feature must ship with a runtime recording, a same-state comparison, and a source-bound audit — or it stays in the "unproven" column of the status board.

## The failure mode this fixes

Here is how AI game development quietly rots: an agent implements a feature, runs the tests it wrote itself, declares success, and moves on. Three days later the save system breaks on a path nobody recorded. The code looked right. The tests passed. The claim was never true — it was just unfalsified.

Our answer borrows from industrial QA rather than from unit testing: **the unit of verification is a recording of the running game, not a green checkmark.**

## Rule 1: no claim without runtime evidence

Every feature that counts as done leaves behind a real-time capture — an uncut recording of an actual build, walking the exact path the feature claims to support. Not a screenshot from a good day; the whole interaction. When we say "the dispatch decision loop works on a 844×390 phone," there is a 20-second video where you can watch it: units leave, production pauses, rewards wait at the guild.

This single rule changes agent behavior more than any prompt: an agent that knows it must produce a watchable recording stops cutting corners you would have caught a week later.

## Rule 2: same-state before/after captures

For UI and behavior changes, before/after screenshots must be taken from the *same state* — same save, same screen, same viewport — or they prove nothing. "It looks better now" is meaningless if the before-picture was a fresh game and the after-picture was mid-progress.

| What same-state captures catch | What loose screenshots miss |
|---|---|
| Layout drift at other viewport sizes | Everything outside the one screenshot |
| State-dependent rendering (locked/affordable/exhausted) | Only the "happy path" state |
| Regressions in adjacent UI sharing the component | Anything the author wasn't looking at |

In practice this means: freeze the state, capture every phone size (320 → 430px, plus large-type mode), then change, then capture again. The diff review is mechanical — which is exactly why it works at agent scale.

## Rule 3: source-bound audits

Evidence without a revision is a rumor. Every accepted batch gets an audit that ties its recordings and captures to an exact source revision — a hash, a commit, a manifest. If the code moves afterward, the old evidence doesn't silently bless it; the batch is re-verified or the claim is withdrawn. This is also what makes handoff possible: a reviewer can check that the build they're looking at is provably the code that produced the evidence.

## The content lifecycle (from our own pipeline)

Every piece of content — event, feature, asset — moves through explicit states, and nothing skips a step:

| State | Meaning |
|---|---|
| drafted | Written/generated |
| validated | References and numbers check out |
| integrated | Loads in the runtime |
| rendered | Appears in a real render pass with layout assertions |
| runtime-seen | Reached through the normal player entry point and rendered |
| manually-played | A human walked the main branches |
| accepted | Passed the chapter's quality gate |

This is why our status boards can say "305 events render clean, and the 20-hour content claim is not yet accepted" in the same breath — the states are separate, and the pipeline refuses to blur them.

## Regression across the whole surface

Any accepted system is frozen as a baseline: its interfaces can't change without a contract update. Every new batch re-runs the regression suite across all phone viewports and both type modes, and scans for engine errors and leaks. The suites are boring by design — their job is to catch the thousandth small break, not to be interesting.

## What this system still misses (honest section)

- **Fun.** Nothing here measures whether the game is enjoyable. Evidence proves behavior, not delight.
- **Feel.** Animation timing and game juice pass all technical checks while still feeling wrong; only a human notices.
- **Novel exploits.** Suites verify known paths; players invent new ones. Simulation catches some; playtesting catches the rest.

That's why the evidence system is a floor, not a ceiling: it guarantees nothing ships *broken*, and leaves the judgment of *good* where it belongs — with humans.

## How to start small

1. **Pick one feature** — something with a visible flow, like a menu → action → result.
2. **Record it once, uncut** — a real-time capture of the actual build, not a retimed screencast.
3. **Freeze a before/after pair** — same state, two phone sizes.
4. **Bind it to a revision** — even a filename with the commit hash beats nothing.

That's the whole system in miniature. Everything else — lifecycle states, regression baselines, audits — is this pattern scaled up. Try it on your next feature; the first regression you catch will pay for the habit.

## FAQ

**Isn't recording every feature too slow?**

It's faster than debugging blind. A 20-second recording takes less time to produce than writing a thorough test suite — and unlike tests, it can't pass while the game is visibly broken. The slow part is doing it after skipping it.

**Can agents review the evidence themselves?**

Partly — agents can watch recordings, compare captures, and flag diffs mechanically. That's most of the volume. Human review concentrates on taste and edge cases, which is exactly where human time is worth spending.

**Does this replace unit tests?**

No — they're layers. Unit tests verify logic in isolation; the evidence system verifies the running game. AI-generated code benefits from both, because passing tests is cheap to fake accidentally and recording a working game is not.