Evidence

Every live run, including the failures.

Each row is one run of the agent, made live on GitHub Actions with Nemotron 3 Super on Nebius Token Factory. Nothing is filtered out: runs that never reached a verified fix are listed with the same detail as the ones that did.

—live agent runs

—patches built and checked in a fresh sandbox

—false "booked" confirmations caught

—estimated Nemotron cost for all of them (list price)

Nemotron on Token Factory, by the numbers

From the usage Token Factory returned with every proposal and lookup, summed by scripts/token-stats.cjs. Cost is the traces' estimate at the public list price ($0.30 / $0.90 per million input / output tokens, cached input counted at the full price), not a billed amount.

—proposals and search_code / read_file lookups with recorded usage (2 malformed answers count only in run totals)

—of prompt tokens served from the prompt cache: each attempt resends the growing conversation

—of output tokens were reasoning, kept in the traces for audit and never sent back

—median from the start of an attempt to Nemotron's proposal, including its lookups (— at p90)

—per verified keyboard fix in agents v4–v5 (all runs' cost ÷ verified fixes)

—per run on the seeded phone control

How the agent improved

Each version changed the loop after reading the previous version's failed runs; from v2 on, results are never mixed across versions (v1 groups the pre-versioning runs). Each version ran 5 keyboard and 5 phone runs (v1: 7 each), one square per run below. The keyboard case has two bugs: the typed date is reset by page code, and reopening a booking resets it again. "Saved correctly" counts runs in which some attempt fixed the first.

Results per agent version
Version	Keyboard: verified	Keyboard: saved correctly	Keyboard runs	Phone: first-attempt fixes	Executed / attempts	Cost	What changed

Live agent runs

Show All runs Keyboard (real bug) Phone (seeded control) Verified only

Replay opens the run in the timeline on the home page. Trace is the copy the page replays. Raw is the complete trace with all of its evidence, pinned to the commit that recorded it; the agent column links to the code the agent ran.

Table of live agent runs, newest first
Case	Verdict	Open	Attempts	False "booked" caught	Duration	Cost	Agent	Started (UTC)

Earlier single-shot trials

Before the agent, we measured single-shot, operator-guided repairs: five frozen Nemotron requests on the keyboard case gave 2 of 5 full passes. There the operator supplied the diagnosis and each proposal got one try, with no feedback. Every request, response and execution is kept in evidence/competition-20261021, with a guide to all the evidence in evidence/README.md.