Completing the 500-task baseline

I completed the first baseline evaluation of my minimal agent across all 500 SWE-bench Verified tasks. The run used gpt-5.6-luna and retained unsuccessful attempts and evaluation failures alongside successes. This meets my original MP1 goal of accounting for every task, rather than requiring the agent to solve them all.

The final outcomes were 344 resolved (68.8%), 154 unresolved, one empty patch and one evaluation timeout. All 500 tasks are included in that rate. A timed-out evaluation is recorded as an error, not as a confirmed failed repair.

The task list, requested IDs and final scores cover the same 500 tasks. Codex checked the 498 resolved/unresolved flags against the saved harness reports for this write-up. These are inspected results, not an independently rerun evaluation. The full collection includes tasks previously used during development, so the aggregate should not be treated as entirely untouched evaluation data.

I added environment-aware ordering, per-instance scoring and image removal, resumption, and stopping after consecutive errors. This made it possible to run the larger collection without keeping every image on disk and to continue after interruptions.

The evaluation ran over several sessions on October 29. The pilot metadata records a dirty checkout at e9c379d, while later sessions record clean commit 5d65428. The exact initial uncommitted diff was not reconstructed in this review. One failed execution attempt was archived and retried. The result is therefore a completed baseline with documented recovery, not an uninterrupted run with no execution failures.

One patch caused its evaluation tests to time out. Because the harness wrote no report for that task, scoring after each subsequent instance tried it again. The batch was stopped, resumed without per-instance scoring, and the remaining predictions were evaluated separately. The timeout remains in the final results.

This exposed a remaining harness issue rather than invalidating the task accounting. Repeated evaluation of timed-out predictions needs addressing before the later comparisons.

Before the full baseline, I recorded a split of 360 comparison tasks and a 140-task development pool. The earlier development tasks are in the pool. Future intervention design and tuning should stay within that pool.

MP2 now includes preparation for sharded runs and checked score merging. That work is unfinished. Narration-related changes, GEPA integration and the final comparisons remain planned. Completing the baseline does not establish that any of those changes improve results.

I wrote the agent with Claude providing scaffolding, explanations and review. Claude also assisted with checks, running and scoring evaluations, and analysis. This account distinguishes those saved results from the record checks performed for the website update.