First batch results and an exit-rule experiment

I completed six evaluation batches on the same 25 SWE-bench Verified tasks, using gpt-5.6-luna. Four batches used the baseline prompt and two used a stronger exit rule. This gives me a first multi-task development baseline, with unsuccessful attempts retained alongside the successful ones. It is not a complete SWE-bench evaluation.

The 25 tasks were sampled with seed 1234, excluding the SymPy task I had already used repeatedly. The baseline batches resolved 14, 16, 18 and 16 tasks out of 25, for 64 resolved attempts out of 100. These are repeated attempts on 25 distinct tasks, not 100 different tasks. The variation between runs matters when judging a small change.

The first batch exposed a patch-capture bug. Some benchmark images already contained changes on top of the task’s base commit, and my patches included those changes. Capture now compares against the commit recorded at container startup, preserving the agent’s edits without including the image’s existing changes. The first batch’s scores were retained after checking the harness’s patch-application fallback. The three baseline batches after the fix resolved 50 of 75 attempts.

The variant asked the agent to run the repository’s existing tests for the changed code before exiting, with a hint to look for the project’s own test runner. It resolved 16 of 25 tasks in each of two batches, or 32 of 50 attempts. One attempt submitted an empty patch. The remaining 17 were unresolved.

The variant’s aggregate resolution rate was 64%, matching the four-batch baseline. The trajectory analysis found more test-running, but these small repeated batches did not demonstrate improved resolution. I kept the variant separate from the default configuration.

One variant attempt put its proposed edits into replies with multiple action blocks, which the parser rejected. It later ran tests on unchanged code and submitted an empty patch. Requiring a passing test run did not establish that the intended edits had actually executed.

Incomplete fixes also remain a problem. Six tasks were unresolved in every batch. Examining the hidden test failures helped diagnose missing cases, but it also means this task set must be treated as development data. Later improvement claims need separate evaluation tasks.

All 150 requested attempts have score records. The 149 nonempty patches have resolution outcomes and repair/regression test lists matching their saved SWE-bench harness reports. The remaining attempt has an empty-patch outcome. None of these batches has an agent-error, failed-capture or evaluation-error score.

I developed the agent with Claude providing scaffolding, explanations and review. For this stage, Claude also assisted with running and scoring batches and analysing failures. Codex checked the saved score records against the task list and harness reports for this write-up, without rerunning the evaluations.

The full benchmark evaluation remains pending for MP1. I am treating improvements to the baseline, including investigating GEPA, as MP2. That separates building and evaluating the initial agent from testing changes to its behaviour and results. GEPA is planned work, not an implemented improvement.