Preparing batch runs and distinguishing failures

I finished the sequential batch runner and extended the scoring records to distinguish agent crashes from other failures. The next step is a small baseline batch on additional tasks. That evaluation has not run yet.

Recording what a batch was meant to do

The batch command accepts task IDs directly or from a file, checks them before starting a container, and runs each task into a shared folder. It rejects empty lists, duplicates and unknown IDs rather than discovering a typo after earlier tasks have already consumed time and model calls.

The run metadata now saves the requested IDs in order. Each completed attempt gets its own trajectory and prediction entry. Comparing those records with the requested list makes it possible to identify tasks a stopped batch never reached. Single-task and batch execution share the same run-and-save helper.

Checking that one crash does not stop the next task

Claude checked the batch runner using three real task IDs, Docker and a substitute agent that raised ConnectionError on the second task. The third task still ran, and the shared folder retained three trajectories and prediction entries. No model API calls were made.

All three checks reused the local SymPy image to avoid downloading additional images. That meant patch capture could not find the other tasks’ starting commits. This was a controlled check of batch execution, recording and cleanup, not three correctly configured benchmark attempts. I implemented the code with Claude supplying explanations, scaffolding and review.

Separating failure status from a possible cause

The score summary now has an agent_error status, so a crashed run is not confused with failed patch capture. It also retains the harness’s infrastructure and ambiguous-failure labels alongside the outcome and reason code. Those labels are diagnostic clues from log patterns, not a definitive explanation of every failure.

Claude checked the classifications using copies of existing reports edited to represent infrastructure failures, ambiguous failures, empty patches and agent crashes. These cases validate how my summary reads the reports. They do not show that I encountered or recovered from those infrastructure failures in real benchmark runs.

What remains before a baseline

The real evaluation evidence is still the earlier 13 attempts on one SymPy task: four resolved, four unresolved and five empty patches. The new batch and classification checks do not add to that performance result.

The next step is to choose and record a small set of additional tasks and the run conditions, then run and score the baseline before experimenting with a stronger exit rule. Automatic resumption and image cleanup remain deferred. A complete benchmark evaluation is still ahead.