Checking the shard pipeline and stopping repeated scoring

I completed the controlled check connecting task splitting, batch execution, scoring and merging. I also updated the baseline runner so a failed evaluation is not repeatedly attempted after each later task. These changes support the upcoming MP2 comparisons without changing MP1’s recorded baseline result.

Claude ran seven development-pool task IDs through two simultaneous shards using the real batch, scoring and merging code. Agent execution, the evaluation harness and image operations were replaced with controlled substitutes. The merged records matched the planned outcomes of three resolved tasks, one unresolved task, one empty patch, one agent error and one evaluation error. These were simulated outcomes, not benchmark performance results.

The check also exposed a record-keeping issue. Shards started in the same second can have the same timestamp-based run ID. Merged results now retain each shard’s directory alongside its run ID. Each shard needs a separate output directory to avoid a run-folder collision.

Previously, a task whose evaluation timed out could be retried after every later task because it had no completed report. The runner now distinguishes a prediction with no evaluation log folder from one with a log folder but no report. It scores the former during the batch and leaves the latter for a separate scoring pass. Reading this state from disk also preserves the distinction when a batch resumes.

In Claude’s controlled check, the simulated timeout was evaluated once during the batch rather than four times. The separate scoring command retried it once at the end, as intended. Empty patches did not start the harness. This verifies the controlled case, while real Docker execution remains to be checked.

Stop messages now show the error type and all nonempty lines of its message, keeping the cause of a Docker failure visible rather than showing only its introductory line. A resumed session with no remaining tasks reports that nothing ran. Resume output also identifies evaluations with no report that remain for the separate scoring command.

The written plan specifies fresh baseline runs alongside the intervention runs, paired comparisons on the same tasks, and separate accounting for evaluation errors. A second baseline will help describe run-to-run variation. The earlier 500-task baseline remains context rather than the comparison baseline.

The plan also records when an additional run is allowed and how uncertainty will be reported. The conditional repeat rule and unadjusted primary tests have statistical limitations, which the plan requires reporting. This is an analysis plan, not an implemented analysis script or a validated improvement claim.

The full pipeline has now been exercised with substitute execution. Real concurrent shards still need a timing and resource-contention pilot. The comparison analysis rules are now recorded before intervention results. The next implementation step is measuring narrated actions and rejected replies in development trajectories. Agent-behaviour changes and GEPA remain planned.

I implemented the changes with Claude providing scaffolding, explanations, review and controlled checks. Codex inspected the code changes and recorded verification for this note without rerunning the checks. The existing 344/500 baseline is unchanged.