Completing the 500-task baseline
Accounting for all 500 SWE-bench Verified tasks, recovering a long run, and identifying a scoring-timeout issue before the next comparisons.
Read note : Completing the 500-task baselineYousuf Golding
Work notes
Accounts of what I am building, the decisions along the way, and what the results show.
Project · Baseline evaluation complete
MP1 · Minimal software-engineering agent
A tutorial-based software-engineering agent with a completed baseline evaluation across all 500 SWE-bench Verified tasks.
Current stage: Original evaluation goal met. The full baseline is recorded, while improvements and GEPA belong to MP2.
Continues in MP2 · Improvements & GEPA →
View project overview →Project work notes
Dated entries about individual work sessions.
First batch results and an exit-rule experiment
Six batches on 25 development tasks, a patch-capture correction, and an exit-rule experiment that did not demonstrate improved resolution.
Read note : First batch results and an exit-rule experimentPreparing batch runs and distinguishing failures
Validating task lists, keeping a batch moving after a recorded crash, and preserving failure labels before the first baseline batch.
Read note : Preparing batch runs and distinguishing failuresKeeping the record when an agent run fails
Preserving failed attempts and partial trajectories, then preparing the next step toward batch execution.
Read note : Keeping the record when an agent run failsScoring the first saved agent patch
Adding a separate evaluation command, preserving distinct failure states, and getting the first resolved result from the SWE-bench harness.
Read note : Scoring the first saved agent patchFinding premature exits in the saved agent traces
Recording individual model messages, guarding against an exit before execution, and checking what a single-task result actually establishes.
Read note : Finding premature exits in the saved agent tracesKeeping the patch and the record of a run
Capturing changes before container cleanup, saving trajectories, and finding a failure in the final summary.
Read note : Keeping the patch and the record of a runMoving run settings into one configuration
Loading configuration once, keeping prompt values consistent with execution, and separating build history from current code.
Read note : Moving run settings into one configurationBuilding the execution environment for my minimal agent
Separating command execution from the agent loop, working through timeouts and cleanup, and checking the single-task runner.
Read note : Building the execution environment for my minimal agent