Yousuf Golding
← All work notes

Project overview · Baseline evaluation complete

MP1 · Minimal software-engineering agent

A tutorial-based software-engineering agent with a completed baseline evaluation across all 500 SWE-bench Verified tasks.

Last updated

Continues in MP2 · Improvements & GEPA →

Based on the minimal-agent tutorial and mini-swe-agent reference. I implement the code with Claude providing scaffolding, explanations, review, and some checks.

Goal

Build and understand my own agent, then evaluate it across a chosen SWE-bench collection, accounting for successful attempts, unsuccessful attempts, and execution failures.

Implemented

Single-task and resumable batch execution, environment-aware ordering, per-instance scoring and image cleanup, saved trajectories and patches, and explicit failure outcomes.

Results so far

The full baseline resolved 344/500 tasks (68.8%), with 154 unresolved, one empty patch and one evaluation timeout. All tasks remain in the denominator. This is one baseline, including previously used development tasks, not evidence of improvement. The run used resumed sessions and one retry, with session-specific code provenance described in the work note.

Next milestone

MP1’s original evaluation goal is met. MP2 continues with comparison preparation, remaining harness issues and planned agent improvements. GEPA is not implemented, and no comparative improvement result is claimed.

Project history

Dated accounts of the work. Earlier notes describe the project at that point in time.

  1. Completing the 500-task baseline

    Accounting for all 500 SWE-bench Verified tasks, recovering a long run, and identifying a scoring-timeout issue before the next comparisons.

  2. First batch results and an exit-rule experiment

    Six batches on 25 development tasks, a patch-capture correction, and an exit-rule experiment that did not demonstrate improved resolution.

  3. Preparing batch runs and distinguishing failures

    Validating task lists, keeping a batch moving after a recorded crash, and preserving failure labels before the first baseline batch.

  4. Keeping the record when an agent run fails

    Preserving failed attempts and partial trajectories, then preparing the next step toward batch execution.

  5. Scoring the first saved agent patch

    Adding a separate evaluation command, preserving distinct failure states, and getting the first resolved result from the SWE-bench harness.

  6. Finding premature exits in the saved agent traces

    Recording individual model messages, guarding against an exit before execution, and checking what a single-task result actually establishes.

  7. Keeping the patch and the record of a run

    Capturing changes before container cleanup, saving trajectories, and finding a failure in the final summary.

  8. Moving run settings into one configuration

    Loading configuration once, keeping prompt values consistent with execution, and separating build history from current code.

  9. Building the execution environment for my minimal agent

    Separating command execution from the agent loop, working through timeouts and cleanup, and checking the single-task runner.