Yousuf Golding
← All work notes

Project overview · Development comparison complete

MP2 · Agent improvements & GEPA

Investigating changes to the agent built in MP1, with the baseline as the point of comparison.

Last updated

Builds on MP1 · Baseline agent →

MP2 continues the same agent implementation. MP1 retains its completed full baseline and work history. Claude continues to provide scaffolding, explanations and review during implementation.

Goal

Investigate which changes improve the agent’s results, what they cost, and where they still fail.

Starting point

MP1 provides the agent and evaluation pipeline, including a full 500-task baseline. A recorded split separates 360 comparison tasks from a 140-task development pool containing the previously used tasks.

Implemented preparation

Environment-aware task splitting and checked score merging are implemented. Claude exercised the complete pipeline with seven controlled task outcomes across two shards. The runner now avoids repeatedly scoring tasks with evaluation logs but no report, and merged records retain shard directories. These checks do not establish benchmark performance.

Development results

Running the first action block in a multi-block reply helped reduce the share of lost steps from 17.4% to 7.2%. The modified agent resolved 36 of 50 attempts against 34 of 50 for the baseline, within the variation between baseline runs.

Evaluation

Keep improvement design and tuning within the development pool, preserving the comparison set. The configuration and analysis rules are fixed for the later comparison. GEPA optimization, a real shard timing pilot and the held-out comparison are still ahead.

Project history

Dated accounts of the work. Earlier notes describe the project at that point in time.

  1. What happened when the agent accepted the first action block

    Two runs per configuration on the same 25 development tasks showed fewer rejected steps and similar repair counts.

  2. Checking the shard pipeline and stopping repeated scoring

    A controlled two-shard pipeline check exposed repeated timeout evaluations and ambiguous run IDs, leading to scoring and record-keeping fixes.

  3. Splitting benchmark work and checking merged scores

    Implementing environment-aware task splits and score-merging checks before trying concurrent benchmark runs.