Work note
Project overview · MP2 · Agent improvements · Work through
What happened when the agent accepted the first action block
When the model returned several shell blocks in one reply, the baseline rejected the entire reply. An edit could be written out in full and never reach the working tree. I tested a rule that runs the first block and tells the model that the remaining blocks were dropped. A second rule refuses to exit with an empty diff.
Counting lost steps
Before changing the parser, I chose a measure I could check in the saved trajectories: the share of steps lost to format errors or premature exits. I ran the baseline and the modified agent twice each on the same 25 development tasks. Across the baseline runs, 85 of 488 steps were lost (17.4%). With the new rules, 33 of 456 were lost (7.2%). Accepting the first block meant more replies could produce an action even when the model still sent several blocks.
Did more tasks get solved?
The baseline solved 16 and 18 tasks across its two runs. The modified agent solved 18 both times, for 36 of 50 attempts compared with 34 of 50. The two-attempt difference was as large as the variation between baseline runs. That leaves the effect on repair success open. None of the 50 modified-agent attempts triggered the empty-diff exit rule, so these runs tell me little about that part of the change.
Decision
I froze this configuration for the later comparison on 360 held-out tasks. The development runs gave me a reason to keep investigating the parser change and a separate question for that comparison: does losing fewer steps help the agent finish more repairs?