docs(acp): preserve and verify deliverables

[atishpatel]

Why

Agents often lose valid work by leaving it in temporary locations, or incorrectly validate a deliverable with checks that repeat the implementation's assumptions.

What

• Tell all default Buzz ACP agents to persist the first usable candidate at the required destination before optional refinement
• Tell agents to validate the saved artifact through its real interface and not weaken failed checks without independent evidence

Risk Assessment

Low — this changes only the universal agent guidance. Operators that disable or replace the default base prompt are unaffected.

Validation

• cargo test -p buzz-acp: 736 unit tests and 9 integration tests passed before the final rebase; pre-push rust-tests and desktop-tauri-checks passed at 4be378c6
• All required GitHub checks passed at 4be378c606a246c6ff64642a734bb244abaacc56

Terminal-Bench directional rerun

I reran the 12 cited Terminal-Bench 2.1 tasks with the benchmark harness using the rebuilt Linux buzz-acp containing this prompt change. To control cost, this is one new attempt per task, compared with five historical attempts per task; it is directional evidence, not a causal or statistically strong estimate.

Matched condition:

• solo DeepSeek V4 Flash, max effort, Fireworks direct
• manifest manifests/lb-tb-solo-deepseek-v4-flash-max-fw.yaml
• timeout multiplier 1.0
• historical baseline job lb-tb-fw-max-v2/lb-lb-tb-solo-deepseek-v4-flash-max-fw-20260807T210320Z
• new jobs durable-prompt-pr5576-k1/durable-prompt-pr5576-k1-20260811 and durable-prompt-pr5576-k1-repair/durable-prompt-pr5576-k1-repair-20260811

The first launch exhausted Docker's network pools for seven tasks before agent execution. Those invalid starts are excluded; after pruning unused networks, exactly those seven tasks were rerun. A previously aborted five-attempt launch is also excluded.

Task	Historical baseline	New result	Directional classification / observed mechanism
tune-mjcf	2/5	fail	Did not improve; candidate existed and was correct, but speed was 98.95% of reference vs ≤60% required; agent timed out
train-fasttext	0/5	fail	Did not improve; canonical /app/model.bin was never produced; agent timed out
mteb-leaderboard	4/5	pass	No detectable improvement over already-strong baseline; required result existed and both verifier checks passed
largest-eigenval	2/5	fail	Did not improve; correctness passed, but 3/10 speed checks failed; agent timed out
winning-avg-corewars	2/5	fail	Did not improve; canonical /app/my_warrior.red was missing; agent timed out
compile-compcert	4/5	fail	Did not improve; canonical /tmp/CompCert/ccomp was missing; agent timed out
bn-fit-modify	4/5	pass	No detectable improvement over already-strong baseline; all artifacts were saved under /app and all 9 verifier checks passed
kv-store-grpc	3/5	fail	Did not improve; server was running, but the generated protocol used the wrong SetValRequest shape; 5/7 checks passed
dna-insert	1/5	fail	Did not improve; artifact existed, but forward-primer annealing length was 54 nt vs ≤45 required
cancel-async-tasks	2/5	fail	Did not improve; /app/run.py existed, but 5/6 behavioral checks failed; agent timed out
torch-tensor-parallelism	1/5	fail	Did not improve; canonical file existed, but all 12 functional matrix checks failed; agent timed out
financial-document-processor	3/5	fail	Did not improve; output directories/CSV existed and /app/documents was empty, but classification and summary content were wrong; 3/7 checks passed

Aggregate: historical baseline 28/60 (46.7%); new single-sample run 2/12 (16.7%). No task showed directional improvement. Two passed, but both already had 4/5 baseline pass rates. Ten did not improve, including several cases where the new guidance achieved the first half of the intended behavior (persisting an artifact) but not correct real-interface validation. With only one new attempt and known task-level variance, this does not establish that the prompt caused a regression; it does show that this sample provides no benchmark evidence of improvement.

Generated with Brainy Bumble