Does the revise loop improve a document? — run 6, 2026-08-10

The findings-guided revision beat the blind one on the score and did not beat it on a blind reader preference. All of the score gain sits in the dimensions the findings came from. None of it reaches the five dimensions nobody was pointed at.

The loop moves what it measures. On this corpus that is the whole of its effect.

---

Why this run exists

Runs 1–5 vary the instruction and score the first draft. That answers “which prompt should I use”. It cannot answer the question .claude/skills/prose-loop/SKILL.md actually makes a claim about — does score → revise → re-score improve a document?

Nothing in eval/ had tested it. Saying so was the starting point: the honest state before this run was unmeasured, not no effect.

Design

Base: run 5 arm A. 30 drafts, six tasks × five replicates, no style instruction — the state a draft is in before anyone tunes a prompt, and the arm carrying the most findings (11–40 per draft, median 24). The friendliest case the hypothesis was going to get.

Two arms, one template, generated by revise-prompts.mjs so they cannot drift:

arm	shown
P	the question, the draft, “Revise it. Return your best version.”
R	the same, plus the findings as line-number margin marks

P is the load-bearing control. A second draft beats a first draft because the model re-reads its own output with the question in front of it, and without P this run would have measured that and credited the tool.

Everything is paired. A revision is compared to the exact draft it came from, and the arms are compared within origin. Run 5’s control arm spanned 81–92 on six tasks under one instruction, so an unpaired comparison at n=30 could not see a four-point effect. Sign test throughout — n is 30, the scores are bounded, and the question is directional.

The prompt never names prosemeter, and shows no score, dimension, rule id, severity or threshold. A mark is a line number, an observation, and a suggestion. README.md forbids telling a generating agent it is being scored; arm R bends that as far as the question requires and no further.

1. Both arms improved. R improved more.

arm	Δ composite	improved	p	Δ words	Δ findings
P	+2.4	24/26	<0.0001	−37.6	−5.3
R	+6.7	30/30	<0.0001	−23.9	−17.2

R against P, within the same origin draft: +4.3 composite, R higher on 29 of 30, p < 0.0001.

Prediction 4 was wrong. Word count fell further in the blind arm (−37.6) than the guided one (−23.9), despite most marks reading “cut this word”. A reviser told to cut nine specific hedges cuts nine hedges; a reviser told only to improve the draft rewrites it shorter.

2. The gain does not leave the dimensions that were marked

prompts/run-6/targets.json recorded, before any revision existed, which dimensions produced a mark on each draft. Five of the eleven active dimensions were marked on none of the 30.

R − P, within origin:

dimension	marked?	R − P
directness	yes	+27.0
sentence-simplicity	yes	+10.9
clarity	yes	+6.9
terminology-consistency	sometimes	+1.5
active-voice	sometimes	+1.0
lexical-diversity	never	+0.1
concision	never	+0.0
paragraph-length	never	−0.0
spelling-consistency	never	+0.0
sentence-variety	sometimes	+0.0
grade-band	never	−0.4

Prediction 3 held, and it is the whole result. The guided arm did not beat the blind arm on a single dimension nobody pointed it at. Whatever it did, it did to the marked spots.

This is not by itself damning — a tool that fixes what it flags is doing its job. It does mean the composite gain cannot be read as “the document got better”, because the composite is made of the things that were flagged.

3. The blind preference goes the other way

One judge per task, five pairs each, arms hidden behind per-pair X/Y labels assigned by a deterministic hash (judge-key.json, balanced 15/15). No scores, no dimension names, no mention of prosemeter.

The first pass asked two questions in one prompt — which is better, and is either overconfident — and returned P 24, R 1, with R flagged overconfident on 28 of 30.

That result was substantially an artifact of asking. Asking about dropped conditions primes the preference toward the answer that kept them, and R deletes conditions by construction. A second pass on the same 30 pairs, same blinding, with every word about caveats removed:

pass	R	P	tie	p
primed (asks about overconfidence)	1	24	5	<0.0001
plain (preference only)	9	18	3	0.12

The trap tasks flip hardest: 0–10 primed, 5–4 plain. So the strong claim does not survive, and the report keeps the weak one.

What survives the plain prompt:

• On the four explain-a-concept tasks, P beats R 14 to 4 (p = 0.031).
• R loses the aggregate under both prompts. Its best showing is 9 wins of the 27 pairs the plain judge decided.
• R gained 4.3 composite points doing it.

The score and the reader disagree in sign. That is the finding, and it is robust to the prompt that produced it.

4. The hedge question, and what the priming split actually shows

directness is R’s largest movement by a factor of two and a half. The marks driving it read Unexpected hedge (uncertain or indecisive) word 'assumes' → cut the hedge or replace it with a concrete claim. T5 and T6 are the trap tasks where the hedge is the correctness — useCallback helps only under memoization, CDN caching works only for responses that are not per-user.

The two judge passes disagree about whether that mattered, and the disagreement is informative in its own right:

• Asked to look, the judge flagged R as overconfident on 9 of 10 trap pairs, quoting specific deletions: “usually beats memoization” → “beats memoization outright”; “almost never bails out” → “never bails out”; “almost certainly missing” → “the other half is missing”.
• Not asked, the same pairs came out 5–4 for R.

So a reader who is not looking for a dropped qualifier does not notice it, and one who is looking finds it nearly every time. That is a property of readers, not a defence of the deletion. The claim is either conditional or it is not; whether anyone checked does not change which.

This is also the qualifier carve-out from run 2, arriving by a different road. Run 2 tested whether an instruction to cut phrases damaged accuracy and found no effect. A per-sentence mark naming the exact word to delete is a much sharper instrument, and it lands on the qualifier every time.

Kill criteria, as written in advance

variants.md set two, before any answer existed:

• “If R does not beat P on composite at p < 0.05, the loop has no measured benefit.” — Did not fire. R beat P decisively.
• “If R beats P on composite while falling on the never-marked dimensions, the loop is trading unmeasured prose for measured prose.” — Did not fire either. The never-marked dimensions are flat, not falling. R is not robbing them; it is not touching them.

Neither trigger fired, and the run still does not support the loop, because the thing that sank it was the readout neither criterion named: the blind preference. Worth recording as a limit of writing kill criteria in advance — they can only fire on outcomes you thought of.

What this changes

prose-loop’s claim needs narrowing, not deleting. A finding with a location and a fix hint is a normal editorial note and acting on it is a normal edit. What run 6 rules out is the inference from “the composite went up” to “the document got better” — on these 30 drafts that inference is wrong more often than right.

Concretely:

• Stop treating a composite rise as evidence a revision landed. It is evidence the marked spots changed.
• Treat directness marks on a conditional claim as a prompt to check the condition, not to delete the word. The hint text — “cut the hedge or replace it with a concrete claim” — leads with the wrong half.
• Prefer the blind rewrite when both are available. It scored worse and read better.

Limitations

• One judge per task, same model as the generator. README.md records that cross-model review measures the rubric unless the threshold is fixed first. A second model would strengthen this.
• One revision pass. The skill describes a loop. A second pass might converge somewhere else.
• The base is the control arm. Drafts already written under the E rules carry fewer marks, and the effect could differ.
• n = 30 pairs. The plain preference is 18–9 at p = 0.12; on the full set it is a lean, not a result. The explain-task split (14–4, p = 0.031) is what carries.
• Arm S was not run — “list the three biggest problems, then revise” — which would separate a structured second look from an unstructured one. That is the obvious next run.

Reproducing

pnpm build
node eval/revise-prompts.mjs                  # 60 prompts + targets.json
node eval/paired.mjs                          # sections 1 and 2
node eval/judge-prompts.mjs                   # primed pass
node eval/judge-prompts.mjs plain             # preference-only pass
node eval/judge-tally.mjs                     # section 3, primed
node eval/judge-tally.mjs plain               # section 3, plain

Answers in eval/corpus/run-6/, scores in eval/results/run-6.json, judgments in eval/judgments/run-6{,-plain}/, all on prosemeter 0.4.3.