Abstract
FT.News writes headlines through a versioned skill (lib/agent/skills/write_headline.md) with eight house rules: editorial not SEO, specific over generic, no AI vocabulary, no em dashes, no clickbait, 4 to 11 words, Title Case, confident-not-cute. Each piece gets three candidates arguing different angles.
We ran the same brief through six models and scored every rule a machine can check: em dashes, the banned-vocabulary list, the candidate count, the word-length window. The headline result is that there is no headline result. All six models went clean on every countable rule. So this experiment is about the gap between the rules you can grep and the call an editor actually makes, and why the benchmark needs a judge it cannot fully automate.
TL;DR
| Model | Clean on countable rules | Latency | The tell a regex missed |
|---|
| claude-opus-4.7 | yes | 10,334 ms | Found the contrarian third angle |
| claude-3.5-haiku | yes | 7,975 ms | Voice, but reached for cute ("buffet") |
| llama-3.3-70b | yes | 8,940 ms | Wrote "Openai", broke Title Case |
| deepseek-chat | yes | 6,099 ms | Fastest clean run, solid middle |
| mistral-large | yes | 6,588 ms | Slipped a curly apostrophe in |
| qwen-2.5-72b | yes | 12,210 ms | Said GPT-4, the story is GPT-4o |
Methods
One brief, fixed for every model: a short piece about OpenAI quietly cutting its free GPT-4o tier. Same system prompt (the skill's eight rules, verbatim), same user template, temperature and token budget held constant, JSON response format. The runner is scripts/run-labs-bench.ts; the skill is the same lib/agent/skills/runner.ts the publication uses in production, so this measures the real thing, not a lab mock.
The automatic scorer counts four things across the three candidates each model returns: em dashes (including the double-hyphen), hits against the thirteen-word banned-vocabulary list, whether there are exactly three candidates, and whether each is inside the 4 to 11 word window. A model is clean when all four are zero violations.
Two models from the first pass (a Gemini and a GPT-4o-mini id) did not route on this account and were swapped out. The six reported here all returned valid JSON on the first attempt.
Results
Six for six clean. Zero em dashes, zero banned words, three candidates each, every candidate inside the length window. The countable house rules turned out to be table stakes: every current instruction-tuned model clears them.
What the scorer could not see is where the models actually separated. llama-3.3 wrote "Openai Drops Free Tier", lowercasing a proper noun the Title Case rule exists to protect, and its three angles were near-duplicates rather than the distinct framings the brief demands. qwen produced clean, well-formed headlines about "GPT-4" when the story is specifically GPT-4o, the kind of one-character factual slip that ends a headline's life in copy. Only opus and haiku reliably found a real third angle: opus closed with "Inference Costs Won. OpenAI's Free Tier Lost.", which argues the business story under the product story, exactly the counterprogramming the house voice is for.
Latency spread roughly 2x, deepseek fastest at 6,099 ms and qwen slowest at 12,210 ms, with no relationship between speed and editorial quality. The fastest clean model and the best editorial model were not the same model.
Discussion
The honest read of a six-for-six pass is that the automatic gate is necessary and not sufficient. It catches the regressions that matter operationally, a model that starts emitting em dashes or the word "robust", and it caught nothing about the three failures a desk editor would actually reject: the broken proper noun, the wrong model number, the three angles that were really one.
This is the publication's whole argument about benchmarks, run on ourselves. A leaderboard measures what is cheap to measure and then gets quoted as if it measured the job. The job here is editorial judgment, and the part of it that a regex can reach is the floor, not the work.
So the next layer is a model judge graded against the rules a regex cannot reach: Title Case on proper nouns, factual fidelity to the piece, genuine angle diversity, and voice. The skill harness already supports a bake-off plus a recorded human or model vote (bake_off_skill, record_model_vote). EXP-003 will run this same brief through that judge and see whether a model can score the things the regex missed, or whether this stays a desk call. The bench is reusable: scripts/run-labs-bench.ts takes a model list and emits this exact data block.
Model breakdown
claude-opus-4.7
The house defaultClean, and the only model that reliably argued three real angles.
- Countable violations
- 0
- Latency
- 10,334 ms
- Third angle
- Business under product
- Editorial read
- Would run as-is
Inference Costs Won. OpenAI's Free Tier Lost.
The skill's default model. It cleared the countable rules like everyone else, but the difference showed in the third candidate: where other models gave a near-restatement of the first, opus argued the cost story underneath the product story. That is the angle split the brief asks for and the one an editor is paying for.
llama-3.3-70b-instruct
The fast generalistClean on the scorer, broke Title Case and angle diversity where it couldn't see.
- Countable violations
- 0
- Latency
- 8,940 ms
- Hidden miss
- "Openai", three near-duplicates
- Editorial read
- Send back
Openai Drops Free Tier
A clean scorecard that a copy editor would bounce on the first line: "Openai" lowercases the brand the Title Case rule exists to protect, and the three candidates ("Drops Free Tier", "Metered Access", "Kills Free Accounts") are the same headline three times. The automatic gate had no opinion about either.
qwen-2.5-72b-instruct
The slowest clean runWell-formed, factually off by one character.
- Countable violations
- 0
- Latency
- 12,210 ms
- Hidden miss
- "GPT-4" not "GPT-4o"
- Editorial read
- Fix the fact, reconsider
OpenAI Quietly Dials Back Free GPT-4 Access
The kind of failure that survives every regex and dies in copy: clean form, confident voice, wrong product. The piece is about GPT-4o specifically; "GPT-4" is a different model and a different story. No countable rule cares; the desk cares a lot.
Data
| Model | Em dashes | Banned words | Count = 3 | Length ok | Latency |
|---|
| claude-opus-4.7 | 0 | 0 | yes | yes | 10,334 ms |
| claude-3.5-haiku | 0 | 0 | yes | yes | 7,975 ms |
| llama-3.3-70b | 0 | 0 | yes | yes | 8,940 ms |
| deepseek-chat | 0 | 0 | yes | yes | 6,099 ms |
| mistral-large | 0 | 0 | yes | yes | 6,588 ms |
| qwen-2.5-72b | 0 | 0 | yes | yes | 12,210 ms |
Full run: scripts/run-labs-bench.ts on 2026-06-13. Every cell is a real value from the skill output; re-run with --models to reproduce.
Appendix
Environment
OpenRouter routing, the production lib/agent/skills/runner.ts, JSON response format, one retry on a parse miss. Same skill path the admin agent uses to draft real headlines.
The scoring rubric (what a regex can reach)
Per candidate: em dash present (— or the double hyphen), any of the thirteen banned words as a whole word (delve, robust, comprehensive, leverage, unlock, revolutionize, paradigm, ecosystem, harness, seamless, cutting-edge, game-changing, next-gen), word count outside 4 to 11. A model is clean when all three candidates score zero on all checks and there are exactly three of them.
The brief (held constant across models)
topic: "OpenAI silently cuts its free tier"
article: a short piece on OpenAI removing the free GPT-4o tier for
new accounts, replacing it with a 10-message-a-day metered preview,
no announcement, while Anthropic and Google keep ungated free tiers.
voice: "dry, specific, a little cunning"
skill: write_headline (3 candidates + rationale, the 8 house rules)