We gave two models the same 8-second job. One never finished a sentence.

Abstract

The Daily Combinator (/play) resolves unknown word pairs with a live LLM call. The call runs under a hard 8-second budget because a player is standing there holding a tile. When the call wins, the player gets a new word with an emoji and a one-liner. When it loses, they get a consolation card that says StillCooking.

Production routed that call to the same model the publication uses for drafting articles: a large reasoning model. Players got StillCooking one hundred percent of the time, and the nightly cache-warming job burned two and a half minutes producing nothing. This experiment documents the failure, the swap, and what the numbers say about matching models to jobs instead of ranking them on vibes.

TL;DR

Methods

The task: given two normalized words (say, telescope + burrito), return strict JSON with a combined result, one emoji, an explanation between 10 and 200 characters, a rarity, and a category. The prompt is the production game prompt, verbatim, reproduced in the appendix. Temperature 0.7, max 300 tokens, response_format json_object.

The budget: 8,000 ms wall clock, enforced by an AbortSignal in lib/game/ai.ts. A miss is not retried in the moment; the player sees the fallback and the pair stays eligible for the next attempt.

Two call sites exercise the task. Live combines fire when a player merges a pair no one has tried. The pre-generation job (game_pregen_runs) warms the twenty most likely next pairs in one nightly pass, sequentially, same prompt, same budget.

Scoring is binary per call: a validated JSON object cached into game_ai_results counts as a completion. A timeout, an empty message, or JSON that fails the zod schema counts as nothing, because that is what the player receives: nothing.

Results

The reasoning model never produced a single cacheable result. Twenty warm-up attempts consumed 2 minutes 36 seconds, almost exactly twenty stacked 8-second timeouts. A live player test confirmed it end to end: 8.3 seconds of waiting, then the fallback card.

After the swap, the same job on the chat model cached 63 results with a median of 2,263 ms and a p90 of 5,108 ms. The slowest call ever recorded was 6,810 ms, still inside the budget. Warm runs went 17, 18, and 19 for 20; the misses were occasional malformed JSON, not timeouts.

The fallback chain added a footnote: deepseek-v4-flash caught two calls at a median of 5,386 ms. Inside the budget, but spending two thirds of it.

Discussion

Nothing here says the reasoning model is bad. It says the job was wrong for it. A model that spends its token budget thinking is a fine choice for drafting a 900-word article overnight and a catastrophic choice for an 8-second party trick, and no leaderboard score communicates that difference.

The fix was architectural, not editorial: per-surface model knobs. The game pins GAME_OPENROUTER_MODEL, the curation scorer pins SCORING_OPENROUTER_MODEL, and the drafting desk keeps whatever the publication wants for prose. The same incident repeated three times in one week, in three disguises (game combines, the daily theme generator, pipeline scoring), and the same one-line fix closed each one.

The general claim we will keep testing: model selection is a per-surface product decision with a latency budget attached, and the budget is part of the benchmark. A follow-up experiment will run our coding tasks and game simulations across models under explicit budgets, in this same format.

Model breakdown

Data

Source tables: game_pregen_runs, game_ai_results, plus the production runs ledger. Cached rows are inspectable in the admin moderation queue.

Appendix

Combine "{a}" + "{b}" into one new thing.
Audience: millennial women 27-42. Relatable, current,
clever-not-try-hard. No corporate buzzwords, nothing mean.
Return JSON: { "result", "emoji", "explanation",
"rarity": "Common|Uncommon|Rare|Legendary", "category" }
temperature 0.7 · max_tokens 300 · response_format json_object
budget 8000 ms (AbortSignal)