The Real Polish

Last week I told you polish is the trap. This week, the real polish.

When optimization IS warranted (the existential-constraint kind from last week), there’s a way to do it that doesn’t end in shrugs. Most teams skip the methodology and call the result unsolvable.

The phrase I hear from executives (outside of my company) most often: “accuracy and cost are unsolvable in AI.” Said with conviction. Usually by someone who hasn’t built the test that would prove themselves wrong.

A few weekends ago I had a real optimization decision in front of me. A personal project uses a vision model to read wine labels off photos. The incumbent provider was working. A newer one had come out claiming better cost. The lazy version of this work is “let me try a few labels and eyeball it.” The right version is a harness.

I built one. Same image, both providers, parallel calls, structured fields out the other side. 20 ground-truth labels chosen for spread (Old World, New World, chateau-style, single-vineyard, named cuvées). Per-field scoring with strict text match. Cost and latency captured server-side. Markdown-based outputs and Claude’s analysis of the results. I will do 200 more if I need to.

Then the part most teams skip: a written decision rule, locked in before I looked at the numbers. The new provider wins if it matches the incumbent within 5 points of accuracy AND costs at least 20% less. No moving goalposts.

Result: 33% cost reduction, and after a thin normalization layer, 5 points better accuracy.

Solvable.

The more important finding came from the failure-mode taxonomy. Two providers can both score 75% and have completely different error shapes. The incumbent failed sporadically (hard to predict, hard to fix, visible to users). The new one failed systematically (predictable, post-processable, but silently corrupted every wine of a given class). For this app, where the country field powers a dedup hash, silent systematic failure would be catastrophic. The normalization layer is the difference between “swap” and “do not swap.”

If you only score aggregate accuracy, you miss this. Both look like 75%. One is fine. One silently rots your data.

This is why “unsolvable” makes me roll my eyes. The pattern is portable:

Ground truth.

Parallel calls.

Per-field scoring.

Written decision rule.

Failure-mode taxonomy.

The prompt and the data change per project. The shape doesn’t.

When optimization is the RIGHT move, polish it like you mean it.