A pilot proves a model can work. Production proves an organization can live with it. Those are different problems, and only one of them is technical.

Across thirty-one engagements we kept meeting the same pattern. The data science was competent. The notebook ran. Accuracy cleared the bar agreed in the kickoff. And then nothing shipped — not because the model failed, but because nobody had agreed who would own the pipeline at 3am, what happens when an input column silently changes type, or which team absorbs the cost of a false positive.

The gap is almost never the model

Three failure modes account for most of it. First, ownership: a pilot lives with a data team, production lives with engineering, and the handover is treated as a formality rather than a design constraint. Second, monitoring: teams instrument accuracy but not drift, latency, or cost per inference. Third, and most often, the decision the model was meant to support was never actually re-designed around it.

If nobody can name the person who gets paged when it breaks, it isn’t a system. It’s a demo with a budget.

What we scope instead

We now write the operating model before the model. Who owns it, what it costs to run, what the rollback looks like, and which existing decision it replaces — not augments. It is a slower first month and a considerably shorter path to production.

• Name the on-call owner before the first sprint, not after the demo.

• Instrument drift, latency and unit cost — accuracy alone hides the failure.

• Redesign the decision the model serves, or it will be routed around.