Only a small fraction of the code in a working machine learning system is the model itself. The rest is plumbing: pipelines, feature stores, monitoring, schema checks, retraining triggers, and the human process that keeps training data honest. That ratio is why a pilot that hits high accuracy in the notebook can post a much lower score weeks after launch and never recover.
Consider a composite case that repeats across industries. A fraud-scoring model demoed beautifully on a curated sample, cleared every review, then quietly degraded once real transactions started flowing through it. The model didn't change. The algorithm wasn't wrong. What changed was everything around the model, and that's the story of most ML failures worth learning from.
For teams thinking through how autonomous systems actually earn their keep once they leave the notebook, this Automatic.co episode about machine Learning Models: Overhyped or Just Underfed is worth a listen. It walks through the same argument from a different angle.
The Demo Ran on a Museum, Not a Kitchen
The training set for that fraud model was a static extract. A large batch of rows pulled from the warehouse, deduplicated, joined against a labeled outcomes table, and handed to the data science team. Every field was populated. Every category was clean, and the class balance had been gently corrected.
The demo accuracy came from that world. Production is a different world entirely.
In production, the same feature arrives in three different currency formats depending on which upstream service emitted it. A merchant category code goes null a small but meaningful share of the time because a partner API silently changed its contract. A timestamp lands in the wrong time zone for a subset of European issuers. None of these are algorithm problems.
They are the pipeline breathing, and the model has no way to tell you it's suffocating.
The Score Slipped Before Anyone Was Watching
The fraud team didn't notice the drop for over a week. Alert volume looked normal. The dashboard the model wrote to was still green because the input schema had not technically broken: fields were there, types matched, row counts were close enough. What had shifted was the distribution of values coming into a handful of features that carried most of the model's signal.
Teams underestimate this failure mode more than any other. A production ML system decays through inputs long before it decays through code. Monitoring that catches a broken API in seconds will happily miss a feature whose mean drifted meaningfully over a fortnight, because from the pipeline's point of view nothing is wrong. Everything is on time.
Everything parses.
Feature Engineering Was Where the Signal Actually Lived
When the team finally pulled the model apart, the fix wasn't in the estimator. It was upstream, in five derived features generated by a notebook nobody had promoted into the production repo. One of them, a rolling seven-day velocity per card, was being recomputed with a window that shifted by one hour every daylight-saving transition.
On the training bench, this was invisible. In the wild, it was rewriting the model's most important input twice a year.
Harvard Business Review has argued for years that bad data makes even excellent models useless, and the point holds harder the more autonomous the system becomes. When the model is a recommendation, a human can catch the weirdness. When the model is deciding, a small upstream defect propagates thousands of times before anyone notices.
The Rebuild Was Mostly Infrastructure
The fix took nine weeks and touched almost none of the model. What it touched was everything the original pilot had skipped:
- Contract tests on inputs. Every incoming feature was pinned to an expected type, range, and null rate. Violations paged someone before they reached the model.
- A promoted feature store. The five notebook-generated features were rewritten as production jobs with owners, tests, and versioning. The daylight-saving bug disappeared with the rewrite.
- Distribution monitoring. Population-level statistics on the top ten features were tracked daily against a rolling baseline, with thresholds that fired on drift rather than on outage.
- A retraining trigger. When drift crossed a defined threshold, the pipeline retrained on fresh labeled data and shadowed the new model against the live one before promotion.
None of that work would have improved the demo. All of it was what the model needed to survive contact with real traffic.
Fund the Layer That Actually Fails
The uncomfortable read of that nine-week rebuild is that the pilot budget had been spent on the wrong thing. Two data scientists tuned a model that was already good enough. Nobody was funded to build the surrounding system that would keep it good enough in production.
If your next pilot is being evaluated on a demo accuracy number, ask a harder question before you sign off. Who owns each input feature in production? What happens when one of them silently drifts? Who gets paged?
The model isn't the project. The data feeding it, day after day, is the project. Fund that, and the demo eventually becomes production. Skip it, and you'll ship another pilot that looks brilliant on stage and dies in the first fortnight.