Read the opening of chapter 1
The opening of chapter 1, exactly as printed.
Chapter 1: The Production Loop: From Experiment to Retraining
A model that scored 0.94 AUC in the notebook and got a round of applause in the Friday demo is not a finished piece of work. It's a hypothesis that hasn't met production traffic yet. You already know this in the abstract; it's the whole reason "ship it" feels different from "train it." What's less obvious until you've lived through a few release cycles is that the gap between those two moments isn't a single deployment step. It's a loop, and the loop doesn't stop once the model is live. It just changes what it's asking of you.
That loop, experimentation, deployment, monitoring, retraining, and back to experimentation, is the actual job. Not the modeling. The modeling is one stage out of four, and arguably the one that eats the least of your calendar once a system has been live for six months. Everything in this book comes back to this loop, and this chapter is where you get fluent in its vocabulary, especially the part most engineers learn the hard way: the three distinct ways a production model can go wrong, and why treating them as interchangeable is how a two-hour incident turns into a two-week one.
Experimentation is the part that looks like data science and mostly still is: feature iteration, architecture choices, hyperparameter sweeps. What separates it at engineering maturity is that none of it happens off the books. Every run gets logged to something like MLflow: the code version, the data snapshot, the hyperparameters, the resulting metrics, the artifact itself. This isn't bureaucracy for its own sake. Six weeks after a model ships, someone, often you, will need to answer "why did we pick this configuration over the other four we tried," and "I remember it being a bit better" is not an answer that survives a postmortem. The experiment tracker is also what makes a rollback more than a guess: you can pull the exact artifact and exact preprocessing that produced last quarter's known-good model instead of reconstructing it from memory and hoping the reconstruction matches.
Keep readingShow less
The part of this that trips people up once they're past toy projects is that "same configuration" doesn't automatically mean "same result." A hyperparameter sweep logged faithfully to MLflow still has to contend with the data snapshot underneath it changing between runs if the pipeline pulls from a live table rather than a frozen extract, and with non-determinism in training itself, GPU-level floating point ordering, unpinned random seeds in a data-loading shuffle, a dependency that silently bumped a minor version between two runs. None of that shows up as an error. It shows up as a run that logs cleanly, looks reproducible on paper, and quietly isn't. The discipline that fixes this is boring and worth doing anyway: pin the data snapshot by hash or timestamped extract, not by "the current state of the table," fix the random seed and log it alongside everything else, and pin dependency versions in the environment the experiment actually ran in, not just the ones you meant to use. Skipping this doesn't cost you anything until the day you need to prove why last quarter's model behaved the way it did and discover the "reproducible" run doesn't reproduce.
Deployment takes whatever experimentation validated and turns it into something a service can call. The model gets packaged into a Docker container, dependencies pinned, runtime consistent regardless of which machine it lands on, and that container runs under Kubernetes, which handles the parts you don't want to hand-roll: restarting a crashed pod, scaling replicas under load, routing traffic to the version you actually intend to serve. The mechanics of containerizing a model aren't the hard part at this stage of your career; you've done it. What gets harder, and what's worth walking through in more detail than "gate it in stages" implies, is the actual sequencing of how a new version earns its way to full traffic.
It typically starts with a shadow deployment: the new model runs alongside the current production model on real, live requests, but its predictions never reach a user. They get logged and compared against the incumbent model's predictions and, once labels arrive, against ground truth. Shadow mode is where you catch the failures that never showed up in offline evaluation, because now the model is seeing the actual, messy, occasionally malformed input traffic instead of a curated test set. A model that shadow-deploys cleanly for a few days, with predictions in the expected range and no latency surprises, earns a canary: real traffic, but a small slice of it, often one to five percent, actually served by the new version. This is the first point where the new model's behavior has real consequences, which is exactly why the slice stays small. You watch the canary's live metrics against the same thresholds you'll eventually monitor in full production, not just accuracy but latency, error rate, and the shape of its output distribution, and you watch them separately from the other 95 to 99 percent of traffic so a regression in the canary doesn't get diluted into an average that looks fine. Only once the canary holds for a defined observation window, long enough to see a full daily or weekly traffic cycle rather than just a quiet afternoon, does the rollout widen: 5 percent to 25, 25 to 50, 50 to 100, each step gated the same way, each step reversible in minutes rather than hours if something degrades.
