Read the opening of chapter 1
The opening of chapter 1, exactly as printed.
Chapter 1: What an ML Engineer Actually Does All Day
It's 9:14 AM, and Priya is looking at a dashboard that says her fraud-detection model got worse overnight. Not broken — worse. Yesterday it flagged 94% of fraudulent transactions correctly. This morning it's at 89%. Nobody touched the code. Nobody changed the model. Somewhere in the last twelve hours, the pattern of transactions flowing through the system shifted just enough that the model started missing things it used to catch.
This is the moment that separates what people imagine an ML engineer does from what the job actually is. If you pictured someone in a quiet room, sketching neural network architectures on a whiteboard and training beautiful models from scratch, that's not quite it. If you pictured someone running a script once, getting a good accuracy number, and moving on to the next idea, that's not it either. The job is watching that dashboard, figuring out why the number moved, and fixing it before it costs the company money or costs a customer their trust.
Here's the one-sentence version of this whole chapter: an ML engineer's real job is making sure machine learning models keep working for actual users, in the real world, not just performing well once in a notebook. Everything else — the code, the meetings, the 2 AM training jobs — exists in service of that one goal. A model that scores 98% accuracy on a test file and then falls apart the moment real customers start using it hasn't done its job. Getting it to hold up under that pressure is the job.
Keep readingShow less
The best way to understand what this looks like day to day isn't a bullet-point list of responsibilities. It's watching one actual day unfold.
The day usually starts with a short team standup — a 10 to 15 minute check-in where everyone says what they worked on yesterday, what they're doing today, and what's blocking them. For an ML engineer, this is also the first chance to look at what happened overnight.
Here's why overnight matters so much in this job: training a machine learning model — the process of having the model learn patterns from historical data — can take hours. Nobody sits and watches a training job run for six hours in real time. Instead, engineers kick off long training runs before they leave for the day and check the results the next morning. So 9 AM standup often opens with something like Priya's situation: a dashboard, a number that moved, and a room full of people trying to figure out what it means before they've finished their coffee.
Sometimes the overnight news is good — a new version of the model that was training overnight beat the old one by 2 percentage points on the metrics the team cares about. Sometimes it's neutral — the numbers are basically flat, so the team decides whether to try a different approach or move on to something else. And sometimes, like with Priya's fraud model, it's a problem: the model that's already running in production, serving real users right now, has quietly gotten worse. That specific kind of problem — a deployed model's performance sliding over time — has a name you'll hear constantly in this field: drift. It happens when the real-world data flowing into a model shifts away from the data the model was originally trained on. Fraudsters change their tactics. Customer behavior shifts with the season. The data the model sees today just isn't quite the data it learned from, and its accuracy pays the price.
Whatever the overnight results say, standup is where the team decides what today's priorities actually are. Nobody starts their day cold on a plan made yesterday. The plan gets adjusted based on what happened while everyone was asleep.
By late morning, the meetings are done and the actual hands-on-keyboard work begins. This block is usually the most technically dense part of the day, and it breaks down into a handful of concrete steps that repeat constantly in this job.
First comes data preprocessing — cleaning and reshaping raw data into a form a model can actually learn from. Raw data is messy. It has missing values, weird formatting, duplicate records, and outliers that don't make sense. Before any model can learn anything useful, someone has to get that data into consistent shape.
Next is feature engineering, and this is worth slowing down for, because it's one of the highest-leverage skills in the entire job. A "feature" is just an input the model uses to make its prediction — a single measurable piece of information. For Priya's fraud model, a feature might be "how many transactions has this account made in the last hour" or "is this purchase amount unusually large for this customer." The raw data rarely arrives already shaped into useful features like that — engineers have to construct them.
