9 Plots That Have Probably Cost You an ML Interview (Without You Knowing)

A résumé classifier scored 91% accuracy and went live. These are the nine plots that would have shown what that number was hiding.

Cover Image created using GPT

Click here to access the article for Free!

We’ve all been there.

Tweaking a résumé to satisfy an ATS. Asking GPT to tighten bullet points. Building side projects mainly to have something worth listing. Then comes the automated email.

Rejected.

At some point, a reasonable question appears: how does that decision actually get made? And more importantly, is there a better way for companies to identify the right candidate?

The answer is often hiding inside the model itself.

Imagine a team that builds a résumé classifier using three years of historical hiring data. The results look promising. Validation accuracy reaches 91%. The model is approved and deployed.

From the outside, everything appears to be working.

What follows, however, is a series of quiet failures.

They accumulate in the background while the accuracy metric continues to look healthy. The warning signs are already there. They sit inside the diagnostic plots the team never opened.

The 9 plots that would have caught it

Confusion matrix

When the model launched, the headline number was 91% accuracy. It sounded solid. Nobody paused to ask what it actually reflected.

Most models are trained on historical hiring data. Every résumé carried a label derived from a past decision: hired meant positive, rejected meant negative.

At first glance, that seems reasonable.

A closer look reveals the trap.

The model is not learning what makes someone a strong candidate. It is learning what previous hiring managers approved. Those are very different objectives.

Any preference embedded in past decisions becomes part of the model’s definition of quality. If past hiring managers had a preference for certain universities, certain company names, or certain career paths, the model absorbs that preference as signal. It has no way to distinguish a genuine quality indicator from a historical bias.

The imbalance becomes obvious when you examine the data.

Let’s say company received roughly 4,000 applications each month. By this model only around 350 applicants, about 9%, eventually received interview call. That small group formed the positive class. The remaining 91% became negatives.

In effect, the model learned to define a “good candidate” using a narrow historical minority.

This is where the reported 91% accuracy starts to look less impressive.

Confusion Matrix

The decision threshold sweeps from 0.95 down to 0.05. Watch the four cells fill in as candidates move between them, while a live bar chart tracks accuracy, precision, and recall side by side. A dashed reference line marks the ≈0.91 score a model gets by rejecting everyone — notice how little accuracy moves even as precision and recall swing wildly. That flatness is the whole problem.

A confusion matrix exposes the problem immediately.

One quadrant contains true positives: strong candidates correctly advanced for review. Another contains true negatives: applicants correctly filtered out. False positives represent weak candidates who slip through the screen.

The most expensive quadrant is false negatives.

These are qualified candidates the model incorrectly rejects. In this case, that cell contained 312 people who should have reached a recruiter and never did.

Accuracy hides this failure because it rewards the majority class. With 91% of applications labeled negative, the model accumulates most of its score simply by predicting “reject” repeatedly.

The confusion matrix forces you to confront what the accuracy number conceals. Before reporting any classifier metric, it is usually the first plot worth examining.

ROC curve

The decision threshold was set at 0.5. Nobody had chosen it deliberately — it was the framework default, left in place on the day the code was written and never revisited.

The model produces a score for every résumé. A score closer to 1 means the candidate looks similar to people who were hired in the past. A score closer to 0 means they do not. Turning those scores into decisions requires a cutoff point. Anyone above it moves forward. Anyone below it is rejected. That cutoff is called the threshold, and in many systems it is left at 0.5 simply because that is the default setting.

The problem is that 0.5 often has no relationship to the actual hiring task.

Consider a dataset where only 9% of applicants were historically hired. The model learns from that imbalance. It becomes conservative and rarely assigns very high scores. Most candidates, including many potentially strong ones, end up clustered between 0.3 and 0.55. A threshold of 0.5 may look like a reasonable middle ground, but it is actually slicing through the part of the distribution where many promising candidates sit. Someone with a score of 0.38 or 0.44 may be as capable as the model’s confidence suggests. They simply do not resemble the narrow pattern of past hires closely enough.

This is where the ROC curve becomes useful. Instead of evaluating the model at a single threshold, it evaluates performance across every possible threshold. The vertical axis shows the true positive rate, or the proportion of strong candidates correctly identified. The horizontal axis shows the false positive rate, or the proportion of weaker candidates mistakenly advanced. Points closer to the upper-left corner represent a better balance: more strong candidates are found while relatively few weak candidates slip through. The diagonal line represents random guessing.

Looking at the ROC curve for this model reveals something important. Lowering the threshold from 0.5 to 0.3 moves the operating point much closer to the upper-left region. The number of strong candidates identified increases dramatically, while the increase in false positives remains relatively modest. Nothing about the model changed. The only change was where the cutoff was placed.

ROC curve

The same threshold sweep, now traced as a path on the ROC curve. Two points are marked permanently: the default threshold (0.5) and the alternative discussed here (0.3). As the threshold slides, the right-hand panel shows where candidates actually sit on the score scale — so you can see exactly how many people move from “rejected” to “advanced” as the operating point shifts.

That is why threshold selection matters. The threshold is a business decision. AUC provides a useful summary of overall model quality, but it does not tell you where the model should operate in practice. The actual hiring outcomes depend on the threshold, and the ROC curve is what makes that trade-off visible.

Precision-recall curve

When someone noted that the ROC-AUC score looked reasonable, a senior engineer raised the more useful question: of the candidates the model was actually advancing, how many were genuinely worth interviewing?

The ROC curve shows how a model behaves across the entire dataset. The precision-recall curve focuses on a much more practical question: when the model advances candidates for review, how many of them are actually worth interviewing, and how many strong candidates is it finding in the first place?

Those two measures are precision and recall. Precision tells you what fraction of shortlisted candidates are genuinely strong. Recall tells you what fraction of all strong candidates the model successfully identifies.

This distinction becomes important when the data is highly imbalanced. In this case, 91% of applicants were historically rejected. Under those conditions, a ROC curve can still appear respectable even when the model struggles to identify the relatively small group of strong candidates. The precision-recall curve removes much of that illusion by concentrating attention on the positive class.

The baseline for a precision-recall curve is the prevalence of the positive class itself. Here, only 9% of applicants were hired historically, so random selection would achieve roughly 9% precision. Any meaningful rise above that baseline indicates that the model is finding real signal rather than selecting candidates at random.

At the chosen operating threshold, the model achieved a precision of 64%. On the surface, that looks encouraging. Nearly two-thirds of the candidates reaching recruiters were genuinely strong. The problem becomes visible when you look at recall. Recall was only 11%, meaning the model identified just over one in ten strong candidates. The remaining 89% were filtered out automatically before a recruiter had the opportunity to evaluate them.

Precision-recall curve

This is the trade-off that matters operationally. Increasing recall usually means reviewing more applications, which introduces additional noise and lowers precision. Reducing the threshold allows more strong candidates to reach recruiters, but it also increases the number of weaker candidates entering the review queue. The precision-recall curve makes that exchange explicit.

The same sweep, now on precision-recall axes. The dashed baseline sits at the model’s real positive rate (around 9–10%), so you can see directly how much signal the model is adding above random selection. The live panel on the right recomputes the shortlist composition at each threshold — watch precision fall as recall climbs.

One feature is particularly useful: the precision cliff. At a certain point, small gains in recall cause precision to drop rapidly. Beyond that region, each additional strong candidate found comes at the cost of reviewing a disproportionately large number of weak applications. The curve helps identify where that balance shifts from productive to impractical.

Unlike the ROC curve, which evaluates general ranking performance, the precision-recall curve speaks directly to the resource constraints of the hiring process. It answers the question recruiters actually face: how many applications are we willing to review in order to avoid missing strong candidates?

SHAP summary plot

Legal’s opening question was straightforward: why had this specific candidate been rejected? Nobody had an answer. The score was 0.31, and that number was the entirety of the explanation on record.

A score by itself does not explain a decision. It only tells you where the model ended up. If a candidate is rejected, the more important question is why. Which parts of the résumé pushed the score down? Which ones helped? That is the problem SHAP (SHapley Additive exPlanations) is designed to solve.

SHAP works by breaking an individual prediction into feature-level contributions. Every feature receives a value showing how much it influenced the final score. Positive values push the prediction upward, making a candidate appear more hireable. Negative values push it downward, making rejection more likely. Instead of treating the model as a black box, SHAP reveals the factors that drove each decision.

The SHAP summary plot extends this idea across the entire dataset. Each row represents a feature and each dot represents a candidate. The horizontal position shows how strongly that feature influenced the prediction, while the colour indicates whether the feature value was relatively high or low for that individual. Looking across a row reveals not only whether a feature matters, but also the direction in which it tends to influence outcomes.

This is often where the most important findings emerge.

Suppose you run SHAP on the résumé screening model and discover that university attended is the single most influential feature. Candidates from a small set of target institutions consistently receive positive contributions, while candidates from other universities are pushed toward rejection. The model has not learned who is likely to perform well in the role. It has learned which universities previous hiring managers preferred.

SHAP summary plot

That distinction matters. Historical preferences can easily masquerade as predictive signals when they are embedded in training data. The model faithfully reproduces those patterns, even when they have little connection to actual job performance. From the outside, the system appears objective because every decision is generated mathematically. A SHAP analysis reveals that the underlying logic may simply be historical bias expressed through code.

Without an interpretability tool, this behaviour remains largely invisible. You can observe that certain candidates are being rejected, but you cannot see which features are driving those outcomes. SHAP turns the model from a black box into something that can be inspected, challenged, and audited. That makes it valuable long before deployment. Finding these patterns after a discrimination complaint is damage control. Finding them before deployment is governance.

Calibration plot

The VP of People looked at the model output for a candidate and said: “A score of 0.72 — so there’s roughly a 72% chance they’d be a strong hire?” The engineer paused before answering.

A model score looks deceptively precise. When a candidate receives a score of 0.72, it is easy to assume that means there is a 72% chance they will be a strong hire. In reality, that interpretation is only valid if the model is calibrated.

Calibration measures whether predicted probabilities match real-world outcomes. If a model assigns scores around 0.72 to a group of candidates, then roughly 72% of those candidates should actually turn out to be strong hires. When that relationship holds across the full range of predictions, the model is considered well calibrated.

Many models are not.

Gradient boosting models, in particular, are usually trained to rank candidates correctly rather than produce accurate probability estimates. Their objective is to place stronger candidates above weaker ones. A model can be excellent at ranking while still producing scores that have little relationship to actual probabilities.

The calibration plot is designed to test this. Predictions are grouped into score ranges, such as 0.0–0.1, 0.1–0.2, and so on. For each group, the average predicted score is compared against the actual proportion of positive outcomes. A perfectly calibrated model produces points that fall along the diagonal. Predicted probabilities and observed outcomes match. Deviations from that line reveal miscalibration.

Calibration plot

Overconfident models fall below the diagonal. They assign scores that are higher than the observed success rates justify. Underconfident models sit above the line, assigning probabilities that are lower than the outcomes eventually observed.

Suppose we evaluate the hiring model and find that candidates scored around 0.70 become successful hires only 48% of the time. The model is clearly overconfident. The issue is not that the ranking is necessarily wrong. Candidates scored at 0.70 may still be stronger than candidates scored at 0.40. The problem is that the score no longer means what people assume it means.

That distinction becomes important the moment scores are used as probabilities rather than rankings. A recruiting team might use them to estimate how many strong candidates exist in the pipeline. Leadership might use them to forecast hiring quality or set acceptance-rate targets. All of those decisions assume that a score of 0.70 represents roughly a 70% likelihood of success. If the model is poorly calibrated, those estimates are built on a false premise.

A calibration plot helps uncover this gap. It answers a simple but surprisingly important question: when the model says 70%, does reality agree?

Learning curve

After the legal incident, leadership moved quickly to a solution: label 10,000 more résumés and retrain. The engineer responsible wanted to check, before committing eight weeks to the effort, whether more data would actually change anything.

The learning curve answers a question that teams often get wrong: should we collect more data, or should we improve the model another way?

The idea is simple. Train the same model repeatedly using progressively larger portions of the dataset, then plot training performance and validation performance against the amount of training data. The shape of those curves reveals where the bottleneck actually is.

Three patterns appear again and again.

The first is a large and persistent gap between training and validation performance. The model performs well on the data it has seen but struggles on unseen examples. That is a classic sign of overfitting. In this situation, additional training data often helps because it gives the model a broader set of examples from which to learn.

The second pattern is more subtle. Both curves gradually converge and then flatten at a relatively modest score. The gap disappears, but performance stops improving. This usually means the model has extracted most of the information available from the current feature set. More examples of the same data are unlikely to change much because the limitation is no longer the amount of data. It is the information contained within that data.

The third pattern is what every team hopes to see. Training and validation performance converge at a strong score and remain stable as dataset size grows. The model is generalising well, and additional data offers progressively smaller gains.

Suppose we generate a learning curve for the résumé screening model. Training and validation scores begin to level off at roughly 3,000 training examples. Eventually they settle around 0.73 and 0.71, with only a small gap between them. The model is no longer overfitting, but it is also no longer improving. It has effectively reached the limit of what its current features can explain.

Learning curve

That insight changes the next decision entirely.

Without a learning curve, the natural response might be to collect more data. The team could spend weeks labeling thousands of additional résumés in the hope of improving performance. The curve suggests otherwise. If both lines have already plateaued, adding more examples of the same information is unlikely to produce meaningful gains.

In this case, the breakthrough came from feature engineering rather than data collection. Adding features such as degree field, career progression rate, and skill-to-requirement match increased the validation score from 0.71 to 0.84. That improvement came from providing the model with better information, not more of the same information.

The learning curve would have pointed in that direction from the start. Before investing time and resources into labeling campaigns, it helps answer a simple question: is the model data-constrained, or is it information-constrained? The difference can save months of work.

Feature drift plot

Three months after the retrained model went live, rejection rates began climbing again. Nothing in the codebase had changed. The model was the same. The applicants were not.

A model can continue making predictions long after the world it was trained on has changed.

Imagine a résumé screening model trained on applications collected between 2020 and 2022. The model performs well at launch, so it is deployed and largely left alone. A year later, the hiring market looks very different. New skills associated with large language models, prompt engineering, AI tooling, and agent frameworks appear across thousands of résumés. Terms that barely existed in the training data have become common indicators of relevant experience.

The problem is that the model has never learned what those signals mean.

From the model’s perspective, these new skills carry little or no predictive value because they were absent during training. As a result, candidates with strong and current experience may receive lower scores than they deserve. Their qualifications are present on the résumé, but the model effectively treats them as background noise.

Feature drift plots are designed to detect exactly this kind of change.

They compare the distribution of input features during training with the distribution observed in live production traffic. By placing the two distributions side by side, it becomes possible to see whether the data reaching the model still resembles the data it originally learned from.

A common way to quantify this shift is the Population Stability Index (PSI). As a rule of thumb, PSI values below 0.1 suggest the feature remains stable. Values between 0.1 and 0.2 indicate noticeable drift that should be monitored. Values above 0.2 often signal that the feature has changed enough to justify a closer review, retraining effort, or a full model refresh.

Feature drift plot

What makes feature drift particularly valuable is its timing.

Most performance metrics depend on ground-truth outcomes. In a hiring system, that means waiting until candidates have been hired, onboarded, and evaluated. The feedback loop can take weeks or even months. By the time accuracy metrics reveal a problem, the model may already have been making weaker decisions for a long period.

Feature distributions, on the other hand, can be measured immediately. The moment incoming data starts looking different from training data, drift becomes visible. That makes feature drift one of the earliest warning signals available in a production ML system.

The value is not that it proves the model is failing. A shifting feature distribution does not automatically mean performance has deteriorated. What it does provide is advance notice. It tells you that the assumptions the model learned from are beginning to diverge from reality. That warning often arrives weeks before any decline becomes visible in downstream performance metrics.

By the time accuracy drops, the damage has already started. Feature drift helps you see it coming.

Residual plot

Having grown cautious about the classifier, leadership turned to a related regression problem — predicting how many days each open role would take to fill. The new model’s RMSE looked reasonable. It was shipped. Senior roles kept filling much later than predicted.

Every regression model makes an assumption, whether you check it or not: the remaining errors should be mostly random. Once the model has extracted the patterns it can learn, whatever is left should look like noise.

The residual plot is one of the fastest ways to test whether that assumption holds.

To create it, you plot prediction errors on the vertical axis and predicted values on the horizontal axis. Each point represents one prediction. If the model is doing its job well, the points form a roughly horizontal cloud centred around zero with no obvious structure. The errors may be large or small, but they should not follow a pattern.

When a pattern appears, it usually means the model is missing something important.

Residual plot

Consider a time-to-fill model trained primarily on junior hiring data. Most of the examples involve roles that are filled within a few weeks, while senior positions make up only a small fraction of the training set. On paper, the model performs reasonably well. RMSE looks acceptable and overall error metrics suggest the model is ready for use.

The residual plot tells a different story.

For roles predicted to take around 20 to 25 days to fill, the errors remain small and evenly distributed. As predictions move into the 60 to 90 day range, the spread of errors grows noticeably wider. More importantly, many of those errors fall below zero, meaning actual hiring times are consistently longer than predicted. The model is systematically underestimating how difficult senior hiring is.

The reason is straightforward. The model learned primarily from junior roles because they dominated the dataset. When faced with senior positions, it falls back on patterns that worked well for the majority class. Aggregate metrics hide the problem because most predictions still involve junior hires. The residual plot exposes it immediately.

Different shapes point to different problems.

A fan shape, where the spread of errors increases as predictions grow larger, suggests that prediction uncertainty scales with the target value. In many cases, applying a logarithmic transformation to the target can help stabilise the variance.

A U-shaped pattern usually indicates that the relationship between features and target is more complex than the model can capture. Important nonlinear effects are being missed.

Clusters or bands often suggest that a missing feature is separating the data into distinct groups that the model cannot distinguish.

The ideal residual plot is almost boring. A featureless horizontal scatter around zero means there is no obvious structure left for the model to learn. The remaining errors behave like noise rather than systematic mistakes.

That is why residual plots remain so valuable. Metrics such as RMSE tell you how wrong the model is on average. Residual plots tell you where it is wrong and whether those mistakes follow a pattern. In practice, they are often the quickest way to discover a failing model before the business consequences make the problem obvious.

QQ plot

Finance requested confidence intervals on the time-to-fill predictions so headcount budgets could be planned with some margin. The model produced them. They were consistently too narrow.

Confidence intervals look reassuringly precise, but they depend on assumptions that are easy to overlook. One of the most important is that the model’s residuals follow a normal distribution. Many standard formulas for prediction intervals assume exactly that.

The QQ plot exists to test whether the assumption is reasonable.

The idea is simple. You sort the residuals from smallest to largest and compare them with the values you would expect if those residuals came from a perfectly normal distribution. If the assumption holds, the points align closely with the diagonal. If they bend away from it, the residuals are behaving differently from what the interval calculations expect.

That difference matters because confidence intervals are only as reliable as the assumptions used to create them.

Hiring timelines provide a good example. They rarely follow a neat bell-shaped distribution. Some positions are filled unusually quickly because a strong referral candidate is already available. Others remain open for months because suitable candidates are scarce or negotiations repeatedly fall through. Extreme outcomes occur more often than a normal distribution would predict.

As a result, residuals from a time-to-fill model often have heavy tails. Most predictions may be reasonably accurate, but a small number of cases miss by a large margin in either direction.

A QQ plot makes this visible immediately. Instead of following the diagonal, the points begin to curve sharply away from it at both ends. This characteristic pattern signals that extreme errors occur more frequently than the normality assumption allows.

Two residual distributions animate in sequence. The first is what a QQ plot looks like when residuals are genuinely normal — points sit on the diagonal, and a stated 90% confidence interval covers close to 90% of outcomes. The second is the actual heavy-tailed time-to-fill residuals — points curve away at both ends, and the live coverage counter drops to roughly 68%, exactly the gap described in the text.

The practical consequence is that prediction intervals become overly optimistic.

Suppose the model reports a 90% confidence interval around each prediction. Under the normality assumption, roughly nine out of ten actual outcomes should fall within those bounds. If the residuals have heavy tails, the real coverage rate may be far lower. An interval labelled as 90% might contain only 68% of actual outcomes.

That gap is not a statistical curiosity. It changes business decisions.

If workforce planning teams are using those intervals to estimate hiring timelines, they will consistently underestimate uncertainty. Roles take longer to fill than the upper bounds suggest. Hiring plans slip, teams remain understaffed, and resource forecasts become less reliable.

One solution is to move away from parametric intervals entirely. Bootstrap confidence intervals make no assumption about the shape of the residual distribution. Instead, they estimate uncertainty directly from the observed data. When heavy tails are present, they often produce intervals that better reflect reality.

The QQ plot is a small diagnostic, but it answers an important question: can the uncertainty estimates produced by the model actually be trusted? Before a business starts planning around confidence intervals, it is worth checking whether the assumptions behind them survive contact with the data.

None of the problems in this scenario appeared suddenly. Every one of them was visible before deployment.

The model’s 91% accuracy was real. It had learned useful patterns from the data and it was making predictions that were better than chance. The issue was never that the model failed to learn. The issue was that important warning signs were hiding behind a single performance number.

A confusion matrix would have revealed that most strong candidates were being rejected despite the impressive accuracy score. The ROC and precision-recall curves would have shown that the chosen threshold was sacrificing recall for the appearance of precision. SHAP would have exposed the model’s reliance on university attended as a proxy for hiring decisions. The calibration plot would have shown that scores being treated as probabilities were not actually probabilities. The learning curve would have revealed that collecting more data was unlikely to help, while feature engineering could. Feature drift monitoring would have provided early warning when the applicant pool began to change. Residual and QQ plots would have highlighted weaknesses in the regression models long before those weaknesses affected planning decisions.

None of these plots are particularly advanced. Most have existed for decades. Together, they form a practical inspection checklist for understanding what a model has learned, where it struggles, and how likely it is to fail once it reaches production.

The difference between teams that get surprised in production and teams that do not is rarely model sophistication. More often, it comes down to whether these checks were performed before the model was shipped.

Looking at a confusion matrix before reporting accuracy.

Examining a ROC curve before locking in a threshold.

Running interpretability checks before deploying a model that influences people’s opportunities.

Monitoring feature drift from day one instead of waiting for business metrics to deteriorate.

Each step takes less effort than fixing the problem after deployment.

The most useful question is not which of these plots matters most. Different models fail in different ways, and each plot answers a different question. The more important question is which of these checks your current workflow skips. Every skipped diagnostic creates a blind spot. The model may still perform well, but there is a good chance it is telling you less than you think.

The plots do not make the model better. They make its behaviour visible. In practice, that visibility is often the difference between a model that merely looks successful and one that remains successful after it meets the real world.