Predictive Maintenance is not 2000 False Alarms to Catch 200 Faults

What precision-recall curves say about fixed-threshold vibration monitoring and why “we detect faults” is not the same as “we detect faults usefully”

Here is the number I keep coming back to. On a test set of 5,980 vibration samples from a medium-sized industrial bearing, of which 598 were genuinely anomalous, a fixed threshold derived from ISO 20816 raised 2,039 alarms. Two hundred of them were real. The other 1,839 were not, and it still missed two-thirds of the actual faults.

Put that in the language of a maintenance team: you get roughly one useful alarm for every ten you investigate, and the bearing that’s actually degrading has a two-in-three chance of not being flagged at all.

The uncomfortable part isn’t that the threshold performed badly. It’s that a system behaving this way can still look like it’s working, because it does raise an alarm when something is wrong, just not only then. If you count the times it was right and don’t count the times it cried wolf, you have a success story. If you’re the technician who walked to the machine nine times for nothing, you have something else.

This came out of our work. I want to walk through what we did and, more importantly, what the precision-recall curve actually tells you, because I think it’s the single most useful thing a person buying or building predictive maintenance can learn to read.

What we actually ran

The data comes from a test rig at the Politecnico di Torino built specifically for medium-to-large bearings, which is unusual since most public bearing datasets are small bearings under controlled conditions, and they don’t behave like industrial ones. The rig takes bearings up to 420 mm outer diameter, is driven by an inverter-controlled 30 kW motor, and applies radial and axial load through independent hydraulic actuators up to 200 kN.

The bearing under test was an SKF 22240 CCK/W33 spherical roller bearing, in four states: healthy, inner race damage, outer race damage, and rolling element damage. The damage was induced by chip removal, 2 mm diameter and 0.5 mm deep. Four load cases, from unloaded up to 124.8 kN radial plus 49 kN axial, and ten nominal speeds from 127 to 997 rpm. That variability matters, and I’ll come back to it.

Vibration was recorded for 30 seconds per condition at 20,480 Hz, split into 4,096-sample segments with 50% overlap, and reduced to eleven time-domain features, the usual set: mean, standard deviation, root amplitude, RMS, absolute maximum, skewness, kurtosis, crest factor, form factor, shape factor, impulse factor.

The setup is novelty detection, not classification, because that’s the situation almost everybody is actually in, you have data from a machine running normally, and you do not have a labelled catalogue of its failures. So the training set was 5,980 samples of healthy data only. The test set was another 5,980, of which 598 (10%) were anomalies drawn from any of the damage classes. Five independent repetitions.

Against that we ran four unsupervised algorithms that need no fault labels such as Isolation Forest, One-Class SVM, Local Outlier Factor, Elliptic Envelope, and one threshold approach: broadband RMS velocity against the 4.5 mm/s benchmark from ISO 20816–1 and 20816–3.

I want to be explicit about that last one, because it’s where this work is easiest to misread. Our use of the threshold is deliberately heuristic. It is not a faithful implementation of the standard’s full procedure, and ISO 20816 was never written as a bearing-defect detector, it describes overall machine vibration severity for classes of machines. What we tested is the thing people actually do in the field: take the number from the standard, wire it to an alarm, and call it condition monitoring.

Why accuracy didn’t show the problem

The threshold’s accuracy was 0.63. Not brilliant, but the kind of number that survives a slide deck. “Sixty-three percent” doesn’t sound like a system that’s wrong nine times out of ten when it speaks.

That gap is what class imbalance does to accuracy. Only 10% of the test set is anomalous, so a detector that simply said “normal” every single time would score 90% accuracy and detect nothing at all. Accuracy is measuring the wrong thing, mostly it’s measuring how many normal samples you correctly ignored, and there are a lot of those.

The same trap is set, more subtly, by ROC curves. It’s entirely possible to get an ROC AUC around 0.92 — which looks excellent — on a problem where the precision-recall AUC is 0.62, because the false positive rate on the ROC axis is diluted by an enormous number of true negatives. A thousand false alarms against five thousand normal samples barely moves the FPR. It completely destroys the precision.

What a precision-recall curve is, without the jargon

Two questions, and they are the only two that matter operationally.

Precision: of the alarms this system raises, how many are real? This is what determines whether people keep responding to it.

Recall: of the faults that are actually present, how many does it catch? This is what determines whether the thing does its job.

The curve traces the trade-off between them as you move the decision threshold. Loosen it and you catch more faults while raising more noise; tighten it and your alarms become trustworthy while more faults slip past. The area under that curve summarises how well the underlying score separates faults from healthy behaviour, independently of where you happen to have set the alarm.

There’s one number you need in order to read a PR-AUC, and it’s the prevalence. Here, 10% of the samples are anomalous, so a score that ranks samples at random gives you precision 0.10 everywhere — a flat line at 0.10 is the floor, not zero.

The ISO threshold’s PR-AUC was 0.105.

ISO 20816–3 threshold. Precision-recall curve.

That is the finding, stated as compactly as I can state it. In this setting — localized bearing damage, variable speed, variable load — broadband RMS velocity carries almost no information for separating damaged from healthy samples. It isn’t a matter of the threshold being set at the wrong value. The quantity being thresholded barely ranks the faults above the healthy data in the first place, so no choice of threshold would have rescued it.

Look at the violin plots of RMS velocity by condition and you can see why: the distributions for healthy, inner race, outer race and rolling element damage overlap almost entirely. A 2 mm spall on a race does not necessarily raise the broadband energy of a large bearing turning at 200 rpm under variable load. It changes the structure of the signal, which is precisely what the frequency domain and envelope analysis are for and which we deliberately left out of this study, because we wanted to test methods that assume no prior knowledge of fault frequencies.

Violin plot

What the trainable models did with the same features

Same eleven features, same healthy-only training, same test set.

Results

The comparison that does the work is the first row against the last. Local Outlier Factor caught 487 of the 598 faults while raising 48 false alarms. The threshold caught 200 while raising 1,839. Same signals, same features, same machine — the only difference is that one method was allowed to learn what healthy looks like for this bearing and the other was handed a number written for a class of machines in general.

Isolation Forest is worth a second look because it’s the one that most resembles what a lot of products do. It has the highest recall in the study, 0.87 — it finds nearly everything. It also raises a thousand false alarms doing it, so its precision is 0.34. If your metric is “detects 87% of faults”, Isolation Forest gives you an excellent slide. Whether it gives you a system people still trust after three months is a different question, and it’s not one that recall can answer.

The point that generalises

A fixed threshold is a single operating point. You get one place on the precision-recall plane and you cannot move it, because the number came from a standard rather than from your machine.

A model trained on healthy data gives you a score, and a score gives you a curve. That means the operating point becomes a choice, and — this is the part that gets skipped — it becomes a choice that belongs to the person who knows what a false alarm costs relative to a missed fault. In a plant where a spurious inspection means twenty minutes and a walk, you sit toward high recall. In one where responding means stopping a line, you sit toward high precision. Neither answer is technically correct or incorrect. They are different businesses, and the same curve serves both.

That’s why I think the precision-recall curve is worth understanding even if you never write a line of code. It’s not a machine learning artifact. It’s the shape of a decision you’re making whether you look at it or not.

Where I’d draw the limits of this

The threshold application is heuristic by design, as I said, and someone applying the full ISO 20816 procedure with machine-class-specific baselines would do better than 0.105. How much better is an open question, and a fair one to put to me.

ISO 20816 is also being asked here to do something it wasn’t written for. It describes overall vibration severity — a general statement about whether a machine is running roughly as it should. Detecting a localized defect on a rolling element is a different problem, and the standard never claimed that territory. The point isn’t that the standard is deficient; it’s that a lot of monitoring installations use it as though it covered a job it doesn’t cover.

On the other side, the models have their own conditions. They need healthy data from the specific machine, which means an installation period before they’re useful, and re-establishing that baseline when the operating regime changes materially. Our anomalies were pooled across damage types and across a wide range of speeds and loads, which is realistic but also means the numbers describe an aggregate rather than a specific regime. Variance across five repetitions was low, which is reassuring, but a broader validation on more machines is the obvious next step. Adapting the detection to the operating regime — rather than training one model across all of them — is where I’d expect the next gain to come from.

Three questions worth asking

If you’re evaluating a predictive maintenance system, yours or someone else’s:

What is the false alarm rate, in alarms per month, on the machines you actually run? Not detection rate. Not accuracy. The number that determines whether the maintenance team keeps answering.

Can the operating point be moved, and by whom? If the answer is that the threshold is fixed by a standard, you are being sold one point on a curve you never got to see.

What is the prevalence in the validation set? Without it, no accuracy figure and no AUC can be interpreted. A PR-AUC of 0.6 is strong at 5% prevalence and mediocre at 50%.

There’s a version of this field where we sell detection rates and let alarm fatigue quietly happen downstream, and a version where we’re straight about the trade-off and let the customer choose where to sit on it. The second one is harder to put on a slide. It’s also the only one that survives contact with the people who have to walk to the machine.

Paper: Di Maggio, L. G., Brusa, E., Delprete, C. (2025). Novelty Detection in Rotating Machinery: Assessment of Unsupervised Machine Learning Models for Medium-Sized Industrial Bearings. 2025 International Conference on Control, Automation and Diagnosis (ICCAD), IEEE. DOI: 10.1109/ICCAD64771.2025.11099203

Dataset: Di Maggio, L. G., Giorio, L., Delprete, C., Brusa, E. (2024). Dataset of Vibration, Temperature and Speed Measurements for Multiple Types of Localized Defects on Spherical Roller Bearings across Multiple Operating Conditions. Zenodo. DOI: 10.5281/zenodo.13913254

Test rig: Brusa, E., Delprete, C., Giorio, L., Di Maggio, L. G., Zanella, V. (2022). Design of an Innovative Test Rig for Industrial Bearing Monitoring with Self-Balancing Layout. Machines, 10(1), 54.