Before Using Anomaly Detection in Your Data Project: Is Your Data Suitable?

Our data team is tiny, and we already manage more data-related work than our time and capacity allow. And, of course, data quality monitoring is one of our major issues.

When I first learned about the Elementary, open-source dbt package with powerful out-of-the-box anomaly detection toolkit, I was excited by its potential. It seemed like a useful way to surface data quality problems, including issues we may not even know exist yet.

However, after looking into it more closely, I realized that adopting it could create more work and complexity than we have now. In my case, the anomaly detection tests implemented in Elementary required more tuning and maintenance than the value they provided.

How Elementary Anomaly Tests Work

Elementary anomaly tests are a good fit when your data has a reasonably stable “normal” pattern. And a bad fit when the process is so irregular or shifting that there is no stable baseline to learn from, or the cost of constant reconfiguration outweighs the value.

Elementary’s anomaly tests monitor a metric (row count, null rate, average value, etc.) by comparing recent buckets to a training set of historical buckets using a z‑score–style range (default ±3 standard deviations, tunable via sensitivity). They assume that, over the training period (optionally conditioned on simple seasonality like day‑of‑week or hour‑of‑day), your data clusters around a “normal” mean and variance for that metric.

That design is powerful when “normal” is a meaningful, relatively stable concept for that metric and time scale, and much less satisfying when “normal” is constantly being redefined by business changes, erratic seasonality, or structural breaks.

Statistically, these are examples of structural breaks: situations where the underlying process changes and historical behavior no longer represents the future. Anomaly detection systems often assume that the past remains a reasonable baseline.

Coefficient of Variation: A Simple Suitability Check

A simple metric can help determine whether anomaly detection is a good fit for your data: the coefficient of variation (CV).

Formula:

Coefficient of Variation

SQL:

select
 avg(volume) as mean_volume,
 stddev_samp(volume) as std_volume,
 stddev_samp(volume) / nullif(avg(volume),0) as cv
 from daily_volume

If CV is below 0.2, default settings may work well. Between 0.2 and 0.3, expect moderate tuning. Above 0.3, expect increased false positives or false negatives.

Here are practical observations from my experiments rather than statistically strict boundaries:

Variability in my data is close to 0.5 with complex monthly seasonality and numerous holiday related drops.

Although the concept sounds simple, the coefficient of variation can change over time as the business changes. Marketing campaigns, new product launches, and seasonal cycles such as the start of the school year can all shift what looks “normal” in your data.

Because of that, anomaly detection settings cannot be configured once and left unchanged. Even a well-tuned setup will need periodic review. A rise in false positive alerts is usually an obvious signal that the configuration needs adjustment. The harder problem is when alerts stop appearing altogether: that may mean the data is healthy, or it may mean the tests have become too insensitive and are missing real issues. False positives create noise, but false negatives create a false sense of confidence. In practice, this may require additional monitoring to confirm whether the system is performing well or whether the sensitivity needs to be lowered because of recent business events.

Experiment: Three Datasets with Different Variability

I created three simplified datasets with different levels of variability to show how much configuration each case requires.

From a statistical perspective, reliable estimates generally benefit from larger samples, often around 30–100 observations depending on the distribution and the problem being analyzed. For these examples, however, I intentionally used Elementary’s default configuration: a 14-day training period and a 2-day detection period. I am not sure why 14 days were chosen as the default training window; perhaps a shorter period reduces the chance that business events, trends, or changing behavior distort the baseline.

You may also notice an important detail: with a 14-day window and a 2-day detection period, the effective baseline training period is only 12 days if detection days are excluded. Elementary provides a special setting for this exclusion (exclude_detection_period_from_training = true), which makes me wonder why the exclusion is not the default behavior. From a statistical perspective, including the detection period in the baseline feels counterintuitive: the observations being tested can influence the average and standard deviation used to decide whether they are anomalous.

From a business perspective, Day 14 in the detection period (520) should trigger an alert.

Now let’s start from default sensitivity 3 and see what happens in each data set.

As dataset variability increased, anomaly detection required more effort to produce the expected results. At a CV of about 18–20%, the default configuration generally works with little or no tuning. Around 25–27%, expected anomalies began to be missed, requiring sensitivity changes and sometimes seasonality. Beyond roughly 30%, the process shifted from monitoring the data to continuously tuning and validating the tests.

Testing anomaly detection on historical data in Elementary is not very straightforward. The package works with current dates only, so to replay historical periods I had to shift historical dates to the current time in a view. I wish it included functionality for tuning and backtesting against historical data. Even if the results could not be used directly in production, it would at least provide a better sense of the amount of future tuning and maintenance required.

Alternatives When Z-Score Detection Fails

When z-score-based anomaly detection is not a good fit, more robust methods may work better depending on the shape and stability of the data.

· IQR method with a moving median: A better option for skewed or irregular distributions than a z-score approach based on a moving mean. Alerts are triggered for values below Q1–1.5×IQR or above Q3 + 1.5×IQR.

· Percentile method: Flag values below the 1st percentile or above the 99th percentile.

· Isolation Forest: A more advanced method that can handle complex distributions and does not rely on normality assumptions.

· Mahalanobis Distance: Multivariate z-score that accounts for correlation between variables

· Local Outlier Factor (LOF): Detects outliers based on local density deviations

· ARIMA residuals: Fit time series model, flag when residuals exceed ±3σ

· Prophet anomaly detection: Automated trend/seasonality decomposition with uncertainty intervals

· Change-point detection: Identify when statistical properties shift

Methods like IQR and percentile-based rules are easy enough to implement in custom tests or lightweight dbt packages like Elementary. I wish Elementary included these approaches as built-in options, since they can be more intuitive and sometimes more practical than tuning standard deviation–based anomaly detection for highly variable datasets.

The remaining techniques are more complex and are usually justified only in exceptional cases — not for routine, broad monitoring across everyday projects.

In most projects, business-rule tests based on percentage change, source freshness, and project-specific audit metrics — combined with data ownership, monitoring, and visualization — are easier to implement, interpret, and maintain than anomaly-detection tests.

Conclusion

Elementary is a strong package with useful features beyond anomaly detection itself, including Slack integration, reporting, and an overall monitoring framework. Many teams use it successfully. My conclusion is not that anomaly detection is ineffective or that Elementary should be avoided. Rather, before adopting any anomaly detection system, it is worth asking a simpler question: is your data suitable for it? Sometimes the biggest challenge is not finding anomalies — it is defining what “normal” means for your business.