When Algorithms Get It Wrong Case#1 — Dutch Probation Risk Assessment

Welcome to a new series.

Over the past months, we’ve explained the EU AI Act — what it requires, who it applies to, and how to comply. Now we’re shifting gears.

Real cases.

Real failures.

And the requirements that would have prevented them.

Because understanding what went wrong

is how we get it right next time.

What is conformity assessment?

It’s the systematic review that proves an AI system meets legal requirements before deployment.

Think of it like a building inspection before occupancy — checking that everything works as intended and meets safety standards.

For high-risk AI systems under the EU AI Act, it’s mandatory.

Each case in this series shows why specific requirements exist and what organisations should verify in their own systems.

Let’s start with a case that made headlines in February 2026.

What Happened

The Dutch probation service (Reclassering Nederland) used algorithms to assess the likelihood of re-offending.

The system processed around 44,000 cases each year.

Judges relied on these assessments to determine sentences and early release decisions.

In February 2026, inspectors from the Dutch Ministry of Justice published their findings:

The system was wrong 20% of the time.

The probation service has since stopped using it.

Where It Went Wrong: Technical Issues

Here’s what inspectors found — and why it matters for any organisation deploying high-risk AI.

1. Training data wasn’t representative

The system used historical data from the Swedish prison population instead of Dutch data.

Imagine training a weather model on Spanish data and using it to forecast Norwegian weather.

Different populations. Different patterns. Different outcomes.

This applies everywhere:

• Credit scoring trained on another market
• HR tools trained on different industries

If your data doesn’t match your reality, your predictions won’t hold.

2. Validation didn’t catch implementation errors

Since 2018 - six years - the formulas for suspects and convicted individuals were swapped.

The model designed for convicted prisoners was applied to suspects.

And vice versa.

No one noticed. For six years.

This is more common than expected.

Testing that a system runs is not the same as testing that it solves the right problem.

3. Critical risk factors were missing

Drug use was not properly included.

Serious psychological conditions were excluded entirely.

As a result, the system consistently underestimated re-offending risk, especially for vulnerable groups.

In any domain:

• Missing variables = blind spots
• Blind spots = systematic errors

4. Accuracy was not fit for purpose

20% error rate.

In a system affecting freedom and public safety.

If your spam filter failed 20% of the time, you’d switch providers.

Here, the system affected human lives.

Accuracy isn’t abstract — it must match the stakes.

Which EU AI Act Requirements Would Have Caught This

Here’s how conformity assessment would have identified these failures.

Article 10: Data and data governance

• Using Swedish data for a Dutch system fails the requirement of representativeness.
• Swapped formulas and missing features reflect weak governance.

This would have been flagged immediately.

Checkpoint:

Is your training data representative of your real deployment context?

Can you explain what your system does, how it was validated, and why it works?

Article 15: Accuracy and robustness

20% error is not acceptable for high-stakes decisions.

Conformity assessment requires evidence, not assumptions.

Checkpoint:

Do you know your real-world accuracy and can you justify it?

Article 9(2)(d): Risk management

The system underestimated risk, thus, creating:

• Public safety risks
• Fundamental rights concerns

Both should have been identified and mitigated.

Checkpoint:

What happens when your system is wrong?

What Organisations Can Learn

This analysis uses public information. It’s not about blame — it’s about learning.

Key takeaways:

• Data quality is non-negotiable
• Validation must catch real errors
• Missing features create predictable failures
• Accuracy must match impact
• Documentation proves your system works

The Bigger Picture

This case is not unique. It’s a pattern.

AI systems:

• deployed in high-stakes contexts
• without proper validation
• without representative data

Problems are discovered only after harm occurs.

The EU AI Act exists to break this pattern.

Key Questions for Organisations

Before deployment, ask:

• Is our training data representative?
• Can we document our validation process?
• Do we have evidence of real-world accuracy?
• Have we identified risks and mitigation measures?

These are not just compliance questions.

They are core engineering questions.

Note: Based on publicly available information to create educational content.

ExplorAI also offers practical EU AI Act training for organisations working with high-risk AI systems. Learn more: explorai.eu/eu-ai-act-training

References

🔗 Dutch News — Probation service used error-ridden algorithms to assess risks

🔗 ExplorAI — AI Audit & Compliance Services https://explorai.eu Independent AI system assessments, high-risk evaluations, and readiness checks for the EU AI Act.

🔗 LinkedIn https://www.linkedin.com/company/exploraieu Follow for practical compliance insights and updates on the AI Act.