Designing a Config‑Driven Dynamic Data Purge Strategy in .NET (Hang-Fire)

As applications grow, so does their data. Event logs, staging tables, documents, and audit records can quickly pile up — leading to increased storage costs and degraded performance. A well‑designed data purging strategy becomes essential for long‑running systems.

In this post, I’ll walk through a config‑driven dynamic purge strategy in .NET that allows teams to manage data retention without changing application code.

The Problem with Traditional Purge Implementations:

In many systems, purge logic is implemented like this:

• Hardcoded table names
• Fixed retention periods
• Code changes required to add or remove purge rules
• Redeployment for even minor configuration tweaks

This approach doesn’t scale well, especially when:

• Different tables need different retention rules
• Purge requirements change frequently
• Operations teams want control without developer involvement

The Core Idea: Configuration Over Code

Move purge rules out of code and into configuration.

Instead of hardcoding what to purge, when to purge, and for how long, we define everything in a Purge Configuration table. A scheduled background job reads these rules at runtime and executes purge logic dynamically.

Architecture Overview

High-Level Flow

1. A scheduled job runs on a fixed schedule (for example, once daily).
2. The job reads purge rules from a configuration source (database table).
3. Only enabled rules are processed.
4. Retention logic is applied dynamically.
5. Related data is purged consistently using a central reference timestamp.

Purge Configuration Table

A typical configuration table might look like this conceptually:

Purge Configuration

• Table Name — Target table for purge
• Retention Days — How long data should be retained
• Is Enabled — Toggle purge on/off without code changes

Example Configuration

This approach makes the purge behavior completely data‑driven.

Event Log as the Source of Truth

One important design choice is using an Event Log table as the central reference point.

Why?

• Many related tables (documents, logged, audits, failures) depend on events
• Purging child tables independently can lead to orphaned records
• A single, authoritative timestamp ensures consistency

How It Works?

• Retention cutoff is calculated using the Event Log timestamp
• All related tables are purged based on that cutoff
• This guarantees that dependent data is never deleted prematurely

Dynamic Purge Execution

At runtime, the purge engine:

1. Fetches enabled purge rules
2. Calculates cutoff date using retention days
3. Validates against the Event Log timestamp
4. Executes table‑specific purge queries

Because the rules are externalized:

• New tables can be added without redeployment
• Retention periods can be adjusted on the fly
• Purge can be instantly paused for any table

Scheduling the Purge Job

In a .NET ecosystem, this works well as a background job.

Example Using Hang-fire Package

• Hang-fire allows reliable, persistent background jobs
• Easily configurable CRON scheduling
• Built‑in monitoring dashboard

Typical Schedule

*CRON: 0 0 * ***

➡ Runs the purge job once daily at midnight

This cadence is sufficient for most compliance and cost‑control scenarios.

Benefits of a Config‑Driven Purge Strategy

No Code Changes Add, remove, or modify purge rules without touching application code. Operational Control Enable or disable purging instantly per table.

Scalable Design Works well as the number of tables grows.

Safe and Consistent Centralized timestamp prevents data inconsistency.

Future‑Proof Easy to extend with filters, batch sizes, or archiving logic.

When Should You Use This Approach?

This strategy is well‑suited for:

• Event‑driven systems
• Systems with audit or compliance requirements
• Applications with many dependent tables
• Long‑running enterprise applications

Sequence Diagram: Dynamic Data Purge Flow

Sequence Diagram: Dynamic Data Purge Flow

Sample Pseudocode

1. Scheduled Purge Job Entry Point

function RunDailyPurgeJob():
    purgeConfigs = FetchEnabledPurgeConfigs()

    for each config in purgeConfigs:
        ExecutePurgeRule(config)

2. Fetch Enabled Purge Rules

function FetchEnabledPurgeConfigs():
    return Database.Query(
        "SELECT TableName, RetentionDays
         FROM PurgeConfiguration
         WHERE IsEnabled = true"
    )

3. Execute Purge Rule Per Table

function ExecutePurgeRule(config):
    cutoffDate = CalculateRetentionCutoff(config.RetentionDays)

    sourceEventIds = FetchExpiredSourceIds(cutoffDate)

    if sourceEventIds is empty:
        return

    PurgeTargetTable(config.TableName, sourceEventIds)

4. Calculate Retention Cutoff

function CalculateRetentionCutoff(retentionDays):
    currentDate = GetUtcNow()
    return currentDate.minusDays(retentionDays)

5. Fetch Expired Source Records

function FetchExpiredSourceIds(cutoffDate):
    return Database.Query(
        "SELECT SourceId
         FROM Table_Name_1
         WHERE CreatedTimestamp < cutoffDate"
    )

6. Purge Records from Target Table

function PurgeTargetTable(tableName, sourceIds):
    Database.Execute(
        "DELETE FROM {tableName}
         WHERE SourceId IN (sourceIds)"
    )

7. Optional Safeguards & Enhancements

• Execute deletes in batches to avoid long locks
• Use transactions per table purge
• Log row counts and execution time
• Support archiving instead of hard deletes
• Add dry‑run mode for validation