Detection-as-Code in One GitHub Action with RSigma

Lint, validate, fields-drift, backtest, and ATT&CK coverage for Sigma rules in one workflow

This is the seventh article in a series on RSigma. The first article introduced RSigma as a CLI tool for evaluating Sigma rules against JSON logs. The second covered running it as a streaming daemon with HTTP and NATS input, stateful correlation, and persistent state. The third showed how to convert Sigma rules into PostgreSQL SQL and run them against TimescaleDB. The fourth wired RSigma into the Grafana LGTM stack for dashboards, metrics, and dynamic alert routing. The fifth plugged live threat intelligence into Sigma rules at runtime via dynamic pipelines. The sixth streamed 1.9 million CloudTrail events through a community IR playbook in seventeen seconds on a laptop.

Six articles, six ways to run detection. None of them talk about how the rules get to production safely. This one does. We will walk through a real Sigma rule that broke in plain sight for years, the class of bug that broke it, and the CI gate that catches it. The gate is published as [timescale/rsigma-action](https://github.com/marketplace/actions/rsigma-detection-as-code) on the GitHub Marketplace; the demo is at mostafa/rsigma-action-demo; both reproduce the original bug end to end.

The bug nobody caught

In April 2026 I sent SigmaHQ pull request #5964. It was a tiny patch: change the field names in the Okta detection rules from lowercase (eventtype, displaymessage) to camelCase (eventType, displayMessage). That is how Okta's actual audit log emits them. The lowercase form had been in those rules since they were first contributed.

Lint passed on the lowercase version. The rule was well formed. Validate passed too: the rule parsed and compiled cleanly. The catch was that the field names in the rule (eventtype, displaymessage) were not the names Okta actually emits in its System Log (eventType, displayMessage). Run those rules against a raw Okta event and they will not match.

In practice, every consumer that had these detections working was applying a field-mapping pipeline as a workaround: lowercase rule fields mapped onto Okta’s camelCase names. The rules looked like they worked because the workaround silently bridged the gap. I was running that workaround myself, including in the original draft of the streaming detection article, until Nasreddine Bencherchali pointed out on Discord that the rule was the bug, not the missing pipeline. PR #5964 fixed the field names at the source so the workaround stopped being necessary.

A static check cannot catch this class of bug. Lint reads the YAML. Validate compiles the rule. Neither of them sees a single real event. The bug lives in the gap between the rule’s field names and the source system’s actual field names, which is a data-shape question, not a rule-shape question.

What would have caught it is a backtest: a check that takes a small corpus of real Okta events, runs the rule against them with no field-mapping workaround in the path, and asserts that it fires at least once. Lowercase the field, the fire count drops to zero, the assertion fails, the gate goes red. That is the gate this article is about.

Three attempts

This is the third time I have tried to ship detection-as-code. The first two taught the lessons that the third one encodes.

The first attempt was a ClickOps SIEM: it let users author and deploy Sigma rules through the UI, with conversion happening in the backend. Behind the scenes the rules and pipelines were committed to Git repos, so there was history and rollback. What there was not was diff-driven review, branch-based experimentation, or a CI gate between the form-submit button and production. Git as a storage backend is not Git as a development culture; UI-first authoring cannot deliver the second.

The second attempt was [grafana/sigma-rule-deployment](https://github.com/grafana/sigma-rule-deployment) (SRD), a GitHub Actions suite that converts Sigma rules to LogQL or Lucene, integrates them into Grafana Managed Alerting payloads, and provisions them on merge to main. I built three of the actions plus parts of the query-testing in another action. I wrote about it in Detection as Code (November 2025). SRD is good at conversion: a rule that compiles will compile cleanly into a backend query and land in Grafana. What SRD cannot do is tell you whether a rule still matches real events after a refactor. It tests for clean conversion, not for behavior. PR #5964 would have passed every SRD check.

The third attempt is what this article is about. RSigma is a single Rust binary that evaluates, converts, and tests Sigma rules end to end, and [timescale/rsigma-action](https://github.com/marketplace/actions/rsigma-detection-as-code) is the Marketplace action that runs it as a PR gate. The headline difference: the gate asserts behavior against a real-event corpus. It backtests. PR #5964 fails it.

Two earlier ecosystem contributions sit behind all three attempts: I wrote the initial Sigma JSON schema for detection rules (originally in the main Sigma repo, later moved to sigma-specification by nasbench), and I built sigma-rules-validator at Grafana and we donated it to SigmaHQ. Three Sigma Enhancement Proposals are in the spec queue too: SEP #212 Array Matching (in-review), SEP #213 Rule-Level Specification Version Declaration (in-review), and SEP #214 Dynamic Time Windows for Correlation Rules (rejected, with a reference implementation living in RSigma’s correlation engine and Postgres backend).

A unit test for a detection rule

A backtest is a unit test for a detection rule. You commit a small NDJSON corpus of representative events to the repo, declare per-rule expectations in a YAML file (Rule X should fire at least once on this corpus, Rule Y should fire exactly twice), and the gate runs every rule against the corpus and compares the actual fire counts to the declared ones. Any drift fails the gate.

The minimum useful corpus is small. Three Okta events and two Windows events is enough to lock in the field names every rule depends on. That is exactly what the demo repo carries, and it is exactly what PR #5964 would have needed.

The CLI is rsigma rule backtest, shipped in v0.17.0:

rsigma rule backtest \
  --rules rules/ \
  --corpus tests/corpus/ \
  --expectations tests/expectations.yml \
  --unexpected fail \
  --junit backtest.xml \
  --report backtest.json

--expectations accepts at_least, at_most, or exactly per rule. --unexpected decides what happens when an unlisted rule fires: fail blocks the gate, warn lets it through with a note, ignore is silent. --junit writes JUnit XML so the gate slots into existing test-report tooling. --report writes a richer JSON report with per-rule and per-logsource rollups.

The expectations file is hand-authored YAML, one entry per rule the team wants to lock down:

expectations:
  - rule: Okta Admin Role Assigned
    at_least: 1
  - rule: Okta API Token Created
    exactly: 1
  - rule: PowerShell Encoded Command
    at_least: 1

That is what would have caught #5964. Lowercase eventType in the rule, the corpus event still says eventType, the fire count drops to zero, at_least: 1 fails.

The five gates in one action

The Marketplace action wraps five rsigma subcommands as composite steps. Each one closes a class of bug the others cannot.

The five gates the action runs on every pull request, plus the artifacts it publishes when they finish.

Lint runs rsigma rule lint --output-format json and turns every finding into a workflow-command annotation on the diff. The JSON envelope is a stable contract: a summary block with file and severity counts plus a findings[] array of {path, severity, rule, message, line}. Annotations carry the lint rule name as their title. The action's lint-fail-level input defaults to warning (stricter than the CLI default of error) because CI is the right place to be strict.

Validate runs rsigma rule validate --resolve-sources [--source ...]. It proves the rules parse, the rules compile, and any dynamic pipeline sources referenced by ${source.*} placeholders can actually be resolved. Catches the case where a rule references a feed that has gone away or whose schema has changed.

Fields drift is the one step that has no native rsigma flag yet, so the action computes it. On a pull request, it runs rsigma rule fields --output-format json against HEAD, checks out the PR merge-base into a temporary worktree, runs fields again, and diffs the field-name sets. Added or removed fields surface in the PR comment as a coverage-drift signal. Non-blocking by default; it is informational, not a gate.

Backtest runs rsigma rule backtest as covered above. The JUnit XML and JSON report upload as build artifacts. The expectations table renders in the sticky PR comment.

Coverage runs rsigma rule coverage --rules ... --navigator layer.json [--targets ...] [--baseline ...] [--atomics ...] [--fail-on-gaps]. It exports an ATT&CK Navigator layer (format 4.5) ready to load into the Navigator UI, and optionally fails the gate when a requested cross-reference reports uncovered techniques. The Atomic Red Team index and the SigmaHQ baseline layer are pulled from their upstream defaults when --atomics true or --baseline true is set.

Exit codes follow rsigma’s house scheme: 1 is findings (gate fails, with annotations), 2 is a rule error (the YAML did not parse), 3 is a config error (a flag or path is bad). A 2 or 3 gets a distinct fail-fast annotation so a broken rule file is never reported as a detection regression.

You can discover the action on the GitHub Marketplace under continuous-integration and security tags. Install with one line:

- uses: timescale/rsigma-action@v1
  with:
    rules: rules/

That minimal call runs lint, validate, and fields-drift. Set corpus to add backtest. Set coverage: "true" to add coverage. The full input list, with defaults, is in the action README. I highly recommend using the SHA hash instead of the version.

Verify, cache, ship

The install step has to download the binary, verify it, and put it on PATH. The verification is two-stage. First, the action fetches the SHA256SUMS manifest from the release and checks the archive against the matching line. Second, it runs gh attestation verify --repo timescale/rsigma, which validates the SLSA build-provenance attestation that the rsigma release workflow signs over every release artifact. There is no insecure fallback flag. If verification fails, the install fails.

The unpacked binary is cached under actions/cache keyed on the resolved version and the runner target triple, so steady-state runs skip both download and verification. Six target triples are supported: Linux, macOS, and Windows on amd64 and arm64.

The action is a composite, not a Docker action. There is no Node runtime to maintain, no image pull latency, no separate registry to harden. Every step is transparent shell. The downside is the runner needs jq and gh; both are pre-installed on GitHub-hosted runners.

The demo end to end

mostafa/rsigma-action-demo is the public consumer. The whole workflow fits on one screen, half of which is the action’s with: block:

name: Detection-as-Code

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  pull-requests: write # the sticky summary comment on pull requests

concurrency:
  group: dac-${{ github.ref }}
  cancel-in-progress: true

jobs:
  rsigma:
    name: rsigma gate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0 # required for the merge-base fields-drift diff

      - uses: timescale/rsigma-action@v1
        with:
          version: v0.17.0
          rules: rules/
          corpus: tests/corpus
          expectations: tests/expectations.yml
          coverage: "true"
          coverage-targets: tests/targets.txt

fetch-depth: 0 is the one requirement worth calling out: the fields-drift step needs git history to compute the merge-base. Without it, the diff step records zero added and zero removed and moves on.

Three rules live under rules/. Two are Okta detections (admin role assignment, API token creation); one is a Windows PowerShell encoded-command rule. Two NDJSON corpus files under tests/corpus/ carry five total events: three Okta, two Windows. The expectations file declares three assertions:

expectations:
  - rule: Okta Admin Role Assigned
    at_least: 1
  - rule: Okta API Token Created
    exactly: 1
  - rule: PowerShell Encoded Command
    at_least: 1

The targets file lists three ATT&CK techniques (T1098, T1098.001, T1059.001) the ruleset is meant to cover.

A green run posts a sticky PR comment with one section per gate:

The sticky PR comment the action posts on a green run, with one section per gate and the table that catches a backtest regression in a single row.

The interesting run is the broken one. Edit rules/okta_admin_role_assigned.yml and lowercase eventType to eventtype. Push to a PR. Lint passes (the YAML is still well formed). Validate passes (the rule still compiles). Fields drift records that eventType is gone and eventtype is added. Backtest fails: the rule no longer matches the corpus event, its actual fire count drops to zero, the at_least: 1 expectation goes red, the gate fails, the PR cannot merge.

That is the exact class of bug PR #5964 was. The gate catches it because behavior is what the corpus measures, not authoring quality.

From CI to program

CI tells you a rule still works. It does not tell you whether the rule still earns its keep. Rules that fire on every event in production are noise. Rules that have not fired in six months are dead weight. Rules that the team consistently triages as false positives belong on the cutting room floor.

rsigma rule scorecard (also v0.17.0) is the program-cadence layer above CI. It joins the backtest report, the coverage report, optional Prometheus volume metrics, and optional triage dispositions, and fuses them into per-rule keep, tune or retire verdicts using the SOC quality thresholds the spec uses by default (retire below 0.10 precision proxy, tune in the review band, keep at 0.80 or higher with recent volume). A retire candidate that is the sole coverage for an ATT&CK technique is automatically downgraded to tune with a coverage-risk note, so the program never silently drops technique coverage.

The scorecard is offline fusion over reports the rest of the toolkit already produces. It is not part of the PR gate (verdicts change too slowly for that). It is part of the quarterly review meeting that decides which rules to retire and which to tune. Same Sigma rules, same RSigma binary, different cadence.

What CI cannot tell you

A backtest gate verifies authoring quality and detection mechanics against a corpus. It is a real gate, much stronger than lint-and-validate, and the absence of one is the reason PR #5964 sat in production for years. But it has limits worth naming.

The gate cannot verify the corpus matches production reality. If the corpus is stale, the gate is stale. The team has to maintain it the way they maintain unit-test fixtures.

The gate cannot replay correlation state over time. A temporal_ordered rule that requires a four-step sequence within a thirty-minute window is hard to express in a small fixture corpus; production replay is the better venue for that class of rule. The streaming detection article covers what production replay looks like.

The gate cannot tell you whether downstream alert noise is sustainable. That is the alert pipeline’s territory: dedup, grouping, silencing, inhibition, the same machinery Alertmanager and other alerting systems run. RSigma ships an alert pipeline in the unreleased section of the CHANGELOG and that is a candidate for a future article.

And the gate cannot tell you whether the rules cover the threats you actually face. Coverage helps; baseline and Atomic Red Team comparisons help; but technique coverage is not threat coverage. That is a threat-modeling conversation, not a CI conversation.

The gate tells you that a refactor did not silently break a rule. That is the floor, not the ceiling. The floor is what was missing.

Wrapping up

Seven articles in, RSigma has been a forensics CLI, a streaming daemon, a SQL compiler, an observability hub, a dynamic-pipelines runtime, a 115k-events-per-second performance engine, and now a Marketplace-installable CI gate. The shape of the toolchain has not changed: one Rust binary, one set of Sigma rules, one rule per YAML file, one canonical schema. The shipping path has.

If you have a Sigma rule repo, install the action:

- uses: timescale/rsigma-action@v1
  with:
    rules: rules/
    corpus: tests/corpus/
    expectations: tests/expectations.yml

For hardened consumers, pin by full commit SHA. For deeper integration, the full inputs and outputs are in the action README, and the demo at mostafa/rsigma-action-demo is fork-and-go.

RSigma is open source under the MIT license:

cargo install --locked rsigma
# or
docker pull ghcr.io/timescale/rsigma:0.17.0

The action is on the GitHub Marketplace under the MIT license too.