← 02 — Articles

Write the detection spec before the rule

Detection engineering6 min read

Most rules in a SIEM are a query and a title. Six months later nobody remembers why a service account is excluded, whether the rule was ever tested, or what it was supposed to catch in the first place. The rule keeps firing — or keeps not firing — and nobody can tell which of the two is correct.

A detection spec fixes that. It is a short document written before the rule: what behavior it is meant to detect, which data it reads and what must be configured for that data to exist, what it will not see, how to prove it works, and what the analyst should do when it fires. This article describes a format, walks through one example and ends with a template you can put next to every rule: detection-spec-template.yml.

It is the last step before rules in this series. The log source inventory tells you what data exists; the audit policy, Sysmon, PowerShell and auditd articles make sure the right events are recorded. The spec is where a rule connects to all of that.

Where the format comes from

The structure is inspired by Palantir’s Alerting and Detection Strategy (ADS) framework: goal, categorization, strategy, technical context, blind spots and assumptions, false positives, validation, priority and response. Our template keeps those ideas, writes them as YAML so they can be checked and searched like the rules themselves, and adds three fields that come from operating rules rather than designing them: a link to the log inventory, an allowlist with review dates, and an append-only tuning log.

The sections, and why each one is there

Goal. One or two sentences: what behavior, and why it matters. If you cannot write the goal without naming the query, you are describing the rule, not the behavior.

ATT&CK. Only techniques the rule actually detects. A rule for a new service on a domain controller detects Create or Modify System Process: Windows Service (T1543.003). It does not detect “persistence” in general, and listing every loosely related technique makes coverage maps lie.

Data. Which sources and event IDs the rule reads, and — the part that is usually missing — what must be configured for them to exist. This is where the spec links to the inventory and the audit baseline. A rule whose requires line points to a subcategory that is off is a rule you know is blind before it ships.

Strategy. How the rule works, in plain language, so that a reviewer can check the logic without reading the query.

Blind spots and assumptions. What the rule will not see, and what must be true for it to work. This is the most valuable section and the one most often skipped. Writing it down turns a hidden gap into either a second rule or an accepted risk.

False positives and allowlist. Expected legitimate matches and how each is handled. Every allowlist entry gets a reason, a date it was added and a date it must be reviewed by. Allowlists without review dates only grow.

Validation. How to prove the rule works: the steps that produce a true positive in a test environment, the events you expect and the alert you expect. Run it before production and after every change.

Priority and response. What triage does with an alert, and the first steps of the response. Priority is a statement about what happens next, not about how dangerous the technique sounds.

Tuning log. Append-only: date, what changed, why, and alerts per week before and after. It is the history that explains every exclusion.

A worked example: new service on a domain controller

The template ships with one complete example. In short:

  • Goal: detect a new Windows service on a domain controller. Services run as SYSTEM and start automatically; new ones on DCs are rare and planned.
  • ATT&CK: T1543.003.
  • Data: 4697 in the Security log, which requires Security System Extension: Success in the audit policy, and 7045 in the System log, which the Service Control Manager writes by default. Two sources for the same event mean the rule survives one of them being misconfigured.
  • Strategy: alert on either event on a DC, join both by host and service name within 5 min, exclude the allowlist.
  • Blind spots: a service created by changing the ImagePath of an existing one produces neither event; a cleared System log leaves only 104. Each blind spot names the rule that covers it, or says that nothing does.
  • False positives: patching and agent upgrades reinstall services. Allowlist by service name and full image path, reviewed quarterly.
  • Validation: sc.exe create ew-det-test binPath= C:\Windows\System32\notepad.exe on a test DC, then sc.exe delete ew-det-test. Expect both events and one alert.
  • Response: who installed it (SubjectUserName on 4697), is there a change ticket, and collect the binary before deleting anything.

The spec is about a page. The rule it describes is a few lines. The page is what lets someone else run, tune and trust the rule.

Keeping specs next to rules

Put each spec in the same folder as its rule, in the same repository, and review them together: a change to the query without a line in tuning_log is an incomplete change. Two checks are easy to automate once specs are YAML:

  1. Every rule has a spec, and every spec has validation.last_run filled in before status: production.
  2. No allowlist entry is past its review_by date.

Neither needs a special tool. A short script in your CI that parses the YAML files and fails on either condition is enough.

Testing the spec itself

A spec is good if someone other than its author can use it. Before a rule goes to production, give the spec — not the rule — to a colleague and ask three questions:

  1. Can they write a rule from it that catches the validation case?
  2. Can they name one thing the rule will not see, without reading blind_spots?
  3. Do they know what to do when it fires?

If the answer to the second question is yes and the spec does not list it, add it.

Limitations

  • A spec does not make a rule correct. It makes its assumptions visible, which is how incorrect rules get found.
  • The format costs time per rule. For a large set of vendor-provided rules, start with specs for the rules you modified or wrote yourself, and for the ones that page someone at night.
  • The example is a starting point. We have not tuned it on your domain controllers; the allowlist is empty on purpose.

Adapting this to your environment

  • ADAPT status and priority to your triage process. If your SOC uses P1–P4, use those.
  • ADAPT data.sources to the names or IDs in your own log source inventory, so the link works both ways.
  • ADAPT the file layout to your repository; one folder per detection with spec.yml next to the rule file works for Wazuh XML, KQL and Sigma alike.

The template is detection-spec-template.yml (CC BY 4.0). With it, the series has covered the ground every rule stands on. If something here is wrong, write to hello@entropywatch.dev and we will correct it in the open.