← 03 — Blog

Event time versus ingestion time: which clock your rule is reading

Tooling2 min read

Every event in a SIEM has at least two timestamps: when it happened on the source, and when the SIEM received it. Most of the time they are seconds apart and nobody cares. A laptop that was offline over the weekend, a congested forwarder, or a cloud API that delivers in batches pushes them minutes or hours apart, and scheduled rules quietly miss events.

Symptom

A test event you generated on Friday evening on a laptop shows up in the SIEM, with Friday’s timestamp, on Monday morning. The detection that should have fired on it never does, and a manual search finds the event without any problem.

Cause

A scheduled rule runs every N minutes and looks back over a window measured in event time. If an event arrives after the window that would have contained it has already run, the next run’s window no longer reaches back far enough. Microsoft documents exactly this in Handle ingestion delay in Microsoft Sentinel: an event generated in the first window but ingested two minutes later is filtered out by the second run because its TimeGenerated is already too old, so “the rule does not fire an alert”.

Where the two clocks live

Platform Event time Arrival time
Microsoft Sentinel / Log Analytics TimeGenerated ingestion_time() function
Splunk _time _indextime
Wazuh the original event’s time, e.g. data.win.system.systemTime for Windows timestamp (when the manager processed it)

Measure the delay per source

In Sentinel:

SecurityEvent
| where TimeGenerated > ago(1d)
| extend DelayMin = datetime_diff('minute', ingestion_time(), TimeGenerated)
| summarize p50 = percentile(DelayMin, 50), p95 = percentile(DelayMin, 95), max = max(DelayMin) by Computer
| sort by p95 desc

In Splunk:

index=wineventlog earliest=-24h
| eval delay_min = (_indextime - _time) / 60
| stats perc50(delay_min) perc95(delay_min) max(delay_min) by host

The 95th percentile per source type is the number that matters: servers in the data center usually arrive within a minute; laptops, branch offices and cloud audit logs do not.

Fix

  • Widen the look-back by the delay you measured, and filter on arrival time so the overlap does not produce duplicate alerts. Microsoft’s pattern: look back over delay + window on TimeGenerated, but only keep events whose ingestion_time() falls within the last window.
  • Use near-real-time or streaming rules for detections where delay matters and the platform offers them.
  • Alert on the delay itself. A source whose delay suddenly jumps from one minute to one hour is a pipeline problem, and pipeline problems are blind spots.

Takeaway

A rule is only as good as the clock it filters on. Measure the delay per source once, write it into the rule’s spec, and recheck it when you add a new kind of source.