← 03 — Blog

Clock skew breaks your incident timeline before it breaks Kerberos

IR2 min read

Every investigation ends up as a timeline: this logon, then that process, then a connection out. The timeline is only as good as the clocks that stamped the events. A server that runs four minutes slow still authenticates fine, because Kerberos tolerates up to five minutes of skew by default, and it quietly reorders your story.

Symptom

A timeline shows a process starting on a server before the logon that launched it. Or a firewall connection appears two minutes before the DNS query that should have preceded it. The events are real; their order is wrong.

Cause

Two different problems produce the same symptom:

  1. The source clock is wrong. In a domain, members sync from domain controllers, domain controllers sync from the domain controller holding the PDC emulator role, and that one syncs from an external source. Break the chain — a virtual machine that takes time from its hypervisor host instead, a PDC emulator pointed at nothing, a server outside the domain on its own NTP — and the clock drifts. Kerberos only complains past the Maximum tolerance for computer clock synchronization (five minutes by default), with failure code 0x25 (KRB_AP_ERR_SKEW) in Kerberos events, or status 0xC0000133 in 4625. Below that, nothing fails and nothing warns.
  2. The time zone is interpreted wrong. Windows events carry UTC (TimeCreated SystemTime), but syslog from network devices often carries local time without a zone. If the collector guesses wrong, a whole source is shifted by one or two hours — twice a year by a different amount.

Check

On a Windows machine:

w32tm /query /source     # where this machine takes time from
w32tm /query /status     # last successful sync, stratum, source
w32tm /stripchart /computer:DC01.corp.example /samples:5 /dataonly

/stripchart prints the offset against another machine; run it against a domain controller and against your reference source. Local CMOS Clock or VM IC Time Synchronization Provider as the source on a domain member is worth a second look.

On Linux, chronyc tracking (or timedatectl timesync-status with systemd-timesyncd) shows the offset and the source.

For the time zone problem, pick one event you can trigger on a device — a failed login on a firewall’s admin page — and compare the timestamp your SIEM shows with the clock on your wall.

Fix

  • Restore the hierarchy: domain members on NT5DS, the PDC emulator on a reliable external source, hypervisor time sync disabled for domain controllers so they don’t fight the domain hierarchy.
  • Normalize every source to UTC at ingestion, and record the original timestamp alongside the parsed one.
  • Log clock changes: event 4616 (The system time was changed) comes from the Security State Change audit subcategory. A large jump on a server nobody touched is worth an alert of its own, because changing the clock is also a way to confuse an investigation.

Takeaway

Five minutes of skew is invisible to authentication and fatal to a timeline. Check clocks before you build one, and note the offset per source in the case file so you can correct for it instead of arguing about it.