← 02 — Articles

Which logs do you actually have?

Coverage9 min read

Every detection rule makes three silent assumptions: the event is recorded, it reaches the place where the rule runs, and it is still there when someone needs to look back. Most teams find out which of those assumptions were wrong during an incident, when the question is “what did this account do last Tuesday” and the answer is a log that rolled over on Wednesday.

A log source inventory is how you find out earlier. It is not glamorous and it does not produce alerts. It is the step that makes every later rule honest about what it can see. This article lays out a method: start from the questions an investigation has to answer, list the sources that should answer them, verify each source on a real system, and rank what is missing. The template that goes with it is log-source-inventory.yml in the armory.

What an inventory is, and what it is not

It is one entry per source type — “Windows domain controllers, Security log”, “perimeter firewall”, “Microsoft 365 unified audit log” — with the facts that decide whether the source is useful: what it records, where it is forwarded, how far back it reaches locally and centrally, how much it produces per day, which clock stamps it, who can change its configuration, and when someone last checked all of that.

It is not an asset inventory or a CMDB. A list of 400 servers tells you nothing about whether any of them logs process creation with the command line. It is also not a SIEM’s list of connectors. A connector that shows “connected” can deliver a fraction of the events you think it does, because the source never generated them.

Before you start

  • Read access to the configuration of a representative system of each type: one domain controller, one member server, one workstation, one Linux server, the firewall, the cloud tenant. Elevated read access, but no changes — every command in this article and in the template is read-only.
  • A rough list of what exists. It does not have to be complete. The inventory will expose the parts you did not know about.
  • A few hours per source type, mostly spent waiting for answers from whoever owns the system.

If you are still choosing a SIEM, do this first: the daily volume per source is also the first number every pricing model asks for, as described in Choosing a SIEM by what you already run.

Step 1: start from questions, not systems

Sources exist to answer questions. Write the questions down first, and you get two checks for free: every source should answer at least one question, and every question should have at least one source. The template starts with six that come up in almost every investigation:

ID Question Typical sources
Q1 Who logged on where, when, and how? Domain controllers, Windows Security on members, Entra ID sign-ins, VPN, Linux auth
Q2 What ran on this machine, with which command line, started by what? Process creation 4688 with command line, Sysmon, EDR, Linux auditd
Q3 What did this machine talk to outside the network? Firewall (allow and deny), DNS queries, web proxy
Q4 What changed in identity and privileges? Domain controllers, Entra ID audit log, cloud control plane
Q5 What was read, sent or shared from mail and SaaS? Microsoft 365 unified audit log, SaaS admin logs
Q6 Who changed the infrastructure? Azure Activity log, AWS CloudTrail, network device configuration logs, sudo

ADAPT the list. A manufacturing network adds questions about engineering workstations and OT gateways; a SaaS-only company may drop half of Q6. Keep the list short enough that you will actually answer every question.

Step 2: list source types, not hosts

Group by type, because configuration is usually applied by type: one Group Policy for domain controllers, one for workstations, one Ansible role for Linux servers. Name hosts only where coverage differs — “all domain-joined Windows, except the kiosk machines outside the domain” is the line that matters.

A workable first cut has six categories: identity, endpoint, server, network, email and SaaS, cloud. Applications come later. They matter, but their logs are rarely usable for detection before the basics are in place.

Step 3: record the facts that decide usefulness

For every source type, the template asks for the same fields. The ones that most often turn out to be wrong:

What is actually recorded. “Windows Security log is collected” says nothing until you know which audit subcategories are enabled. Process creation (4688) without the Include command line in process creation events policy tells you that powershell.exe ran, not what it ran. Firewall logs that cover only denied connections cannot answer Q3 for a connection that was allowed — which is the one you care about.

Local retention. This is how far back the log on the system itself reaches today, measured, not configured. A Security log on a busy domain controller is sized in megabytes and overwrites the oldest events when it is full. If forwarding stops, local retention is all you have.

Central retention, per tier. Hot, searchable storage and cheaper archive tiers usually have different retention and different query speed. Write both down, and the target you need next to them.

Volume per day. Measured from the SIEM or from a week of raw logs. You need it for licensing, for storage, and to notice when a source suddenly produces half of what it used to.

Time. Which clock stamps the event, and in which timezone. Syslog from network devices often carries local time without a timezone. A one-hour shift does not break a rule; it breaks the timeline you build at 3 a.m.

What alerts when collection stops. For most sources the honest answer is “nothing”. A forwarder that stops is silent by definition.

Verified. A date and the method — the command, the query, the test event. An entry without a verification date is a belief, not an inventory.

Step 4: verify on a real system

Every claim in the inventory should come from a command, not from memory or from the original deployment plan. The template lists these checks; all of them are read-only.

On Windows, run elevated:

# Which audit subcategories are enabled
auditpol /get /category:*

# Maximum size and retention behavior of the Security log
wevtutil gl Security

# How far back the local Security log reaches right now
Get-WinEvent -LogName Security -MaxEvents 1 -Oldest | Select-Object TimeCreated

Run the last command on a domain controller during a busy period, not on a quiet Sunday. The difference between the oldest event and now is your real local retention.

On Linux:

auditctl -s                          # is auditd enabled, how many events were lost
auditctl -l                          # loaded rules; "No rules" means nothing beyond defaults
journalctl --disk-usage              # space used by the journal
journalctl -o short-iso | head -n 1  # oldest journal entry still on disk

Then check the other end. In the SIEM, find sources that went quiet. In Microsoft Sentinel:

union withsource=SourceTable *
| where TimeGenerated > ago(7d)
| summarize LastEvent = max(TimeGenerated) by SourceTable
| sort by LastEvent asc

This scans every table, so narrow the time range on large workspaces. In Wazuh, /var/ossec/bin/agent_control -l on the manager lists agents with their status; anything Disconnected or Never connected is a host you believed was covered.

Step 5: turn findings into a ranked gap list

Each gap gets one line in the gaps section: which source, which question goes unanswered, what the fix is, and how expensive the fix is. The cost scale is deliberately coarse:

  • Low — a configuration change: enable an audit subcategory, enable command-line logging, raise a log size.
  • Medium — new collection or storage: forward DNS queries, add a syslog source, extend retention in the SIEM.
  • High — a new product or a project: EDR on servers that have none, log collection from a network you do not manage.

Rank by the question first, cost second. A low-cost gap on Q1 or Q2 — identity and process execution — is almost always the first thing to fix, because most detections and most investigations depend on those two.

Some gaps you will accept: a source is too expensive to collect, or the system is being retired. Mark them accepted and write down why. An accepted gap with a reason is a decision. An unlisted gap is a surprise waiting for an incident.

Step 6: keep it alive

An inventory is a snapshot. Agents get uninstalled, Group Policy gets rewritten, a firewall is replaced and the new one logs to a different collector. Two habits keep the file true:

  1. Review after every change to the SIEM, the agents or the audit policy, and on a fixed schedule — the template defaults to quarterly.
  2. Turn the silent-source check into a detection. “No events from source type X for N hours” is one of the most valuable rules you can write, and it needs no attack knowledge at all. Set N per source from its normal volume: a domain controller that is quiet for an hour is a problem, a branch firewall at night may not be.

Testing your inventory

Verification commands prove the configuration. A test event proves the whole path. For each source type, generate one harmless event you can recognize and follow it end to end:

  1. Generate it. For example, a failed logon to a member server with a dedicated test account and a wrong password. The member server records 4625. With Kerberos authentication, the domain controller records a pre-authentication failure as 4771; with NTLM, it records 4776.
  2. Find it in the SIEM. Search by the test account name, not by event ID, so you also see events you did not expect.
  3. Compare the timestamps. The event time on the host, the event time in the SIEM and the time it arrived. In Sentinel, ingestion_time() - TimeGenerated shows ingestion delay per event. A constant offset of whole hours is a timezone problem; a growing delay is a pipeline problem.
  4. Write the result into verified with the date.

Do this with an account created for testing and agreed with whoever monitors the SIEM, so the failed logons do not start an incident of their own.

Limitations

  • An inventory tells you whether a log exists and arrives, not whether it is parsed correctly. A source can be complete and still useless if its fields land in the wrong place; that is a separate check per rule.
  • Local retention changes with load. A measurement on one day is a lower bound for that day, not a guarantee.
  • Cloud and SaaS retention depends on license and changes over time. The template deliberately does not quote numbers: check the vendor’s current documentation and record the date you checked.
  • The finished inventory is sensitive. It is an accurate map of what you cannot see. Store it like any other internal security document, and never publish a filled-in copy.

Adapting this to your environment

  • ADAPT the questions to your threat model and your industry before listing any source.
  • ADAPT retention_target_days per source: how late your incidents are typically detected, plus any legal or contractual requirement. Write down the reason next to the number.
  • ADAPT the scope for a small team: domain controllers, workstations, the firewall and the Microsoft 365 or Google tenant cover most of Q1 to Q5. Do those four well before adding the fifth.

The template — questions, source fields, gap list and the read-only checks above — is log-source-inventory.yml (CC BY 4.0). Delete the example entries once you have your own. If something here is wrong or missing, write to hello@entropywatch.dev and we will correct it in the open.