Can a log be evidence

An investigation asks a log two questions. The first is easy: is this record genuine, who wrote it and when? The second is much harder: which of these records was deleted?

A plain file or a search index cannot answer the second one. A deleted line leaves no trace — you are left with the survivors, and they look complete. Removing a single document from an Elasticsearch index is a `DELETE /index/_doc/id`, and the index looks perfectly healthy afterwards. In ClickHouse an `ALTER TABLE ... DELETE` mutation behaves the same way: the table is there, and the row somebody did not want seen is not.

For an attacker that is the cheapest move available. On Windows, clearing the event log leaves exactly one event — 1102, "the audit log was cleared" — which is precisely why a competent attacker does not clear the log at all: they take one row out of one index in the SIEM, because no equivalent of 1102 exists there.

So "we keep logs" means nothing to an auditor and nothing to a court. Storage is a claim about existence; evidence is a claim about immutability. They are two different technical requirements, met by two different mechanisms.

Many systems store the log normalised and throw the original text away. That is convenient for search: fields get consistent names, dates become ISO 8601, noise is stripped. For evidence it is fatal — what you can produce is no longer the source but your own interpretation of it.

One example is enough to see the difference. Say nginx logged `GET /a?b=%2e%2e%2f HTTP/1.1` in `$request`. The normaliser decodes it and stores `GET /a?b=../`. You are now showing an attack — but not what the client actually sent. A lawyer will ask about that gap first, because decoding is itself a decision and decisions can be wrong.

The right approach keeps both. The normalised view for search and correlation, the untouched original for production. They are linked to each other, but the original is never rewritten.

There is a second benefit, and in day-to-day work it pays off more often than the first. When a new parsing rule appears, or an old one turns out to be wrong, it is re-applied to the stored original. In a system that discarded the source text the error is permanent: there is nothing to recompute from, and six months of data has to be accepted as-is.

A checksum is computed for every record — SHA-256 in practice, because it is fast and because people take it seriously. Records are gathered into blocks, a block hash is derived, and each block hash includes the previous one.

This is an ordinary hash chain. No blockchain, no network, no tokens. Exactly this mechanism runs in systemd’s journald under `Seal=yes`, where it is called Forward Secure Sealing; Certificate Transparency logs use the same idea as a Merkle tree. It is standard practice rather than an exotic one.

The result is that altering or deleting one record breaks its block hash, which breaks the next, and so on to the end of the chain. Hiding a single line means recomputing the entire history after it.

That is exactly where the value lies: the chain does not make tampering impossible, it makes tampering visible. For evidence, visible is what is required. Nobody can prove a file was never touched; what is provable is that touching it would show.

If the chain sits beside the logs under the same permissions, it is decoration rather than protection. Anyone with root on the log server recomputes both and nothing shows. This is not a theoretical objection: the log server is frequently the first target, precisely because everything is collected in one place.

So block hashes are stored separately: a different system, a different account, a mode where a written value cannot be changed. In practice that means S3 Object Lock in compliance mode, a shelf of WORM media, or simply a machine in another organisation.

Some organisations ship the hashes to a third party as a daily report — to an auditor or a notary. That works too, because what matters is not the copy but the separation of control. Another option is an RFC 3161 timestamping authority, which signs the statement "this hash existed at this moment".

The practical test fits in one question: can a full administrator of the logging system alter the hash journal? If the answer is yes, you do not have integrity control — you have a document about integrity control.

A chain detects nothing by itself. It speaks only when it is checked. An unverified chain is a guarantee that was written down and never read.

Verification therefore has to run in the background and on a schedule: once after a block closes, and regularly across the whole period. The interval matters because a break can sit unnoticed for months. Where a daily full pass is too expensive on a large archive, sample it — a few hundred blocks a day plus the last week in full.

The second requirement is an integrity report per period: which blocks were verified, which one diverged, and when. After an incident the auditor asks for that report, not for the log files. And it has to appear in seconds; a report produced by hand is never produced.

What happens on a mismatch has to be decided in advance too. In many systems it writes one line to a log and stops there. The correct answer is to raise an incident: an integrity failure is an event in its own right and belongs in a queue a person looks at.

Integrity is not only about writing. Who searched, what query they ran, what they exported, who changed the retention policy — all of it belongs in a separate audit trail.

Otherwise the question moves: the log may well be unaltered, but who looked at it and took it outside remains unknown. In internal investigations that second question is frequently the more important one — especially where the SIEM holds customer data, payment traces or staff correspondence.

The most commonly forgotten case is a change of retention. Cutting retention from 90 days to 7 has the same effect as deleting the lines, and no "delete" operation is recorded anywhere: a policy simply ran. Policy changes are part of the audit trail for that reason, and they should require separate approval.

Exports are the same. A downloaded CSV is data leaving the system, and what happens to it next is outside your control. At minimum you need to know who took it, when, how many rows, and under what filter.

The second weak point in any evidence is time. Records arrive from dozens of servers and every clock runs its own way. If you cannot say which of two events happened first, you have a set of events but no sequence.

So every source is pinned to one NTP source — chrony or ntpd in practice, and they log their own state as well. Each record then carries two timestamps: the time the source claimed and the time the system received it. The gap between them is a useful signal in itself: a large skew usually means a broken clock or a record inserted after the fact.

Time zones are a trap of their own. If the record does not carry `+05:00`, you are only assuming which zone it was written in — and that assumption breaks the moment servers move. Everything is stored in UTC; local time is applied at display.

Block boundaries are drawn on time as well. That is what turns "the records between 14:05 and 14:10 were not altered" into a testable statement rather than a request to be believed.

Four questions are enough when choosing a system or a provider. Is the original record kept alongside the normalised one? Where does the hash journal live, and who can change it? On what schedule does verification run, and can you show the report? Are reads and exports recorded?

If the answers are yes, the next step is a demonstration. Pick an arbitrary period and ask for the integrity report covering it. The second request is more useful still: on a test system, alter one record deliberately and time how long it takes the report to say so.

None of these requirements is expensive and none is exotic. Section 10 of PCI DSS and the log-protection control in ISO 27001 ask for the same thing in different words.

They do have to be set at the start of a project, because they cannot be added later: a chain cannot be computed for last year’s logs. A hash computed today protects only what comes after it.