Field Notes

Notes on time zones in log pipelines

· Alex Moreno

I spent an afternoon chasing a bug that turned out to be two services disagreeing about what time it was. One logged in UTC, the other in the server's local zone, and neither wrote an offset.

Rules I now follow

  1. Every machine runs with its clock in UTC, regardless of where it is.
  2. Every timestamp in a log line carries an explicit offset: 2025-12-08T09:14:03Z, never 2025-12-08 09:14:03.
  3. Conversion to a local zone happens only at display time, in the tool a human is looking at.

A quick check

timedatectl
date -u +%Y-%m-%dT%H:%M:%SZ

It is boring advice. It is also the kind of thing that costs a day when it is ignored.

← All posts