Syncing Edge Timestamps with NTP Servers for IIoT Pipelines

Reliable clock synchronization is one concrete prerequisite inside time-series database sync: if an edge gateway’s clock drifts even a few tens of milliseconds from the central historian, the persistence layer’s idempotency and ordering guarantees quietly fail. Out-of-order writes fragment the sliding windows that feed availability math, broker redeliveries land before the events they supersede, and a daylight-saving step turns a continuous production run into a phantom downtime interval. This page covers how to assign NTP sources at the edge, how to enforce slew-only correction with chrony, how to quarantine telemetry whose timestamps fall outside a drift budget, and how to normalize heterogeneous PLC clocks to a single UTC epoch — the four decisions that keep an Overall Equipment Effectiveness (OEE) pipeline computing on a coherent timeline.

NTP Source Selection and Stratum Design at the Edge Gateway #Permalink to this section

Industrial networks span several synchronization strata. The enterprise core usually disciplines its clocks against a stratum 1 reference (a GPS-fed or atomic appliance) or a stratum 2 upstream; the edge gateway should then act as a stratum 3–4 local server, redistributing time to field devices across an isolated OT VLAN rather than letting every PLC reach out to the internet. The single most damaging mistake here is querying one source: with no peer to cross-check, a single falseticker is followed blindly. Configure at least three sources so the selection algorithm can discard the outlier.

Asymmetric routing is the quiet enemy of NTP accuracy. The protocol assumes the request and reply traverse paths of equal delay; industrial switches with deep QoS buffers, stateful firewalls, and one-way traffic shaping break that assumption and bias the computed offset by half the path asymmetry. When asymmetry on the OT VLAN exceeds roughly 15 ms, NTP’s offset estimate degrades past what a ±100 ms ingestion budget can absorb, and the correct answer is hardware-assisted Precision Time Protocol (PTP, IEEE 1588) with boundary or transparent clocks — not more NTP tuning. Keep the same canonical addressing you established during PLC tag standardization so each synchronized device maps cleanly to its ISA-95 work unit when its samples reach the store.

# /etc/chrony/chrony.conf — edge gateway acting as a local stratum-4 server
# Three enterprise sources so the selection algorithm can reject a falseticker.
server ntp1.enterprise.local iburst minpoll 6 maxpoll 10
server ntp2.enterprise.local iburst minpoll 6 maxpoll 10
server ntp3.enterprise.local iburst minpoll 6 maxpoll 10

# Reject any source whose root distance exceeds the ingestion budget.
maxdistance 0.05            # 50 ms — sources beyond this are not trusted

# Redistribute time only to the OT subnet; never expose NTP to the enterprise side.
allow 10.20.0.0/16
local stratum 4

# Persist drift so the oscillator is pre-compensated across reboots.
driftfile /var/lib/chrony/drift

Slew-Only Correction: Why You Must Disable Step Adjustments #Permalink to this section

A time-series database treats a backward clock jump as a stream of out-of-order rows. If chrony (or ntpd) is allowed to step the clock — to discontinuously set it to the correct value — a 200 ms correction can manufacture 200 ms of duplicate or missing timestamps, and a backward step can make a freshly written row appear older than rows already committed. On TimescaleDB or InfluxDB that surfaces as compaction conflicts, broken continuous aggregates, and OEE windows that double-count or skip cycles. The production contract is therefore slew-only: speed the clock up or slow it down gradually until it converges, never jump it.

The one defensible exception is the very first measurement after boot, when the oscillator may be wildly off and slewing would take hours. makestep 1.0 3 permits a step only if the offset exceeds one second, and only for the first three updates; after that, corrections are slew-only forever.

# Slew-only in steady state; allow a single large step only at startup.
makestep 1.0 3             # step only if offset > 1 s, first 3 updates only
maxslewrate 500            # cap slew at 500 ppm so corrections stay smooth

# Emit measurement, statistics, and tracking logs for drift correlation.
logdir /var/log/chrony
log measurements statistics tracking

Verify the daemon actually settled into slew-only behavior before trusting the line’s data:

# 'Leap status : Normal' and a small, stable 'System time' offset confirm
# the gateway is disciplined and not mid-step.
chronyc tracking
chronyc sources -v        # each source's offset, jitter, and round-trip delay

Leap seconds deserve the same caution as DST: a hard leap insertion injects a duplicate second that stalls strictly-ordered ingestion. Point the gateway at leap-smear sources that spread the correction across hours per the NTP timescale described in RFC 5905, so the smear is just a slow slew the TSDB never notices. Aligning that smear across plants in different regions is exactly the problem handled by correcting timezone shifts across global plants.

Drift-Tolerance Windows and Quarantine at Ingestion #Permalink to this section

Even a well-disciplined gateway drifts between corrections, and during a network partition it free-runs on its local oscillator. So the ingestion boundary must defend itself with an explicit drift budget rather than trusting every incoming timestamp. Compare each payload’s acquisition time against the gateway’s synchronized clock; values inside the tolerance window are written, values outside it are quarantined for reconciliation instead of silently corrupting an aggregate. This check is the temporal sibling of the value-sanity work done in precision and rounding limits, and it must run before any clock drift correction attempts to realign a free-running stream.

from dataclasses import dataclass
from enum import Enum


class TimestampVerdict(Enum):
    ACCEPT = "accept"
    QUARANTINE = "quarantine"


@dataclass(frozen=True)
class DriftCheck:
    """Validate edge acquisition time against the gateway's synced clock.

    tolerance_ms is the drift budget; it must be no larger than the TSDB's
    out-of-order tolerance, or quarantined rows still leak into aggregates.
    """

    tolerance_ms: int = 100

    def evaluate(self, acq_ms: int, gateway_ms: int) -> TimestampVerdict:
        skew = abs(acq_ms - gateway_ms)
        if skew > self.tolerance_ms:
            return TimestampVerdict.QUARANTINE
        return TimestampVerdict.ACCEPT

During a partition, an edge device should fall back to a hardware real-time clock with a documented drift coefficient (commonly ±20 ppm, about ±72 ms per hour) and tag every sample produced in that state so the historian knows it is RTC-derived, not NTP-disciplined. When the link returns and chrony re-converges, those marked windows can be re-timestamped or handed to bounded gap-filling algorithms — for instance linear interpolation for missing sensor values — rather than written as if they were trustworthy live data.

Normalizing Heterogeneous PLC Clocks to UTC Epoch Milliseconds #Permalink to this section

A factory rarely speaks one timestamp dialect. OPC UA servers expose ISO 8601 strings with sub-millisecond precision, Modbus TCP registers often truncate to whole seconds, and legacy serial devices may carry no fractional time at all — while many controllers stamp updates with a free-running local wall clock that drifts independently of the gateway. The durable pattern is to strip the PLC-local timestamp at the ingestion boundary, replace it with a gateway-synchronized UTC value, and carry both the acquisition and publish instants in the payload envelope so consumers can separate true process time from broker transit latency. That envelope is what the MQTT topic hierarchy transports, and keeping QoS 1 redeliveries in order downstream depends on these timestamps being monotonic.

Two precision rules keep the normalization deterministic. Never call datetime.now() without an explicit timezone — a naive local timestamp shifts under DST and leap handling. And when downscaling sub-millisecond precision to milliseconds, use floor division rather than banker’s rounding, so high-frequency sampling does not accumulate a half-tick of drift per sample.

import time
from typing import Optional


def normalize_plc_timestamp(
    raw_ts: Optional[float],
    source_precision: str = "seconds",
) -> int:
    """Normalize a heterogeneous PLC timestamp to UTC epoch milliseconds.

    Floor-divides sub-millisecond precision (no banker's rounding) to avoid
    cumulative drift. Falls back to the gateway's synced clock when the PLC
    supplied no usable time.
    """
    if raw_ts is None:
        return int(time.time() * 1000)  # gateway clock is the source of truth

    if source_precision == "microseconds":
        return int(raw_ts) // 1_000
    if source_precision == "milliseconds":
        return int(raw_ts)
    if source_precision == "seconds":
        return int(raw_ts * 1_000)
    raise ValueError(f"Unsupported precision: {source_precision!r}")

For interval measurements — polling cycle duration, debounce windows — reach for time.monotonic_ns() instead, and reserve wall-clock time.time_ns() strictly for absolute event timestamps. The monotonic clock never goes backward across an NTP slew, so it is the only safe basis for measuring elapsed time; see the Python time module documentation for the per-OS guarantees.

Gotchas and Anti-Patterns #Permalink to this section

• Stepping the clock in production. Leaving makestep unbounded lets a routine correction jump the clock backward and inject out-of-order rows. Cap it to startup only and slew everywhere else.
• A single NTP source. With no peer to outvote it, one falseticker is followed without question. Always configure three or more, and set maxdistance to your ingestion budget.
• Ignoring path asymmetry. NTP assumes symmetric delay; a one-way QoS-shaped OT link can bias the offset by tens of milliseconds. Past ~15 ms asymmetry, move to PTP rather than tuning NTP harder.
• Trusting PLC wall-clock timestamps. Controller clocks drift independently and rarely see NTP. Restamp at the gateway and keep acquisition and publish times as separate fields.
• A drift budget looser than the TSDB. If your quarantine window exceeds the database’s out-of-order tolerance, “accepted” rows still corrupt continuous aggregates. Make the budget no larger than the store allows.
• Naive datetime.now() and leap-second-blind daemons. Both reintroduce discontinuities the rest of the pipeline works to eliminate; pin timezones and use leap-smear sources.

Quick Reference: Time-Sync Decision Matrix #Permalink to this section

Related #Permalink to this section

• Time-Series Database Sync for Manufacturing Telemetry — the parent topic this synchronization step feeds
• Correcting Timezone Shifts Across Global Plants — apply UTC discipline across multi-region sites
• Implementing Linear Interpolation for Missing Sensor Values — reconstruct windows lost during a partition
• Best Practices for MQTT QoS Levels in Factory Networks — keep redeliveries ordered by accurate timestamps
• Handling Floating-Point Drift in Sensor Readings — the value-precision counterpart to temporal precision