Handling Floating-Point Drift in Sensor Readings

Floating-point drift in manufacturing telemetry is rarely a mathematical curiosity; it is a deterministic pipeline failure mode. When analog process variables traverse from PLC analog input modules through edge gateways into cloud aggregation layers, IEEE 754 binary representation introduces cumulative rounding artifacts that silently corrupt cycle-time baselines, throughput counters, and quality-yield thresholds. This is a focused problem within Precision & Rounding Limits, the precision subsystem of Core Architecture & Data Mapping: here we deal with the specific case where a value such as 12.45 arrives as 12.450000000000001 and trips a hard threshold it should never have crossed. For industrial engineers and IIoT developers, unmanaged drift directly skews the Availability, Performance, and Quality terms of OEE, producing phantom equipment stops, false threshold crossings, and inconsistent run-to-run yield reporting.

The defensible fix is not a single rounding call buried in a dashboard query. It is an architecture that isolates the serialization boundary, transmits exact values across MQTT, and enforces deterministic quantization before telemetry reaches the OEE calculation engine. The sections below cover the root cause and the three mitigation strategies that actually hold up at factory throughput.

Where floating-point drift enters the pipeline and the two points where it must be fixed Horizontal flow from a 16-bit ADC through edge EU scaling, JSON serialization, the MQTT broker and a time-series database downsample. A precision band shows the value is exact until the JSON serialize hop, where binary residue appears and is then carried and compounded. Two interventions are annotated: a scaled-integer transmit at the edge that prevents the float from ever existing, and a single deterministic round at ingestion that repairs the value before persistence and comparison. OT · EDGE GATEWAY IT · TRANSPORT → STORE 16-bit ADC raw count 12450 integer · exact Edge EU Scale × 0.001 → 12.45 bar fixed-point · exact JSON Serialize IEEE 754 double 12.450000000000001 MQTT Broker transports as-is drift carried TSDB Downsample rolling-window resample drift compounds drift enters here exact integer / fixed-point binary residue carried & compounded FIX 1 · at the edge send scaled int + decimal scale — no float ever on the wire FIX 2 · at ingestion one deterministic round, before persist: Decimal(str(x)) · int64
The value is exact until the JSON serialize hop coerces it into an IEEE 754 double; from there drift is carried by the broker and compounded at downsample. Fix it at the edge by never emitting a float, or at ingestion with a single deterministic round before anything persists or compares.

Where the drift enters: fixed-point PLCs vs. IEEE 754 serialization

The primary driver of drift is the impedance mismatch between PLC native fixed-point scaling and cloud-native floating-point expectations. Most mid-tier and legacy PLCs store analog inputs as scaled integers — 16-bit raw counts mapped to engineering units (EU) through a linear scale factor, EU = (Raw - Offset) × Scale. This integer-and-scale convention is exactly what disciplined PLC tag standardization is meant to preserve end to end. The trouble starts when MQTT topic hierarchies serialize those values as JSON floats for cross-platform interoperability.

During that conversion, decimal fractions that cannot be exactly represented in binary — 0.1, 0.2, 0.3 and most non-power-of-two fractions — pick up representation error. For float32 the error floor is roughly 1e-7 of the magnitude; for float64 it is roughly 1e-15. Each hop that re-parses and re-serializes the value can re-quantize it. By the time the payload reaches a time-series database sync process, the drift compounds across partition boundaries, particularly during high-frequency resampling or rolling-window aggregation. A vibration sensor reading 12.450000000000001 instead of 12.45 is harmless until it crosses a hard-coded > 12.45 comparison in a state machine and triggers an unwarranted line stop.

The error is bounded but real. A reading rounded into float32 near 12.45 carries an absolute uncertainty on the order of ε≈2−23⋅23≈1×10−6\varepsilon \approx 2^{-23} \cdot 2^{3} \approx 1\times10^{-6}, which is why a tolerance band of ±1e-6 is the practical detection floor for single-precision payloads and why exact equality or hard >/< comparisons on raw floats are unsafe.

Option 1 — Transmit scaled integers at the edge

The cleanest mitigation removes the float from the wire entirely. Instead of letting the gateway emit {"value": 12.450000001}, transmit the raw scaled integer plus the metadata needed to reconstruct the engineering unit deterministically downstream. A well-structured topic tree keeps raw telemetry, aggregates, and scaling metadata on separate branches:

factory/line_01/station_04/raw/temperature_c
factory/line_01/station_04/agg/temperature_c_1m
factory/line_01/station_04/meta/temperature_c_scale

The payload carries the integer count and an exact decimal scale factor, never the pre-multiplied float:

{
  "ts": 1718452800000,
  "v": 124500,
  "scale": "0.0001",
  "unit": "degC",
  "precision": "float32"
}

The consumer reconstructs the value with exact decimal arithmetic, so 124500 × 0.0001 resolves to 12.4500 with no binary residue. Sending scale as a decimal string rather than 1e-4 avoids reintroducing a float at the very step you are trying to protect. This pattern is the strongest defence because it eliminates the lossy hop instead of trying to repair its output, and it pairs naturally with discrete-state telemetry delivered over QoS 1 to avoid duplicate event counters.

from decimal import Decimal

def decode_scaled_payload(payload: dict) -> Decimal:
    """Reconstruct an engineering-unit value from a scaled-integer MQTT payload.

    Keeps arithmetic in Decimal so the integer count and string scale factor
    never touch IEEE 754 binary representation.
    """
    raw = int(payload["v"])
    scale = Decimal(str(payload["scale"]))
    return Decimal(raw) * scale  # e.g. 124500 * Decimal("0.0001") -> Decimal("12.4500")

Option 2 — Deterministic rounding with Python’s decimal module

When you cannot control the edge firmware and floats are already on the wire, enforce a deterministic round at the ingestion boundary — before the value is persisted or compared. Python’s decimal module sidesteps binary representation and lets you pin both precision and rounding mode, so every node in the pipeline produces byte-identical results.

from decimal import Decimal, ROUND_HALF_EVEN, getcontext
import pandas as pd

# Configure a single, explicit context for manufacturing precision.
getcontext().prec = 18
getcontext().rounding = ROUND_HALF_EVEN  # banker's rounding: unbiased over many samples

def quantize_telemetry_series(series: pd.Series, precision: int = 4) -> pd.Series:
    """Apply deterministic rounding to a Series of sensor readings.

    Converts via str() so the *displayed* float (12.45), not its binary
    expansion (12.450000000000001), is what gets quantized.
    """
    quantizer = Decimal(10) ** -precision
    return series.apply(lambda x: Decimal(str(x)).quantize(quantizer))

raw_df = pd.DataFrame({"temp_c": [12.450000001, 12.449999999, 12.450000002]})
raw_df["temp_c_clean"] = quantize_telemetry_series(raw_df["temp_c"], precision=4)

Two details matter. First, Decimal(str(x)) rather than Decimal(x) — the former captures the human-meaningful value, the latter faithfully reproduces the binary error you are trying to remove. Second, ROUND_HALF_EVEN (banker’s rounding) is the correct default for SPC and OEE: ROUND_HALF_UP introduces a small positive bias that accumulates into inflated performance rates over millions of samples. Choose precision from the sensor’s physical resolution, not the database’s column width — see the precision budget in the parent Precision & Rounding Limits reference.

Option 3 — Fixed-point integer emulation for high-throughput aggregation

Decimal is correct but slow; at hundreds of thousands of rows per second it becomes the bottleneck. For hot paths, emulate fixed-point with int64 arithmetic and keep aggregation entirely in integer space:

import numpy as np
import pandas as pd

def to_fixed_point(series: pd.Series, decimals: int = 4) -> np.ndarray:
    """Scale floats to int64 fixed-point. Rounds at the boundary so the
    integer count is exact; all downstream sum()/count()/compare stay exact.
    """
    factor = 10 ** decimals
    return np.rint(series.to_numpy() * factor).astype(np.int64)

def aggregate_fixed_point(scaled: np.ndarray, decimals: int = 4) -> dict:
    """Aggregate in integer space, divide back to float only for reporting."""
    factor = 10 ** decimals
    total = int(scaled.sum())          # exact integer sum, no drift
    count = int(scaled.size)
    return {
        "sum": total / factor,
        "mean": (total / count) / factor if count else 0.0,
        "count": count,
    }

readings = pd.Series([12.450000001, 12.449999999, 12.450000002])
agg = aggregate_fixed_point(to_fixed_point(readings))

The rule is: round once, at the int64 boundary, then never touch a float again until the visualization or reporting layer. sum(), count(), and threshold comparisons in integer space are exact by construction, which completely neutralizes drift in the inputs to OEE. The same discipline underpins downstream OEE formula validation and prevents off-by-epsilon errors when tuning thresholds for microstops, where a single false threshold crossing can register a phantom stop.

Watch the dynamic range: an int64 holds about 9.2×10¹⁸, so four decimal places leaves headroom up to ~9.2×10¹⁴ engineering units — ample for process variables, but verify before scaling counters with very large magnitudes.

Diagnosing drift at the serialization boundary

To prove where drift originates rather than guessing, log both the raw payload bytes and their IEEE 754 hex representation at the deserialization point. A lightweight interceptor flags any value that exceeds tolerance before it reaches the broker.

import struct
import numpy as np

def inspect_float_drift(raw_bytes: bytes, expected: float, tol: float = 1e-6) -> dict:
    """Decode a 4-byte float32 payload and compare its IEEE 754 form against
    the expected engineering value, surfacing the exact hex bit pattern."""
    val = struct.unpack("!f", raw_bytes)[0]                 # network byte order
    hex_repr = f"0x{np.float32(val).view(np.uint32):08X}"
    drift = abs(val - expected)
    return {
        "raw_hex": hex_repr,
        "deserialized_f32": val,
        "expected": expected,
        "absolute_drift": drift,
        "exceeds_tolerance": drift > tol,
    }

print(inspect_float_drift(struct.pack("!f", 12.45), 12.45))

If the hex pattern is already wrong at the gateway, the PLC scaling or edge serialization is the culprit; if it is clean at the edge but drifts after a TSDB round-trip, the aggregation layer is re-quantizing. This is the same boundary-inspection discipline used when syncing edge timestamps with NTP servers — instrument the exact hop before assigning blame.

Gotchas & anti-patterns

  • Rounding only at the dashboard. Cosmetic rounding in a Grafana query leaves drifted values in storage, so alerts, continuous queries, and state machines still compare against the dirty number. Round at ingestion, before persistence.
  • Decimal(x) instead of Decimal(str(x)). Passing a float straight into Decimal faithfully copies its binary error — exactly the artifact you meant to remove.
  • ROUND_HALF_UP on SPC data. The default-feeling choice adds a tiny positive bias that compounds into inflated performance rates across millions of samples; use ROUND_HALF_EVEN.
  • Exact equality or hard >/< on raw floats. A > 12.45 threshold on un-quantized telemetry fires on 12.450000001. Compare quantized values, or use an explicit tolerance band.
  • Re-serializing through intermediate float hops. Every gateway that parses and re-emits a value can re-quantize it. Carry scaled integers, or quantize once and mark the tag as normalized so downstream stages do not “helpfully” round again.

Quick-reference: choosing a mitigation

Strategy Best when Throughput cost Exactness Key caveat
Scaled-integer edge payload You control the gateway firmware Lowest (no float on wire) Exact Requires scale/unit metadata contract
decimal deterministic round Floats already on the wire; correctness over speed High (per-value Python) Exact to set precision Must use Decimal(str(x)) and ROUND_HALF_EVEN
int64 fixed-point aggregation Hot path, millions of rows/sec Low (vectorized integer math) Exact for sum/count/compare Mind int64 dynamic range
Hex/byte interceptor Diagnosing an existing drift bug Logging only (sampled) n/a (diagnostic) Adds I/O; sample, don’t log every payload

Treating precision as a first-class pipeline property — set at the edge, enforced at ingestion, preserved through aggregation — turns floating-point drift from an intermittent mystery into a closed engineering constraint, and keeps OEE reporting deterministic and audit-ready.