Decimal vs Float for Cumulative Production Counters

A cumulative production counter — total parts produced this shift, total kilograms dosed, total kilowatt-hours consumed — is a running sum, and running sums are exactly where IEEE 754 floating point quietly betrays a pipeline. This page narrows precision & rounding limits to a single, common failure: accumulating a counter in float32, watching it look correct for the first hour of a shift, and having it diverge from the true value by the time Availability and Performance are computed. The error is not a rare edge case; it is the deterministic, reproducible consequence of adding a small increment to a value whose binary representation cannot exactly hold most decimal fractions, repeated thousands of times.

The float32 accumulation-drift failure #Permalink to this section

float32 gives roughly seven significant decimal digits; float64 gives roughly fifteen to seventeen. Both are binary fractions, so a value like 0.1 — one energy-meter tick, one dosing increment — has no exact binary representation and is stored as the nearest representable approximation. Adding that approximation to a running total tens of thousands of times per shift does not average the rounding error away; the direction of the error is a deterministic function of the bit pattern, and it accumulates monotonically in one direction far more often than it cancels.

import numpy as np


def demonstrate_float32_drift(increment: float = 0.1, n_samples: int = 28_800) -> None:
    """28,800 samples ~= one increment per second across an 8-hour shift.

    Compares a float32 running sum against exact Decimal arithmetic and
    reports where the two diverge by a full unit.
    """
    from decimal import Decimal, getcontext
    getcontext().prec = 30

    total_f32 = np.float32(0.0)
    total_exact = Decimal("0")
    inc_exact = Decimal(str(increment))
    first_full_unit_error_at: int | None = None

    for i in range(1, n_samples + 1):
        total_f32 = np.float32(total_f32 + np.float32(increment))
        total_exact += inc_exact
        error = abs(Decimal(str(float(total_f32))) - total_exact)
        if first_full_unit_error_at is None and error >= Decimal("1.0"):
            first_full_unit_error_at = i

    print(f"float32 total after {n_samples} samples: {total_f32}")
    print(f"exact total: {total_exact}")
    print(f"absolute error: {abs(Decimal(str(float(total_f32))) - total_exact)}")
    print(f"crossed 1.0-unit error at sample: {first_full_unit_error_at}")


# Typical output on this workload (exact total is 2880.000 kWh):
# float32 total after 28800 samples: 2880.4592...
# absolute error: ~0.46 kWh after a full shift, growing with every additional sample

A float32 counter that is never reset — a lifetime production total, an odometer-style energy meter — has an even harsher failure mode than gradual drift: once its magnitude exceeds $2^{24}$ (16,777,216), the 24-bit mantissa can no longer represent every integer exactly, and small increments stop changing the stored value at all. A part-count totalizer that reaches that magnitude does not merely drift; it silently freezes while production continues underneath it. The failure is worse than a cosmetic rounding artifact once the counter feeds OEE math: Performance is (Ideal Cycle Time × Total Count) / Run Time, and a Total Count that has silently drifted biases every subsequent shift’s ratio, not just the number displayed on a dashboard. Because the drift is systematic rather than random, it does not wash out in a weekly rollup — it compounds, which is the same class of problem addressed for wall-clock time in handling floating-point drift in sensor readings.

Integer counters and rollover handling #Permalink to this section

The cheapest, fastest, and most auditable fix for a discrete counter — parts produced, cycles completed — is to never use floating point at all. Store the raw count as an integer and let it be exactly representable, because int64 addition has no rounding error by construction. The two production concerns are rollover (PLC counters are commonly 16-bit or 32-bit and wrap) and monotonicity (a counter reset mid-shift must not read as negative production).

from dataclasses import dataclass


@dataclass
class RolloverAwareCounter:
    """Reconstructs a monotonic total from a wrapping PLC register.

    register_bits: bit width of the source register (commonly 16 or 32).
    """
    register_bits: int
    _last_raw: int | None = None
    _total: int = 0

    @property
    def _modulus(self) -> int:
        return 1 << self.register_bits

    def ingest(self, raw_value: int) -> int:
        """Feed the next raw register read; returns the reconstructed total."""
        if not (0 <= raw_value < self._modulus):
            raise ValueError(f"raw_value {raw_value} outside register range")

        if self._last_raw is None:
            self._total = raw_value
        else:
            delta = raw_value - self._last_raw
            if delta < 0:
                # Register wrapped (or was reset). Distinguish the two:
                # a wrap adds (modulus + delta); a hard reset restarts at raw_value.
                plausible_wrap = self._modulus + delta
                if plausible_wrap < self._modulus // 2:
                    # Small forward delta after wrap: trust the wrap interpretation.
                    self._total += plausible_wrap
                else:
                    # Large negative jump: treat as an operator/PLC reset, not a wrap.
                    self._total += raw_value  # count from zero again, don't invent history
            else:
                self._total += delta

        self._last_raw = raw_value
        return self._total

int64 gives headroom for roughly 9.2 quintillion counts — no realistic production counter reaches that ceiling within a retention window — so once the wrap logic is correct, the accumulator itself is exact for the life of the asset. This is also the representation to persist in the time-series database: a BIGINT column for the reconstructed total, with the raw register value kept alongside for audit, mirrors the immutable-raw-plus-corrected-value pattern used for clock drift correction.

Decimal and fixed-point for rates and totals #Permalink to this section

Integers handle discrete counts cleanly, but many totalizers are not counts — energy in kWh, mass in kg, volume in liters — where the increment itself is fractional. Here the correct primitive is Python’s decimal.Decimal, which represents decimal fractions exactly (subject to a configured precision) and performs arithmetic without binary rounding error, at the cost of being slower than native floats and requiring explicit quantization discipline.

from decimal import Decimal, ROUND_HALF_EVEN, getcontext

getcontext().prec = 28  # ample headroom for a shift-long accumulation


class DecimalTotalizer:
    """Exact accumulation for fractional totals (energy, mass, volume).

    Quantizes to a fixed number of decimal places on every read so the
    stored precision matches the sensor's true resolution, not the
    accumulator's internal working precision.
    """

    def __init__(self, places: int = 3):
        self._quant = Decimal(1).scaleb(-places)  # e.g. Decimal('0.001')
        self._total = Decimal("0")

    def add(self, increment: str | Decimal) -> Decimal:
        # Construct from str, never from float, to avoid importing binary
        # rounding error before the Decimal arithmetic even starts.
        inc = increment if isinstance(increment, Decimal) else Decimal(increment)
        self._total += inc
        return self._total

    def read(self) -> Decimal:
        return self._total.quantize(self._quant, rounding=ROUND_HALF_EVEN)


totalizer = DecimalTotalizer(places=3)
for _ in range(28_800):
    totalizer.add("0.1")   # string literal — never Decimal(0.1), which imports the float error
print(totalizer.read())    # exactly 2880.000, matching the true physical total

The Decimal(increment) constructor called on a float (Decimal(0.1)) reproduces the exact binary approximation of 0.1 rather than fixing it — the error has already happened by the time the float literal was created. Always construct from the source’s string or integer representation, ideally the raw scaled-integer payload described in handling floating-point drift in sensor readings, never from a Python float literal.

For extremely high-frequency accumulators where Decimal’s per-operation cost is measurable (millions of increments per second, uncommon on a factory floor but real in some analog-integration scenarios), scaled fixed-point integer arithmetic gives the same exactness with native-integer speed: store the value as int64 milli-units (multiply by 1000, round once at the boundary) and only convert to a display Decimal when reading out.

Gotchas & anti-patterns #Permalink to this section

• Accumulating in float32 “because the sensor is float32.” The sensor’s native precision has nothing to do with the accumulator’s required precision. Cast to Decimal or scaled int64 at ingestion, before any addition happens.
• Decimal(0.1) instead of Decimal("0.1"). Constructing Decimal from a float literal imports the float’s binary rounding error verbatim; the whole point of switching to Decimal is lost.
• Treating a rollover as a hard reset. Miscounting a 16-bit register wrap as a shift-ending reset silently erases legitimate production near the register boundary — distinguish the two by delta magnitude, not by assuming every drop is a fault.
• Mixing float and Decimal in the same expression. Decimal("1.0") + 0.1 raises TypeError in Python by design; if a shim silently coerces instead, it reintroduces the binary error the Decimal type exists to prevent.
• Storing totalizers as DOUBLE PRECISION in the TSDB. Even if the accumulation was exact in application code, persisting the result as a database float re-quantizes it on every write/read round trip; use NUMERIC/DECIMAL or a scaled BIGINT column for anything that feeds OEE formula validation.

Quick reference #Permalink to this section

Related #Permalink to this section

• Precision & Rounding Limits — the parent contract for deterministic numeric handling
• Handling Floating-Point Drift in Sensor Readings — the same IEEE 754 root cause applied to instantaneous values
• OEE Formula Validation — where a drifted counter silently biases Availability and Performance
• Time-Series Database Sync — column-type choices for persisting totalizers without reintroducing drift