Precision & Rounding Limits for Manufacturing Telemetry Pipelines

Precision and rounding limits define the mathematical boundary between actionable operational intelligence and systemic noise, and getting them wrong is one of the most common ways an otherwise sound telemetry stack silently corrupts its own metrics. This is one subsystem of Core Architecture & Data Mapping: where PLC tag standardization decides what a value means and MQTT topic hierarchies decide where it travels, precision control decides how exactly it is allowed to be represented at every hop. These limits are not software preferences; they are hard physical constraints dictated by analog-to-digital converter (ADC) bit depth, PLC scan-cycle resolution, and network serialization tolerances. When unmanaged, IEEE 754 floating-point representation becomes a silent source of cumulative drift that distorts availability tracking, performance-rate calculations, and quality-yield metrics. Establishing a deterministic rounding contract at ingestion ensures downstream analytics operate on mathematically consistent baselines rather than stochastic approximations.

Precision Budget Reference #Permalink to this section

Every sensor carries an inherent quantization error set by its native hardware resolution. A 16-bit analog input module sampling a 4–20 mA pressure loop resolves roughly 0.0015% of full scale per count; it cannot legitimately support six decimal places in a cloud database. The table below maps common acquisition tiers to the maximum decimal precision that is physically defensible — anything beyond it is false granularity that misleads statistical process control (SPC), inflates storage, and injects phantom variance into OEE.

The rule the rest of this page enforces: declared precision must never exceed the source tier’s defensible decimals, and the round that imposes it must happen once, at ingestion.

Core Concept and Design Contract #Permalink to this section

Precision must be treated as a controlled variable carried in tag metadata, not an emergent property of whichever language last touched the value. The design contract has three invariants:

• Single rounding authority. The deterministic round is applied exactly once, at the first transformation step (edge or ingestion), and never re-applied at query time. Query-time rounding masks historical inconsistency and makes two dashboards disagree on the same interval.
• Precision travels with the value. Each tag declares max_decimal_places, scale, and rounding_mode (see PLC tag standardization). A reading without its precision contract is treated as untrusted, not as full-precision.
• Fixed-point for accumulation. Any value that will be summed, integrated, or compared against a hard threshold is converted to integer micro-units before the operation. This is the only reliable defense against the IEEE 754 caveats that make 0.1 + 0.2 != 0.3.

PLC tag resolution sets the ceiling on meaningful precision. Integer-based scaling factors, IEEE 754 single-precision (REAL) representations, and proprietary vendor types each impose distinct rounding behavior, and real lines mix them on a single bus:

• Siemens S7: REAL (32-bit IEEE 754) versus LREAL (64-bit) must be cast explicitly at the gateway to avoid silent truncation — see mapping Siemens S7 tags to OPC UA.
• Rockwell ControlLogix: REAL tags are frequently scaled via CPT instructions; preserve raw integer counts until the final engineering-unit conversion.
• Modbus / OPC UA: register mapping must declare DataType, ScaleFactor, Offset, and DecimalPrecision in the tag dictionary so the consumer never guesses.

The contract aligns with the IEEE Standard for Floating-Point Arithmetic (IEEE 754): the standard guarantees cross-platform bit-level consistency, but it does not give you engineering precision — that layer is yours to impose on top.

Implementation #Permalink to this section

The following ingestion component imposes the contract on the 80% case: it converts immediately to Decimal to avoid binary float drift, validates against NaN/Inf, applies the declared rounding mode, and rejects values that exceed the source’s physical resolution. It uses Python’s decimal module (reference) rather than round(), which is itself subject to float representation error.

import logging
from dataclasses import dataclass
from decimal import Decimal, ROUND_HALF_UP, ROUND_HALF_EVEN, InvalidOperation
from typing import Optional

logger = logging.getLogger(__name__)

_ROUNDING = {"HALF_UP": ROUND_HALF_UP, "HALF_EVEN": ROUND_HALF_EVEN}


@dataclass(frozen=True)
class TagMetadata:
    tag_id: str
    engineering_unit: str
    max_decimal_places: int           # ceiling from the precision-budget table
    rounding_mode: str = "HALF_EVEN"  # banker's rounding minimises accumulation bias
    fs_min: Optional[Decimal] = None  # physical lower bound of the source
    fs_max: Optional[Decimal] = None  # physical upper bound of the source


def round_telemetry_value(
    raw_value: float | int | str,
    meta: TagMetadata,
) -> Optional[Decimal]:
    """Apply the single, deterministic ingestion round defined by tag metadata.

    Returns the rounded Decimal, or None for values that must be quarantined
    (NaN/Inf, unparseable, or outside the source's physical resolution).
    """
    try:
        # Convert via str() so we never inherit a float's representation drift.
        value = Decimal(str(raw_value))
    except InvalidOperation:
        logger.error("decimal_parse_failed", extra={"tag": meta.tag_id, "raw": raw_value})
        return None

    if value.is_nan() or value.is_infinite():
        logger.warning("non_finite_value", extra={"tag": meta.tag_id, "raw": raw_value})
        return None

    # Reject readings the hardware physically cannot produce -> engineering review.
    if meta.fs_min is not None and value < meta.fs_min:
        logger.warning("below_full_scale", extra={"tag": meta.tag_id, "raw": raw_value})
        return None
    if meta.fs_max is not None and value > meta.fs_max:
        logger.warning("above_full_scale", extra={"tag": meta.tag_id, "raw": raw_value})
        return None

    quantum = Decimal(10) ** -meta.max_decimal_places
    try:
        return value.quantize(quantum, rounding=_ROUNDING[meta.rounding_mode])
    except (InvalidOperation, KeyError) as exc:
        logger.error("quantize_failed", extra={"tag": meta.tag_id, "error": str(exc)})
        return None

For values destined for aggregation, pair the round with a fixed-point conversion so sums and threshold comparisons are exact:

def to_micro_units(value: Decimal, decimals: int) -> int:
    """Scale to integer micro-units for exact accumulation (sum/integral/compare)."""
    return int((value * (Decimal(10) ** decimals)).to_integral_value(rounding=ROUND_HALF_EVEN))

Telemetry payload design carries this forward across the network. JSON serialization coerces numerics to IEEE 754 doubles, so a value repeatedly parsed and re-serialized accumulates representation drift. To keep precision deterministic across distributed hops, payloads should transmit raw ADC counts alongside the scaled value, include an explicit precision field, and use binary serialization (CBOR or Protocol Buffers) for high-frequency vibration or power-quality streams where JSON float conversion degrades signal fidelity. Routing these through a well-structured MQTT hierarchy ensures the precision metadata travels with the data — but enforcement happens at publisher and subscriber, never at the broker.

Edge Cases and Failure Modes #Permalink to this section

Real factories break naive precision handling in predictable ways:

• Re-serialization drift across hops. A value that is correct at the edge becomes 12.450000000000001 after edge → gateway → broker → ingestion if any hop round-trips it through a JSON double. The fix — fixed-point on the wire — is covered in depth in handling floating-point drift in sensor readings.
• Threshold straddling. An unrounded 12.34999998 and a rounded 12.35 land on opposite sides of a hard-coded > 12.35 state-machine guard, producing phantom microstops. Always compare the rounded, fixed-point value against the threshold.
• sum/integral on high-cardinality tags. Floating accumulation across millions of points shifts aggregate baselines by 0.1–0.5%, enough to move an OEE Performance figure a full point. Accumulate in integer micro-units.
• Scan-cycle aliasing. When the publish rate is not an integer multiple of the PLC scan cycle, deadband-filtered samples land on inconsistent fractional values; align aggregation windows to scan-cycle boundaries before rounding.
• Mode mismatch (HALF_UP vs HALF_EVEN). Two services rounding the same stream with different modes will disagree on .5 cases and drift apart over a shift. The mode is part of the tag contract, not a local choice.
• Out-of-range readings injected as data. A wire fault reading below 4 mA on a 4–20 mA loop should be quarantined for engineering review, not rounded and counted. Route these to a dead-letter topic rather than into production metrics.

Verification and Testing #Permalink to this section

Precision contracts are easy to assert deterministically, so they belong in CI rather than in a runbook. The unit test below pins the rounding mode, NaN handling, and full-scale rejection:

from decimal import Decimal
import pytest

META = TagMetadata(
    tag_id="PRESSURE_LINE_01", engineering_unit="bar",
    max_decimal_places=2, rounding_mode="HALF_EVEN",
    fs_min=Decimal("0"), fs_max=Decimal("250"),
)


@pytest.mark.parametrize("raw,expected", [
    ("12.3456789", Decimal("12.35")),
    ("12.344", Decimal("12.34")),
    ("12.345", Decimal("12.34")),   # HALF_EVEN rounds to nearest even
    ("12.355", Decimal("12.36")),
])
def test_deterministic_rounding(raw, expected):
    assert round_telemetry_value(raw, META) == expected


@pytest.mark.parametrize("bad", ["NaN", "Infinity", "300.0", "-1.0"])
def test_quarantined_values_return_none(bad):
    assert round_telemetry_value(bad, META) is None


def test_fixed_point_sum_is_exact():
    vals = [Decimal("0.1"), Decimal("0.2"), Decimal("0.3")]
    total = sum(to_micro_units(v, 4) for v in vals)
    assert total == 6000  # exact; float sum would not be

After ingestion, confirm the database never stored more precision than the contract allows. On a TimescaleDB/PostgreSQL hypertable, this query surfaces any rows whose scale exceeds the declared limit — it should return zero:

SELECT tag_id, value
FROM telemetry
WHERE scale(value) > 2          -- max_decimal_places for this tag class
  AND time > now() - interval '1 hour';

For broker-level confidence, subscribe to the raw topic and assert the precision field is present on every payload before it is allowed past the ingestion gate (Pydantic or JSON Schema validation rejecting any payload missing precision or unit).

Performance and Scale Considerations #Permalink to this section

Decimal is correct but roughly an order of magnitude slower than native floats. At fleet scale the pattern is: round once with Decimal at the ingestion boundary, then carry the value as integer micro-units through every hot-path aggregation so the rest of the pipeline runs at integer speed. This keeps the slow, exact operation off the per-message critical path while preserving exactness where it matters.

Other scale levers:

• Write precision = source precision. Writing 12.3456789012 when the hardware resolves 12.35 wastes storage and defeats columnar compression. Aligning write granularity to the precision budget materially shrinks the hypertable and speeds downsampling. See time-series database synchronization for write-path and partitioning detail.
• Downsample with explicit clamping. Continuous queries and materialized views must apply mean/last with a fixed quantum; avoid sum/integral on float columns entirely.
• Backpressure preserves precision. When ingestion queues spike, drop redundant high-frequency samples rather than truncating decimal places — losing a sample is recoverable, silently degrading every value’s precision is not.

The payoff of treating precision as a first-class pipeline constraint is concrete: phantom variance disappears, TSDB storage drops, and the OEE roll-up — $\text{OEE} = \text{Availability} \times \text{Performance} \times \text{Quality}$ — reflects true equipment performance rather than arithmetic artifacts. That clean numerical foundation is what OEE formula validation and outlier detection downstream depend on.

Related #Permalink to this section

• Handling floating-point drift in sensor readings — fixed-point transmission and aggregation-boundary guards.
• PLC Tag Standardization — declaring scale, type, and precision in the tag contract.
• MQTT Topic Hierarchies — carrying precision metadata through the namespace.
• Time-Series Database Synchronization — write-time precision and retention.
• OEE Formula Validation — the downstream math that depends on consistent precision.
• Parent: Core Architecture & Data Mapping