Forward-Fill Decay Windows for Drifting Analog Sensors

Zero-order hold is the right primitive for a discrete machine-state gap, but a bounded version of the same idea also has a place on the analog side of the gap-filling algorithms matrix: a slowly-drifting thermocouple, load cell, or pressure transducer that stops reporting for ninety seconds should not fall straight through to MISSING, yet the last reading it sent should not be trusted with full confidence for the rest of the shift either. The fix is a forward-fill that decays — the held value stays constant, but the confidence attached to it shrinks the longer the gap runs, until the pipeline stops trusting it altogether and falls back to something safer than a frozen number.

Confidence in a held sensor reading decaying exponentially as the gap since the last real sample widens Confidence starts at 1.0 at the last real sample and decays exponentially with a two-minute time constant as the gap widens. A trusted zone above 0.5 confidence lasts until 83 seconds; a decaying, flagged-stale zone runs from 83 to 276 seconds; beyond 276 seconds confidence drops below 0.1 and the value is expired in favor of a fallback baseline. gap since last real sample (seconds) → confidence conf 0.5 · staleness flag · 83s conf 0.1 · expire to baseline · 276s TRUSTED DECAYING · flagged stale EXPIRED → fallback baseline last real sample · conf = 1.0 confidence(Δt) = exp(−Δt / τ), τ = 120 s

The unbounded forward-fill hazard

ffill() with no limit is the fastest way to hide a dead sensor: a six-hour cellular outage on a remote tank farm becomes six hours of a perfectly plausible, perfectly frozen reading, indistinguishable in the historian from a genuinely stable process.

import pandas as pd

# Naive and hazardous: unlimited forward-fill hides a dead sensor indefinitely.
stale = raw_pressure.ffill()

Nothing downstream is wrong-looking about stale — it is a smooth, plausible line, which is exactly the problem. Consider a jacketed reactor’s temperature probe reporting 82.4°C, then going silent for forty minutes during a cellular backhaul outage while the batch continues heating. An unbounded forward-fill reports 82.4°C for the entire outage, a process engineer reviewing the historian sees a perfectly controlled batch, and the actual excursion — the reactor may have drifted 6–8°C above setpoint with no corrective action taken — is invisible until a quality deviation surfaces downstream with no telemetry to explain it. A stuck-at fault detector built for detecting frozen sensor faults may eventually flag the flatline as anomalous, but that is a defense-in-depth backstop running minutes to hours later, not a substitute for bounding the fill at the point where it is created.

Decay-window / max-hold with a staleness flag

The first defense is a hard ceiling on how many periods may be held, paired with a staleness_periods counter so every consumer downstream can see exactly how far a value is from its last genuine observation rather than inferring it.

import pandas as pd


def bounded_forward_fill(
    series: pd.Series, max_hold_periods: int
) -> pd.DataFrame:
    """Forward-fill an analog reading up to `max_hold_periods`, flagging every
    held sample with the number of periods since the last real observation.
    """
    was_real = series.notna()
    held = series.ffill(limit=max_hold_periods)

    # Staleness resets to zero at each real observation, then counts upward.
    group = was_real.cumsum()
    staleness = held.groupby(group).cumcount()
    staleness = staleness.where(held.notna())

    return pd.DataFrame({
        "value": held,
        "is_held": (~was_real) & held.notna(),
        "staleness_periods": staleness,
    })

max_hold_periods should come from the same error budget that sizes ring buffers at the edge: a slow-moving process temperature can tolerate a longer hold than a fast-changing hydraulic pressure before a frozen value becomes a materially wrong input to OEE. Note that groupby(group).cumcount() resets staleness_periods to zero the instant a real observation reappears, including a single isolated real sample sandwiched between two gaps — this is deliberate: staleness measures distance from the most recent genuine observation, not from the start of an outage, so a brief flicker of connectivity correctly buys the signal a fresh trust window rather than being ignored. Once max_hold_periods is exceeded, held reverts to NaN and the gap is handed to the MISSING path in the parent selection matrix rather than continuing to synthesize a value nobody can defend.

Confidence decay blending toward a baseline

A hard cutoff at max_hold_periods is a step function — fully trusted, then suddenly discarded — and a smoother alternative often serves OEE math better: decay the confidence in the held value continuously, and blend it toward a fallback baseline as that confidence approaches zero, rather than holding at full trust right up to the cliff edge.

import numpy as np
import pandas as pd


def decayed_confidence(staleness_seconds: pd.Series, tau_s: float = 120.0) -> pd.Series:
    """Exponential confidence decay: 1.0 at the last real sample, ->0 as the hold widens."""
    return np.exp(-staleness_seconds / tau_s)


def blend_toward_baseline(
    held_value: pd.Series,
    baseline: pd.Series,
    confidence: pd.Series,
) -> pd.Series:
    """Blend the held (frozen) reading toward a fallback baseline as confidence decays.

    At confidence 1.0 the output is the held value unchanged; as confidence -> 0
    the output converges on the baseline (a setpoint, a rolling seasonal mean,
    or a model prediction), rather than reporting a stale number as-is.
    """
    return confidence * held_value + (1.0 - confidence) * baseline

With tau_s = 120, confidence crosses 0.5 at 83 seconds of staleness and 0.1 at 276 seconds — the two thresholds marked in the diagram above, derived from −τln(0.5)-\tau \ln(0.5) and −τln(0.1)-\tau \ln(0.1) respectively. Choosing tau_s is a process-engineering decision, not a data-engineering one: it should track how fast the physical quantity can plausibly move on its own, which is exactly the same judgment that sizes the interior-gap thresholds in the parent gap-filling matrix. A reactor jacket temperature with meaningful thermal mass might reasonably use tau_s of several minutes, while a hydraulic line pressure that can spike or collapse in seconds needs tau_s an order of magnitude shorter, or the decay never catches up to how fast the real process is capable of diverging from the frozen reading.

The blend function is intentionally linear in confidence rather than a hard switch, because a hard switch produces a discontinuity in the reported value at exactly max_hold_periods — the reported reading jumps from the held value to the baseline in a single sample, which looks like a sensor fault of its own to any downstream anomaly detector. Blending continuously means the transition is smooth even though the underlying trust in the number is falling the entire time; the confidence column itself, not just the blended value, should be persisted so an analyst can distinguish “a real reading” from “a heavily blended fallback that happens to look plausible.” The baseline itself is commonly a rolling seasonal mean, a process setpoint, or the output of a lightweight forecast model — never another ffill(), which would just relocate the same hazard one layer downstream.

Gotchas & anti-patterns

  • Calling ffill() with no limit anywhere in the pipeline. A single unbounded call upstream — in a notebook, a one-off backfill script, or a well-intentioned “just fill the gaps” utility — defeats every decay-window discipline applied after it, and the hazard reappears exactly where nobody is looking for it.
  • Setting tau_s from a convenient round number instead of process dynamics. A slow tank-level signal and a fast vibration RMS should not share the same decay constant; tie tau_s to how quickly the real process can move between two genuine samples, ideally derived empirically from the observed rate-of-change distribution rather than picked once and never revisited.
  • Blending toward a stale baseline. If the fallback baseline is itself computed from a rolling window that includes the same outage, both sides of the blend degrade together and the output looks stable while being just as wrong; source the baseline from a signal or model independent of the failing sensor, such as a redundant transducer or a physics-based estimate.
  • Feeding held values into OEE math without the confidence weight. A Performance calculation that reads a decayed, low-confidence reading as full-precision input silently understates or overstates a rate; propagate the weight all the way through to the report, not just the blended value.
  • Resetting staleness on a duplicate rather than a genuinely new sample. At-least-once delivery can redeliver the last real reading; dedupe on sequence ID before computing staleness_periods, or the counter never advances even though the gap is real, and confidence stays pinned near 1.0 through an outage that should have decayed it.

Quick reference

Gap length since last real sample Confidence (τ = 120 s) Action
0–83 s 1.0 – 0.5 Trust the held value as-is
83–276 s 0.5 – 0.1 Hold, flag is_stale=True, still usable with reduced weight
276 s – max_hold_periods 0.1 – near 0 Blend toward the fallback baseline; downweight heavily in OEE
> max_hold_periods n/a Stop holding; mark MISSING, exclude from the metric basis

Related in this section