Ring Buffer Sizing and Disk-Spill Thresholds at the Edge

A buffer with no size limit is not more durable than one with a limit — it is just a slower way to run out of storage. This page works out the sizing math that Edge Buffering and Store-and-Forward requires but does not derive: how large an in-memory ring buffer must be to absorb the outages that actually occur on a plant floor, where the high and low watermarks that trigger a spill to disk should sit, and what a gateway should shed once even the disk-backed spool approaches full. Undersize the buffer and a routine five-minute Wi-Fi hiccup drops data; oversize it and a gateway with 4 GB of eMMC spends its budget on buffer headroom it will use once a year, starving the OS and the application of flash it needs for everything else.

Sizing the in-memory ring from message rate x worst-case outage #Permalink to this section

The in-memory ring buffer’s job is narrow: absorb short interruptions — a switch reboot, a brief Wi-Fi association drop — without touching disk at all, because memory writes are orders of magnitude cheaper than fsync’d disk writes and most outages are seconds, not hours. Its capacity is a direct function of two measured quantities: the asset’s steady-state message rate and the outage duration you want to absorb without spilling.

$\text{ring\_capacity} = \text{msg\_rate}_{\text{peak}} \times t_{\text{spill-threshold}} \times \text{safety\_factor}$

Use the peak, not average, message rate — a line that normally emits 20 msg/s but bursts to 80 msg/s during a changeover will overflow a ring sized on the average within seconds of a burst coinciding with an outage. A safety factor of 1.5–2.0x covers measurement error and the fact that message size (and therefore per-message memory cost) is rarely perfectly uniform.

from dataclasses import dataclass


@dataclass(frozen=True)
class RingSizingInputs:
    peak_msg_rate_per_s: float      # measured, not nameplate
    avg_bytes_per_msg: int
    spill_threshold_s: float        # how long to stay in-memory before spilling
    safety_factor: float = 1.75


def size_ring_buffer(inputs: RingSizingInputs) -> dict[str, float]:
    """Compute ring buffer capacity in messages and bytes."""
    capacity_msgs = (
        inputs.peak_msg_rate_per_s * inputs.spill_threshold_s * inputs.safety_factor
    )
    capacity_bytes = capacity_msgs * inputs.avg_bytes_per_msg
    return {
        "capacity_messages": round(capacity_msgs),
        "capacity_bytes": round(capacity_bytes),
        "capacity_mb": round(capacity_bytes / (1024 * 1024), 2),
    }


# Example: a CNC cell reporting 12 tags at 10 Hz, ~180 bytes/msg,
# tolerating 30s in memory before spilling to disk.
sizing = size_ring_buffer(RingSizingInputs(
    peak_msg_rate_per_s=12 * 10,
    avg_bytes_per_msg=180,
    spill_threshold_s=30.0,
))
# -> {'capacity_messages': 6300, 'capacity_bytes': 1134000, 'capacity_mb': 1.08}

A 30-second in-memory threshold before spilling is a reasonable default for gateways where flash write cycles are a real constraint (industrial SD or eMMC with finite program/erase cycles): it filters out the transient blips that account for the large majority of real disconnects without touching disk, while still spilling well before an actual outage grows long enough to matter for OEE.

The inputs to this formula are not one-time constants — they drift as a line changes. A packaging cell that adds three new vibration tags after a predictive-maintenance retrofit silently raises peak_msg_rate_per_s well past the value the ring was sized against, and the first symptom is not an error message but a slightly higher rate of disk spills during ordinary short blips that used to stay in memory. Re-measure the peak rate whenever tags are added or removed from a gateway’s scan list, and treat size_ring_buffer as a function you re-run as part of the change, not a number you set once at commissioning and forget.

High/low watermark spill-to-disk #Permalink to this section

A single threshold that flips a boolean “spilling” flag tends to flap under a bursty, borderline load — the buffer crosses the line, spills a little, drains back under it, stops spilling, and repeats every few seconds, which thrashes the disk write path worse than either staying in memory or spilling continuously would. Two watermarks with hysteresis between them fix that: cross the high watermark to start spilling, but only stop once fill drops below a distinctly lower low watermark.

from enum import Enum, auto


class SpillState(Enum):
    MEMORY_ONLY = auto()
    SPILLING = auto()


class WatermarkController:
    """Hysteresis-based spill controller: prevents flapping near capacity."""

    def __init__(self, capacity_msgs: int, high_pct: float = 0.80, low_pct: float = 0.40):
        self.capacity = capacity_msgs
        self.high = capacity_msgs * high_pct
        self.low = capacity_msgs * low_pct
        self.state = SpillState.MEMORY_ONLY

    def update(self, current_depth: int) -> SpillState:
        if self.state is SpillState.MEMORY_ONLY and current_depth >= self.high:
            self.state = SpillState.SPILLING
        elif self.state is SpillState.SPILLING and current_depth <= self.low:
            self.state = SpillState.MEMORY_ONLY
        return self.state

Setting the high watermark at 80% and the low watermark at 40% of ring capacity gives a wide hysteresis band; the exact numbers matter less than the gap between them being large enough that a single burst cannot cross back and forth in one polling interval. Emit a state-change event on every transition — spill_state_transitions_total — so an operator dashboard can distinguish “spilled once during a real outage” from “flapping every ten seconds,” the latter being a sizing bug, not a network problem.

Shedding and back-pressure when disk nears full #Permalink to this section

Disk capacity is finite and, unlike the in-memory ring, cannot simply grow to absorb an unusually long outage. Once the disk-backed spool itself approaches its configured ceiling, the gateway must make an explicit, auditable choice about what to drop — silent overwrite of the oldest rows is the default SQLite ring-table behavior if you are not careful, and it is rarely the right choice for production telemetry, where the newest data is usually the most operationally relevant.

import logging

logger = logging.getLogger("shed_policy")


class ShedPolicy:
    """Decide what to drop once the disk spool nears its hard ceiling."""

    def __init__(self, max_rows: int, shed_threshold_pct: float = 0.95):
        self.max_rows = max_rows
        self.shed_at = int(max_rows * shed_threshold_pct)

    def should_shed(self, current_rows: int) -> bool:
        return current_rows >= self.shed_at

    def choose_shed_batch(self, conn, current_rows: int, target_free: int = 500) -> list[int]:
        """Return the lowest-priority rows to delete: oldest low-priority metrics first.

        Downsampled/derived metrics (e.g. 1-second rollups already computed
        elsewhere) are shed before raw high-value counters like part counts.
        """
        cur = conn.execute(
            "SELECT seq FROM spool "
            "WHERE priority = 'low' "
            "ORDER BY seq ASC LIMIT ?",
            (target_free,),
        )
        victims = [row[0] for row in cur.fetchall()]
        if len(victims) < target_free:
            # No low-priority rows left; fall back to oldest overall,
            # but log loudly — this means a real, unplanned loss event.
            remaining = target_free - len(victims)
            cur = conn.execute(
                "SELECT seq FROM spool ORDER BY seq ASC LIMIT ?", (remaining,)
            )
            fallback = [row[0] for row in cur.fetchall()]
            logger.error(
                "shedding %d high-priority rows; disk spool exhausted, real data loss",
                len(fallback),
            )
            victims.extend(fallback)
        return victims

Tagging records with a coarse priority at enqueue time — production counters and state transitions as high, high-frequency vibration or temperature samples that are already summarized elsewhere as low — turns an emergency shed into a graceful degradation instead of an undifferentiated data massacre. When even low-priority rows are exhausted and high-priority data must be shed, that event is a release-blocking incident, not a routine log line, and it should page someone rather than scroll past in a log file: it is the one path in the whole store-and-forward design where the zero-data-loss guarantee is knowingly broken.

Gotchas & anti-patterns #Permalink to this section

• Sizing the ring on average rate instead of peak. A ring sized for average throughput overflows during every burst that coincides with a network blip — size on measured peak with a safety factor, not on a nameplate spec sheet number.
• A single threshold instead of watermarks. One boolean spill flag flaps under borderline load, thrashing the disk write path. Use high/low watermarks with real separation between them.
• Silent oldest-row overwrite as the default shed policy. SQLite has no built-in ring-table eviction; without an explicit ShedPolicy, an unbounded INSERT simply fills the disk until every write fails, which is a worse outage than the one you were buffering against.
• Ignoring flash wear when tuning the in-memory threshold. A spill_threshold_s set too low (spilling every burst) accelerates wear on gateways with a multi-year unattended deployment lifetime; measure real outage duration distribution before tuning it down.
• No alert on the shed-high-priority fallback path. If low-priority shedding silently escalates to high-priority shedding, the zero-data-loss guarantee has quietly failed. That branch must be loud.

Quick reference #Permalink to this section

Related #Permalink to this section

• Edge Buffering and Store-and-Forward — the durability contract this sizing math supports
• Build a Zero-Data-Loss Edge Buffer with SQLite and Replay — the spool and replay implementation these thresholds bound
• Backpressure and Bounded Queues in Asyncio MQTT Consumers — the same watermark discipline applied to live in-process queues