Schema Evolution for Versioned Sensor Payloads

A sensor payload schema is never really “done” — a firmware update adds a second accelerometer axis, a plant standardizes a new quality-flag enum, or a retrofit sensor needs a calibration-offset field that the original design never anticipated. This page is the concrete evolution playbook under Payload Serialization Formats: how to change a schema that is already deployed across thousands of edge nodes without a synchronized flag-day rollout, which on a factory floor is rarely even possible because some gateways cannot be remotely reflashed and remain on old firmware for years. Getting evolution wrong does not throw a clean error — it silently corrupts the OEE math computed on top, because a partially-decoded record with a missing field often defaults to zero rather than failing loudly.

The core discipline across every format here is the same: schema changes are either compatible (an old consumer can still read new data, or a new consumer can still read old data) or they are not, and the format’s tooling should make an incompatible change hard to ship by accident rather than something you find out about from a downtream dashboard.

Backward, forward, and full compatibility #Permalink to this section

The three compatibility terms are precise and worth defining exactly once, because they get used loosely and that looseness is where mistakes hide:

• Backward compatible: a new consumer (running the latest schema) can read data written by an old producer. This is what lets you deploy a new decoder before every edge device has updated firmware.
• Forward compatible: an old consumer (still running a previous schema version) can read data written by a new producer. This is what lets a firmware rollout ship ahead of a fleet-wide decoder redeploy — the common case in manufacturing, where firmware updates and analytics deployments are on independent schedules.
• Full compatible: both hold simultaneously. This is the target for any schema that a production sensor fleet depends on, because you rarely control the deployment order of producers and consumers precisely enough to rely on only one direction.

from dataclasses import dataclass
from enum import Enum


class Compatibility(str, Enum):
    BACKWARD = "BACKWARD"   # new reader, old data
    FORWARD = "FORWARD"     # old reader, new data
    FULL = "FULL"           # both directions hold


@dataclass(frozen=True, slots=True)
class SchemaChange:
    description: str
    compatibility: Compatibility | None  # None means breaking, no direction holds


# The additive-only rule set that keeps a change FULL-compatible:
SAFE_CHANGES = [
    SchemaChange("add optional field with a default value", Compatibility.FULL),
    SchemaChange("add a new enum symbol at the end", Compatibility.FULL),
    SchemaChange("widen an integer type (int32 -> int64)", Compatibility.FULL),
]
BREAKING_CHANGES = [
    SchemaChange("rename an existing field", None),
    SchemaChange("remove a field without reserving its tag/name", None),
    SchemaChange("change a field's type incompatibly (string -> int)", None),
    SchemaChange("reuse a previously retired field number", None),
]

Additive-only field rules per format #Permalink to this section

Protobuf and Avro enforce additivity through different mechanisms, and mixing up which rule applies to which format is the most common way a “safe” change ships broken.

Protobuf: reserve retired field numbers and names. Protobuf’s wire format identifies fields purely by number, not name, so removing a field is safe for the wire but dangerous for humans — someone will eventually add a new field and reuse the freed number, and an old consumer will misinterpret the new field’s bytes as the old field’s type. The reserved keyword makes that reuse a compile error instead of a runtime data-corruption bug.

# telemetry_v2.proto — v1 had a "raw_unit_code" field (tag 7) that is
# retired. Reserving both the tag and the name prevents accidental reuse.
PROTO_SCHEMA_V2 = '''
syntax = "proto3";
package plant_eu.telemetry;

message TelemetryRecord {
  string asset_id = 1;
  string metric_name = 2;
  int64 ts_unix_ms = 3;
  double value = 4;
  Quality quality = 5;
  uint32 seq = 6;

  reserved 7;                  // was raw_unit_code (int32); retired in v2
  reserved "raw_unit_code";    // belt-and-suspenders: block name reuse too

  double calibration_offset = 8;  // new in v2, additive, defaults to 0.0

  enum Quality {
    GOOD = 0;
    UNCERTAIN = 1;
    BAD = 2;
  }
}
'''

A v1 consumer reading a v2-produced message simply does not know field 8 exists; proto3 silently drops unrecognized fields during parsing, which is forward compatibility working as designed — no exception, no crash, the old consumer just never sees calibration_offset until it upgrades.

Avro: every new field needs a default, and resolution is explicit. Avro does not identify fields by number; it resolves a writer’s schema against a reader’s schema by field name at read time, and the resolution rules are precise: a field present in the writer schema but absent from the reader schema is ignored (forward compatibility), and a field present in the reader schema but absent from the writer schema is filled from its default (backward compatibility) — but only if a default is declared. Omitting the default on a new field breaks backward compatibility outright.

import fastavro

# v1 schema — the original writer schema for records already on disk/in-flight.
SCHEMA_V1 = {
    "type": "record", "name": "TelemetryRecord", "namespace": "plant_eu.telemetry",
    "fields": [
        {"name": "asset_id", "type": "string"},
        {"name": "metric_name", "type": "string"},
        {"name": "ts_unix_ms", "type": "long"},
        {"name": "value", "type": "double"},
        {"name": "quality", "type": {"type": "enum", "name": "Quality",
                                      "symbols": ["GOOD", "UNCERTAIN", "BAD"]}},
        {"name": "seq", "type": "int"},
    ],
}

# v2 schema — adds calibration_offset WITH a default. Without the
# "default" key this change would fail backward compatibility: a v2
# reader given v1 data (no calibration_offset present) would have
# nowhere to source the value from and resolution would raise.
SCHEMA_V2 = {
    "type": "record", "name": "TelemetryRecord", "namespace": "plant_eu.telemetry",
    "fields": [
        *SCHEMA_V1["fields"],
        {"name": "calibration_offset", "type": "double", "default": 0.0},
    ],
}


def read_v1_data_with_v2_reader(payload: bytes) -> dict:
    """Demonstrates Avro's explicit writer/reader schema resolution."""
    import io
    buf = io.BytesIO(payload)
    writer = fastavro.parse_schema(SCHEMA_V1)
    reader = fastavro.parse_schema(SCHEMA_V2)
    return fastavro.schemaless_reader(buf, writer, reader)
    # returns calibration_offset=0.0, sourced from the reader's default

A registry compatibility gate in CI #Permalink to this section

The rules above are only useful if something enforces them before a schema change merges, rather than trusting every engineer to remember the additive-only discipline under deadline pressure. The gate below runs in CI against every proposed schema change, checking it against the currently deployed schema before a pull request can merge.

from __future__ import annotations

import sys
from dataclasses import dataclass

import fastavro


@dataclass(frozen=True, slots=True)
class CompatibilityResult:
    ok: bool
    violations: list[str]


def check_avro_backward_compatible(old_schema: dict, new_schema: dict) -> CompatibilityResult:
    """A new-schema reader must be able to resolve data written by old_schema.

    This is the direction that matters most for sensor fleets: you deploy
    the new decoder before every edge device has the new firmware, so the
    new schema must still make sense of records the old firmware wrote.
    """
    violations: list[str] = []
    old_fields = {f["name"]: f for f in old_schema["fields"]}
    new_fields = {f["name"]: f for f in new_schema["fields"]}

    for name, field in new_fields.items():
        if name not in old_fields and "default" not in field:
            violations.append(f"new field {name!r} has no default: breaks backward compat")

    for name, field in old_fields.items():
        if name in new_fields and new_fields[name]["type"] != field["type"]:
            violations.append(f"field {name!r} changed type {field['type']!r} -> "
                               f"{new_fields[name]['type']!r}: incompatible")

    try:
        fastavro.parse_schema(new_schema)
    except Exception as exc:  # noqa: BLE001 - surface any parse failure as a gate failure
        violations.append(f"new_schema does not parse: {exc}")

    return CompatibilityResult(ok=not violations, violations=violations)


if __name__ == "__main__":
    result = check_avro_backward_compatible(SCHEMA_V1, SCHEMA_V2)
    if not result.ok:
        for v in result.violations:
            print(f"::error::{v}", file=sys.stderr)
        sys.exit(1)
    print("schema change is backward compatible")

Wire this as a required check on the pull request that touches the schema file, pointed at the schema currently registered in the shared registry (not just the previous commit in git), so a developer branching from a stale checkout cannot accidentally approve a change against an already-superseded schema.

# .github/workflows/schema-compatibility.yml
name: schema-compatibility
on:
  pull_request:
    paths:
      - "schemas/telemetry/**.avsc"
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install fastavro requests
      - name: Fetch currently registered schema and diff
        run: |
          python scripts/fetch_registered_schema.py \
            --subject plant_eu.telemetry.TelemetryRecord \
            --out /tmp/registered_v1.avsc
          python scripts/check_avro_compat.py \
            --old /tmp/registered_v1.avsc \
            --new schemas/telemetry/telemetry_record.avsc

Gotchas & anti-patterns #Permalink to this section

• Removing a field instead of deprecating it. Deleting a Protobuf field without reserved or an Avro field without a documented removal plan means the next addition can silently reuse the slot and old consumers misread the new data as the old type. Reserve, don’t delete.
• Renaming a field for readability. A rename is, to both Protobuf and Avro’s resolution rules, indistinguishable from removing the old field and adding an unrelated new one — it breaks both directions of compatibility simultaneously, exactly as shown for value → reading in the diagram above.
• Adding a required field. Avro fields without a default and Protobuf fields marked required (proto2 only; proto3 has no required keyword, which is itself a deliberate design choice) both break backward compatibility the moment old data is read by the new schema.
• Trusting the producer’s schema version tag without validating it. A firmware bug that ships schema_version: 2 while actually still encoding v1-shaped data defeats every compatibility guarantee upstream tooling assumed. Validate structurally (parse against the declared schema and check it succeeds), not just by trusting the version label.
• Skipping the CI gate for “trivial” changes. The changes that break compatibility are rarely obviously risky at review time — a rename that looks purely cosmetic is exactly the failure mode the automated gate exists to catch before a human has to.

Quick reference #Permalink to this section

Related #Permalink to this section

• Payload Serialization Formats — parent topic covering the wire-format contract this evolution discipline protects
• JSON vs Protobuf vs Avro for MQTT Sensor Payloads — the format trade-offs that determine which evolution rules apply
• Designing Sparkplug B Topic Namespaces for Factory Telemetry — a fixed protobuf profile with its own alias-based evolution pattern
• MQTT Topic Hierarchies — where a schema_rev marker in the payload complements topic-level versioning