When telemetry breaks its contract: quarantine and replay in Aurora

I built Aurora to explore a practical data-engineering question: when a measurement is duplicated, incomplete or physically implausible, how do you keep it from quietly shaping downstream calculations?

Aurora answers that question with a synthetic telemetry pipeline. It generates energy-like records, injects controlled faults, validates each row, separates rejected records and tries a small set of known repairs. The source data and settlement values are simulated; this project does not connect to PSE, ENTSO-E or a live electricity market.

Start with a broken batch

The local API creates a batch of synthetic readings for a catalog of energy assets. A chaos injector can then duplicate events, move timestamps, mutate a value as if kilowatts had been sent where megawatts were expected, or introduce nulls and negative renewable output.

I use generated inputs because they make failure cases repeatable without operational grid data. The command-line runner sets a fixed generator seed, while the API seeds its generator from the current time. The checked-in browser demo is therefore a record of one execution, not a promise that every run will return the same counts.

Make rejection explainable

The validator checks that each row has an asset and timestamp, that the asset exists in the catalog, and that the batch does not repeat an asset-and-timestamp key. It also rejects missing power or voltage, negative output for renewable assets, and readings more than five percent above an asset’s nameplate capacity.

An invalid record carries its original payload, a short error code, the rule that failed and the observed value. That gives the quarantine path enough context to explain why the row was held back instead of reducing every problem to “bad data.”

The valid rows continue to the Silver output. The quarantined rows enter a recovery path with only a few explicit transformations: a suspected kilowatt-to-megawatt overflow can be scaled down, negative renewable output can be clamped to zero, and selected missing fields can receive known defaults. Aurora runs the repaired row through the validator again before it can join Silver. A row that still fails stays out.

That last step matters. A repair is a hypothesis about the source, not proof that the corrected value is true. In particular, a missing power measurement has no safe default in this prototype, so it cannot be “fixed” just to improve the batch score.

Use a score as a gate, not a certificate

Aurora combines freshness, completeness, validity and uniqueness into a project-defined score. The implementation weights those dimensions at 35%, 25%, 25% and 15%; the pipeline holds back its Gold summary when the score falls below 75.

The score is a simple control inside this experiment. It is not an industry standard, a calibrated confidence measure or evidence that a settlement is financially correct. A recorded local API run returned 206 records, with 11 quarantined and replayed and a score of 98.23. Those are example output values from the repository’s demo recording; the API generates a new synthetic batch, so they vary between runs.

The Gold calculation is simulated too. It uses a fixed market-price assumption and derives a nomination from the generated output. It demonstrates where a quality gate could stop downstream work; it does not recreate a real market settlement.

What I verified in the repository

The repository contains a fixed-seed integration-style test that generates a batch, injects selected faults, checks that the validator separates valid and quarantined rows, then revalidates records handled by the replay worker. The current GitHub Actions workflow compiles the Python tree but does not run pytest, so the CI record contains no passing result for that test. The test is evidence of an intended scenario in the code; the next verification step is to run it in CI and record the deterministic output.

The current recovery is also deliberately small. The replay worker receives quarantine records from the current in-process run. It is not a durable queue with operator review, scheduled retries or replay after a process restart. The project writes local Bronze and Silver JSONL files, and its database layer can fall back to SQLite; the presence of PostgreSQL, Redpanda, dbt and cloud configuration elsewhere in the repository does not mean this core run is integrated with those services.

The chaos injector can shift event timestamps, but the validator does not implement a late-arrival watermark or an ordering policy. I would need those rules, durable quarantine storage and explicit source contracts before describing this as a production streaming pipeline.

Why Aurora withholds the Gold summary

Aurora’s useful engineering result is the boundary between its Silver records and Gold summary. A row can enter Silver only after validation; a recognized repair must pass validation again; and the Gold summary is held back when the batch score is below 75. The example score of 98.23 is one synthetic run, not evidence that a market settlement would be correct.

The current failure is durability: quarantine and replay state live in the process, so a restart cannot resume an operator-reviewed recovery. My next implementation step is to persist rejected rows and replay attempts, then run the fixed-seed test in CI for duplicate replay, unresolved records and late arrivals. I would also test the score calculation separately from the policy that blocks Gold output.

Aurora’s implementation repository is private, so this article shows its architecture and boundaries without publishing source code. For two related examples, read how I used a transactional outbox between PostgreSQL and Kafka and how Retentia AI moves synthetic examples through a Unity model.

← All notes