Four reported events, hundreds of alert episodes

An anomaly detector can precede every event that happens to be labeled and still be a poor alerting system. The missing half of that sentence is the workload it creates for the person expected to review those alerts.

I explored that trade-off in APU Sentinel, a human-reviewed monitoring prototype for railway air-compressor telemetry. On one chronological test split, Isolation Forest preceded all four reported event periods at the 95th percentile of its training scores. The same setting produced 773 alert episodes. A simpler median-and-MAD baseline produced 37 episodes and preceded three of the four reports.

Those figures are not precision, recall, or a deployment recommendation. The dataset has only a small number of reported event periods, and an alert outside those periods is not automatically a false alarm. The useful question is narrower: what does this evaluation support, and what evidence is still missing?

Build a chronological test, not a random split

The public UCI MetroPT-3 dataset describes pressure, temperature, motor-current and air-intake-valve readings from an air-production unit on a metro train. UCI lists 1,516,948 instances and the dataset under a CC BY 4.0 license. The snapshot used here was downloaded on 14 October 2026; its CSV fingerprint begins db30ccb4 and ends 93e24.

The pipeline normalizes the timestamped rows, builds causal right-closed five-minute windows, and derives per-signal summaries such as mean, spread, last value, slope and digital state changes. The first 30 days form the training segment; later windows form the test segment. Median imputation and robust scaling are fit on training data before the Isolation Forest is fit. A train-only median/MAD deviation score serves as a simpler baseline. Each method gets its own threshold at the same percentile of its training-score distribution, so raw score values are never treated as comparable units.

There were 52,752 test windows in the saved evaluation snapshot. The minimum coverage rule retained 43,174 for alert and event evaluation, leaving about 18% out. That exclusion matters twice: it reduces the evaluation sample, and gaps can split or hide alert episodes. A result over only the retained windows should not be read as though the timeline were complete.

Define what “preceded an event” means

The report contains four source-reported event periods. For each one, the evaluator searches for an alert episode whose start falls between 24 hours before the reported event start and the event start itself. The episode must first pass a two-consecutive-window opening rule; a lower closing threshold adds hysteresis, and episodes within a 30-minute cooldown can be merged.

This is a clear, reproducible rule, but it is a generous lead window. An alert that starts almost a day before an event counts as “preceding” it even if no technician could act on that warning. The metric does not establish useful lead time, causality, or a maintenance decision. Nor does it estimate event recall against all failures: the labels cover only four reported periods.

At the 95th training percentile, Isolation Forest preceded four of four reports and emitted 773 episodes, or 4.22 per test day. The robust baseline preceded three of four and emitted 37, or 0.20 per day. Raising the Isolation Forest threshold to the 98th percentile still preceded all four in this split, while reducing the count to 408 episodes. At the 99th percentile it preceded three and emitted 241. These are useful points on a trade-off curve, not a validated operating threshold.

The column called “unlinked episodes” needs the same care. It counts episodes that were not the single earliest matching episode associated with one of the four reported periods. Because unreported anomalies may exist, unlinked does not mean false positive. Calling these episodes false alarms would claim knowledge the labels do not provide.

A data-quality flag is only as good as its unit

One implementation detail changes how I interpret the 43,174 retained windows. The current quality_ok flag compares the total number of rows in a five-minute window with an expected count derived from the median timestamp spacing across the full input frame. Since feature windows are built before the chronological split, a later change in cadence can influence the test-period coverage gate. The flag also does not enforce a separate non-null coverage minimum for every sensor. If one signal is absent while other rows remain, the window may still pass; missing feature values can then be filled with training medians before scoring.

Those behaviors are visible in the code, but the headline window count cannot tell us how often they affected the result. They are measurement-design gaps, not evidence that the reported scores are wrong. The next version should retain per-sensor valid counts, expose missingness as a feature or explicit quality state, and base expected cadence on a declared or training-period reference. Then I can report how many windows were excluded for each signal and test whether imputation changes alert rankings.

This also changes the right unit of analysis. A row count answers “did any numeric record arrive?” It does not answer “did the compressor have enough valid pressure, temperature and current observations for this decision?” Data-quality rules should describe the evidence the model actually needs.

Keep the monitor in a human review loop

APU Sentinel ranks windows by unusualness relative to its training period. A high score is not a calibrated probability of failure, a diagnosis, or remaining useful life. A list of the largest feature deviations can point an analyst toward signals to inspect; it does not explain the physical cause. Operating-mode shifts, sensor drift and incomplete inputs can all look unusual.

For that reason, the prototype is deliberately a review aid. It does not issue stop-train or maintenance commands, and its synthetic API demo is not the trained evaluation model. The evaluation snapshot is committed with the project documentation, while the generated machine-readable report, raw dataset and fitted model artifact are local. The source project is private, so this article describes the reviewed implementation and snapshot without presenting a public code link or a turnkey reproduction package.

The next evaluation should start with clearer evidence, not a more complex model: verify sensor-level coverage, review candidate episodes with a domain expert, widen the labeled history where possible, and test on another unit or a later prospective period. I would then report event-level lead times alongside alert episodes per day and the amount of usable telemetry. A threshold should be chosen against an agreed review capacity and measured labels—not because one percentile looks conventional.

The MetroPT-3 data is shared by its creators under CC BY 4.0. Please cite Davari, Veloso, Ribeiro and Gama, “MetroPT-3 Dataset,” UCI Machine Learning Repository, when reusing it.

For another example of how incomplete event histories affect evaluation, read Reliability Lab’s note on right-censored outcomes.

If you are building monitoring software where a person has to act on each alert, I’d be glad to talk through the evidence and trade-offs.

← All notes