Keep the last complete feed when the next refresh fails

A mirror should not make its readers depend on the source being healthy at the exact moment they ask for data. I wanted the API to serve the newest archive that had been fully accepted and stored, even if a later scheduled refresh failed halfway through.

I explored that contract in a private Transit Feed Reliability Platform prototype for one operator-configured Wrocław GTFS feed. Its most useful guarantee is deliberately narrow: a failed attempt can update the run history, but it cannot advance the pointer to the latest archive. The local evidence comes from a synthetic ZIP and an injected HTTP 503—not from a live city feed or a production deployment.

Make publication a separate step

The key distinction is between receiving new bytes and publishing them. A successful HTTP response is not enough to make those bytes the API’s latest version. The worker first streams the response into a uniquely named temporary file while calculating SHA-256 and enforcing a size limit. It then checks that the file is non-empty, readable as a ZIP, and contains text-file entries.

That is a structural acceptance check, not full GTFS validation. The worker does not prove that every route, trip, stop and service calendar is semantically correct. I keep that boundary explicit because the system’s purpose is to mirror an archive safely, not to claim that every source schedule is valid.

After inspection, the worker writes the archive to an S3-compatible object key derived from its hash. Only when that object write succeeds does it open a PostgreSQL transaction: upsert the version metadata, move feed_state.latest_version_id, insert the successful run, and commit. The API reads the pointer from PostgreSQL and streams the corresponding object; it does not fetch the source on a visitor’s request.

The useful invariant is simple: the latest pointer only moves as part of a committed success record, after the archive bytes are stored. Content-addressed keys also mean identical bytes resolve to the same hash-derived key instead of creating a new identity for the same archive.

Let an outage change the history, not the download

I exercised the boundary locally with a Compose stack and a feed stub. The stub first returned a tiny synthetic ZIP containing agency.txt, routes.txt and trips.txt. The worker stored it with SHA-256 f7305e98d4cf3b0adbc78693cc4fcf5bb45feb38a81ae304e51b4320c201887e.

I then changed the stub to return HTTP 503 with Retry-After: 1 and restarted the worker. After its bounded attempts, the API’s run history showed a failed attempt with source_http_error. The feed status reported the last attempt as failed, while latestSha256 still pointed to the earlier archive. I also downloaded /latest again and checked that its bytes had the same SHA-256 as before the failure. Readiness remained ready through the local Caddy reverse proxy.

The failed run took 2.591 seconds, including retries. That number describes this one local scenario; it is not a recovery-time objective or a measure of a production incident. The same drill recorded that the public Caddy edge returned 403 for /metrics, which is intentionally scraped only on the internal network.

The distinction is useful to consumers: status can honestly say both “the most recent attempt failed” and “the previous accepted archive remains available.” Hiding the failure would make the mirror look healthier than its source path. Replacing the latest pointer on download would make consumers inherit the failure. Keeping attempt history and published state separate avoids both mistakes.

The boundary still has failure modes

PostgreSQL and object storage do not share a transaction. If the object upload succeeds but the database transaction later fails, the new object may be left unreferenced. The old pointer remains safe, but storage reconciliation is still needed to find and remove orphaned objects. The current flow favors serving a known stored version over trying to roll back a cross-service write that cannot be made atomic.

The worker also uses a PostgreSQL advisory lock keyed by feed ID so two copies do not ingest the same feed concurrently. That protects this single-feed path from overlapping scheduled runs; it does not turn the one-host Compose deployment into a highly available service. The project documents one VPS as a single point of failure and makes no public uptime promise.

What the drill proves—and what it does not

The repository’s local evidence record says the valid synthetic archive was stored, the later HTTP 503 was recorded, readiness remained available, and the old archive could still be downloaded with identical bytes. The recorded unit suite passed 3/3 archive-inspection tests, and both development and production Compose configurations parsed successfully. Those unit tests cover ZIP acceptance and rejection; the last-known-good behavior was checked in the manual local Compose drill, not by a full automated integration suite.

This is evidence for one failure path, not a live transit-feed evaluation. I did not measure recurring source availability, public uptime, a backup restore, or recovery on a real server. The project has a restore runbook, but a written runbook is not a completed restore drill. The code repository is private, so this article describes the design and the recorded synthetic test without presenting the source as public evidence.

For a mirror, “nothing new was published” is often a safer outcome than “the newest bytes must be served.” I designed the pointer to make that decision explicit—and the failure drill to show that an error can stay visible without taking the previous download away.

← All notes