PROJECT AT A GLANCE
Project scope
Data engineering · Ingestion · Analytics. Individual responsibilities are not yet documented.
Keep the source before parsing it
ProcureScope is a data-engineering prototype for public IT procurement notices associated with Poland. Its documented path begins at the TED Search API, fetches and validates pages with Python, stores the original JSON response in MinIO, and writes normalized records and rejected rows to PostgreSQL. Airflow schedules ingestion; dbt builds analytical models for reporting queries.
The design choice I wanted to make visible is the order of operations: archive each received source page before parsing. If normalization fails, the original response remains available for inspection. The repository also documents deterministic keys and upserts so that a rerun can avoid creating duplicate records.
The screenshot is an architecture view
The visual above is generated from the repository’s architecture documentation. It is not a screenshot of a running dashboard and contains no live procurement counts. The project includes Metabase configuration and SQL questions, but the dashboard must be configured locally after the services and data are available.
I did not start the full Docker Compose stack during this review because Docker was not installed in the environment. The screenshot communicates the documented flow; it does not claim an end-to-end ingestion run was verified here.
Scope and next verification
This is a monitoring and analysis project, not a procurement marketplace, legal tool or guarantee of complete market coverage. The documented implementation does not yet expose a command to reprocess archived Bronze pages after a parser change.
The public repository contains the source, Compose setup, dbt models, Airflow DAG, SQL questions and runbook. A useful next check is to run the documented short ingestion range, verify raw-page hashes and quarantine behavior, then record the exact query window and row counts from that run.
Evidence record
The schematic and source excerpts are based on the public repository at the revision linked above.Individual ownership boundaries, testing history, failure reports and benchmark measurements have not been supplied.
System diagrams on this page describe documented capabilities. Profile and Source state explicitly which supporting materials are available.
Continue exploring
Browse the project archive or run an engineering experiment.
