Data pipelinesData quality · Finance
Airflow data quality pipeline
New stock data waits in staging, passes 4 checks, then swaps into production in one move
4checks before any data reaches production
See the code on GitHub (opens in a new tab)
Proof you can check:Code on GitHub
Data pipelines
Write, audit, publish on Iceberg
- 1Wait for dataa sensor waits until the upstream table is ready
- 2Load to stagingnew data is kept apart from production
- 34 checksnot empty, no missing prices, prices above zero, volumes not negative
- 4Swap into productionone atomic partition swap, never half-written
- 5Clean upthe staging table is dropped
Lubo Bali · lubobali.com
The problem
Most pipelines write straight to production, so when bad data slips through, the people reading the reports see it before the team does.
What I built
An Airflow pipeline that waits for upstream data, loads it into a staging table, runs 4 checks (not empty, no missing prices, prices above zero, no negative volume) and only then swaps the day into production and cleans up.
Tech & why
Apache Iceberg on S3 with the AWS Glue catalog gives an atomic partition swap: production is never half-written, and any day can be re-run safely.
- Airflow
- Apache Iceberg
- AWS Glue
- Amazon S3
- Python
The result
If any check fails, the pipeline stops and production keeps yesterday's good data. Every run is repeatable and can be backfilled.
Similar projects
- Data pipelinesSQL Server → PostgreSQL migrationMoving a payments database to AWS, and proving the money still adds up83,892 report rows that silently stopped matching
- Data pipelinesBitcoin streaming pipelineLive Bitcoin trades, cleaned as they arrive, and bad ones kept aside instead of lost0 rows silently dropped: every bad trade kept with a reason
- Data pipelinesWrite, audit, publish (dbt + Airflow)Bad data stops at the door: tested first, published only if every test passes3 SQL files replace 300+ lines of Python
Have a problem like this? Get your free AI blueprint in about a minute.
