Data pipelinesData quality
Write, audit, publish (dbt + Airflow)
Bad data stops at the door: tested first, published only if every test passes
3SQL files replace 300+ lines of Python
See the code on GitHub (opens in a new tab)
Proof you can check:Code on GitHub
Data pipelines
Bad data never reaches production
- Writenew data lands in an audit table first
- Auditdbt tests: unique, not empty, prices above zero
- Publishmerged into production only if every test passed
- Production table
- Pipeline stopped
Lubo Bali · lubobali.com
The problem
When bad data lands in production tables, every dashboard and report built on them is wrong before anyone notices.
What I built
A Write-Audit-Publish pipeline: new data lands in an audit table, dbt tests run there (unique, not empty, prices above zero), and only if all pass does it merge into production.
Tech & why
dbt for tested SQL instead of custom Python, Airflow (with Cosmos) to run it on a schedule and show exactly where it stopped, Snowflake as the warehouse.
- dbt
- Airflow
- Snowflake
- SQL
The result
One failed test stops the pipeline, so bad data never reaches production, and 300+ lines of Python became 3 SQL files.
Similar projects
- Data pipelinesSQL Server → PostgreSQL migrationMoving a payments database to AWS, and proving the money still adds up83,892 report rows that silently stopped matching
- Data pipelinesAirflow data quality pipelineNew stock data waits in staging, passes 4 checks, then swaps into production in one move4 checks before any data reaches production
- Data pipelinesAirflow + Soda quality checksData quality rules written in plain lines, run on a schedule, logged on every run4 data quality rules checked on every run
Have a problem like this? Get your free AI blueprint in about a minute.
