Data pipelinesData quality
Airflow + Soda quality checks
Data quality rules written in plain lines, run on a schedule, logged on every run
4data quality rules checked on every run
See the code on GitHub (opens in a new tab)
Proof you can check:Code on GitHub
Data pipelines
Rules anyone can read, checked on every run
row_count > 0missing_count(dag_id) = 0missing_count(execution_date) = 0duplicate_count(run_id) = 0
Lubo Bali · lubobali.com
The problem
Empty tables, missing values and duplicate rows usually show up in a report first. Nobody checks by hand every day.
What I built
An Airflow DAG that runs Soda checks against a PostgreSQL table on a schedule; the rules live in one readable file, and every scan is logged in Airflow.
Tech & why
Soda rules read like sentences (row count above zero, no duplicates), so anyone can add one without writing code; Docker runs it all locally.
- Airflow
- Soda
- PostgreSQL
- Docker
The result
Every run checks the table and leaves a log, and a new rule is one line in the rules file, with no extra code.
Similar projects
- Data pipelinesSQL Server → PostgreSQL migrationMoving a payments database to AWS, and proving the money still adds up83,892 report rows that silently stopped matching
- Data pipelinesWrite, audit, publish (dbt + Airflow)Bad data stops at the door: tested first, published only if every test passes3 SQL files replace 300+ lines of Python
- Data pipelinesAirflow data quality pipelineNew stock data waits in staging, passes 4 checks, then swaps into production in one move4 checks before any data reaches production
Have a problem like this? Get your free AI blueprint in about a minute.
