Data pipelinesFinance · Data quality
Bitcoin streaming pipeline
Live Bitcoin trades, cleaned as they arrive, and bad ones kept aside instead of lost
0rows silently dropped: every bad trade kept with a reason
A small test run: 18 trades.
See the code on GitHub (opens in a new tab)
Proof you can check:Code on GitHubLimits said openly
Data pipelines
Bad trades kept aside, never lost
Lubo Bali · lubobali.com
The problem
Streaming pipelines usually drop bad records without a trace. Later nobody can say what was lost, or why the numbers don't match.
What I built
Live BTC-USD trades from a WebSocket flow through Bronze, then split: clean trades to Silver, bad ones to a quarantine table with the reason, and Silver feeds 1-minute price candles in Gold.
Tech & why
Delta Live Tables checks every trade (price, size, time, type); clean plus quarantined always equals everything that came in.
- Databricks
- Delta Live Tables
- PySpark
- Structured Streaming
The result
In the test run, 18 trades came in: 15 clean, 3 held in quarantine with their reason, none lost. Gold builds 1-minute candles from the clean ones.
Similar projects
- Data pipelinesSQL Server → PostgreSQL migrationMoving a payments database to AWS, and proving the money still adds up83,892 report rows that silently stopped matching
- Data pipelinesSpark pipeline tuningA stock signal pipeline made 4× faster, and the bug it was hiding found4× faster: 183 seconds down to 47 per run
- Data pipelinesAirflow data quality pipelineNew stock data waits in staging, passes 4 checks, then swaps into production in one move4 checks before any data reaches production
Have a problem like this? Get your free AI blueprint in about a minute.
