Data pipelinesFinance
Spark pipeline tuning
A stock signal pipeline made 4× faster, and the bug it was hiding found
4×faster: 183 seconds down to 47 per run
See the code on GitHub (opens in a new tab)
Proof you can check:Code on GitHub
Data pipelines
Same pipeline, 183 s down to 47 s
Before183 s
After47 s
- Load 29→2
- Daily totals 4→1
- Indicators 47→5
- Correlation 51→15
- Signal report 19→14
- Delta write 31→10
- Summary 1→1
Lubo Bali · lubobali.com
The problem
A Spark pipeline gave correct-looking results but ran slowly, and on real data volumes it would take hours or crash. It was also quietly wrong.
What I built
Measured all 7 stages on 1.7 million minute bars, found 8 slow patterns (Python UDFs, needless shuffles, thousands of tiny files) and fixed each one, then compared the results before and after.
Tech & why
PySpark on Databricks with Delta Lake, where this kind of data lives. Each fix is a plain Spark change any team can keep.
- PySpark
- Databricks
- Delta Lake
- Unity Catalog
The result
4× faster overall, up to 15× on one stage, and 268 buy/sell signal changes (26%) that the old code silently dropped are now caught: it skipped weekends.
Similar projects
- Data pipelinesDelta Lake stock pipelineStock data laid out so a query reads only what it needs, and other teams find it by name1 of 8 stocks' partitions read when you ask about one stock
- Data pipelinesBitcoin streaming pipelineLive Bitcoin trades, cleaned as they arrive, and bad ones kept aside instead of lost0 rows silently dropped: every bad trade kept with a reason
- AI agentsMarketMind AIA market analyst agent: live prices, news mood and signals, answered in plain English2.6 s average answer, up to 4 tool calls
Have a problem like this? Get your free AI blueprint in about a minute.
