Data pipelinesFinance
Delta Lake stock pipeline
Stock data laid out so a query reads only what it needs, and other teams find it by name
1 of 8stocks' partitions read when you ask about one stock
See the code on GitHub (opens in a new tab)
Proof you can check:Code on GitHub
Data pipelines
A query reads only the data it needs
A query for NVDA reads 5 of 40 partitions.
Lubo Bali · lubobali.com
The problem
Stock data piles up fast. Stored carelessly, every question scans everything, and other teams need to know file paths to use it.
What I built
A pipeline that pulls minute-level prices for 8 stocks, writes them to a Delta table split by stock and day, keeps the files tidy (OPTIMIZE, Z-ORDER, VACUUM) and registers the table in Unity Catalog.
Tech & why
Questions almost always filter by stock or date, so partitioning the data that way lets each query skip the rest; the catalog lets anyone query it by name.
- PySpark
- Delta Lake
- Unity Catalog
- Databricks
The result
About 40 partitions instead of thousands, a query for one stock reads only its own, and the table is shared by name. Rate limits of the price API are handled with pacing and retries.
Similar projects
- Data pipelinesSpark pipeline tuningA stock signal pipeline made 4× faster, and the bug it was hiding found4× faster: 183 seconds down to 47 per run
- Data pipelinesBitcoin streaming pipelineLive Bitcoin trades, cleaned as they arrive, and bad ones kept aside instead of lost0 rows silently dropped: every bad trade kept with a reason
- AI agentsMarketMind AIA market analyst agent: live prices, news mood and signals, answered in plain English2.6 s average answer, up to 4 tool calls
Have a problem like this? Get your free AI blueprint in about a minute.
