Data pipelinesPayments · Finance · Data quality
SQL Server → PostgreSQL migration
Moving a payments database to AWS, and proving the money still adds up
83,892report rows that silently stopped matching
Generated test data: no real customers or cards.
See the code on GitHub (opens in a new tab)
Proof you can check:Code on GitHubLimits said openly
Data pipelines
Every row arrived. 83,892 still came out wrong.
Lubo Bali · lubobali.com
The problem
Moving the data is the easy part. Knowing it arrived correct is the hard part: a lost cent or a date shifted by a time zone throws no error. It just gives you wrong numbers, forever.
What I built
A full move from SQL Server 2022 to AWS RDS PostgreSQL, every resource in Terraform: AWS DMS with live replication, 15 checks that fail on any difference, a timed backup and restore, and Parquet copies for analytics.
Tech & why
AWS DMS keeps the old system taking payments during the move (3.0 s behind). Terraform makes it repeatable. The checks run in CI, so a wrong number can't ship.
- SQL Server
- PostgreSQL
- AWS RDS
- AWS DMS
- Terraform
- Python
- Snowflake
The result
Every row arrived and all 15 checks passed, yet a status filter returned a third of the rows: SQL Server ignores upper and lower case, PostgreSQL doesn't. Found before any report went wrong.
Safe by design
Private, encrypted database with IAM sign-in; no SSH into the old server; passwords live in AWS Secrets Manager, never in a file.
Similar projects
- Data pipelinesAirflow data quality pipelineNew stock data waits in staging, passes 4 checks, then swaps into production in one move4 checks before any data reaches production
- Data pipelinesBitcoin streaming pipelineLive Bitcoin trades, cleaned as they arrive, and bad ones kept aside instead of lost0 rows silently dropped: every bad trade kept with a reason
- Data pipelinesWrite, audit, publish (dbt + Airflow)Bad data stops at the door: tested first, published only if every test passes3 SQL files replace 300+ lines of Python
Have a problem like this? Get your free AI blueprint in about a minute.
