All projects
PulseIQ / Analytics Platform
Analytics and business intelligence platform built around financial and operational data.
Problem
Business teams need current answers about portfolio and operational performance, but running analytical queries against production databases degrades the transactional systems those same teams depend on.
Solution
A change-data-capture pipeline that replicates production changes into a separate analytical store. Reporting reads from that store, so analysis stays close to real time while production load is unaffected.
My Contribution
- Designed and built the CDC-based data pipeline.
- Set up managed replication from production stores into the data platform.
- Modelled datasets for analytical query patterns rather than transactional ones.
- Built the query and reporting layer used by business teams.
- Isolated analytical load from production workloads.
Key Features
- Change data capture from production sources
- Near real-time replication into the data platform
- Analytics-oriented data modelling
- Ad-hoc querying over the data lake
- Operational and business dashboards
- Full isolation of analytical load from production
Architecture
Production DB—Transactional source of truth, untouched by analytics.
Production changes flowing into an isolated analytics layer.
Technologies
- AWS DMS
- CDC
- PostgreSQL
- Athena
- S3
- Python
Impact
- Near real-time visibility into portfolio and operational metrics
- Analytical workloads fully isolated from production systems
- Self-serve reporting reduced ad-hoc engineering requests
Want to talk through the details?
Happy to go deeper on any of the engineering decisions here.