ETL Pipelines for 20 Mn SAP Payments / Month:
- Owned 10+ ETL pipelines processing 20 Mn vendor payments / month from SAP source.
- 10+ dashboards, refreshed weekly by 500+ compliance managers, in industry-leading effort to rectify fraudulent financial behaviour.
Backend Migration to Databricks Lakehouse:
- SPOC for migration from SQL logic on Synapse Analytics stored procedures to PySpark logic on Databricks Lakehouse.
- Reduced the data refresh and QA timeline from 8 working days to 2 working days.
- Cut cloud costs by 80% month-on-month through optimised compute resource allocation.
- Enhanced data quality by testing data at the SAP source directly, improving the handling of fraudulent transactions.
Data Quality Management:
- 10+ DQ monitors alerted senior leadership to any data quality issues at SAP source and edge cases within ETL.
- SPOC for resolving to highlighted data errors with SAP team, as well as gaps at in the ETL process due to technical debt.