Real-time streaming ETL on Google Cloud Dataflow:
- 1M daily ad events streamed real-time to BigQuery through Apache Beam pipeline on GCP Dataflow.
- Achieved 99.99% data ingestion rate; crucial for accurate billing to advertisers.
- Reduced monthly streaming costs from $12K to $1K, a 91% cost optimisation.
- Implemented transparent DevOps solution to update the ETL in-stream without data loss.
Django REST APIs for Targeted Ad Delivery:
Delivered targeted real-time ads to 500K+ daily active users in 100+ countries with sub-500 ms latency by-
- eliminating server cold starts,
- minimal microservice use,
- caching end-user data,
- frequency caps on ad requests,
- optimising various small processes (e.g. lightweight auth),
… while maintaining low cloud costs in the process.
Architected geo-targeting for ads, increasing CTR by 50% over competing ad networks in Web3 ads.
Dashboards on Metabase:
Created dashboards providing key insights to CEO for stakeholder meetings.
Lightweight BigQuery SQL queries to showcase ad event data (10+ ad event types) segmented by location, category, spend, advertiser, and publisher.
A notable strategic decision based on these dashboards –
impressions from low-spend users in Pakistan and Bangladesh identified; leading to the implementation of frequency caps on ad delivery API requests to mitigate cost bleeding.