The Client
A leading Indian e-commerce logistics platform processing over 2 million shipments daily, the company operates one of the country’s largest parcel delivery networks serving merchants, marketplaces, and direct-to-consumer brands. With analytics driving real-time operational decisions across orders, shipments, tracking, finance, and customer experience, the reliability and cost efficiency of its data infrastructure had a direct impact on business performance.
The Challenge
Unchecked infrastructure spend and fragmented data architectures directly erode profit margins and create significant operational overhead. Without a unified, cost-optimized framework, the business struggles to scale while maintaining financial and operational control.
- Escalating and Unpredictable Cloud Costs: Snowflake’s MAR-based variable pricing had become unpredictable at scale, compounded by EMR cluster and S3 Iceberg costs adding significant AWS infrastructure overhead.
- Fragmented 11-Tool Architecture: Snowflake, EMR, Glue, Lambda, Trino, Klipfolio, Metabase, SageMaker, Kafka, S3, and Cassandra operated as separate components with no unified governance or data lineage.
- SQL Dialect and Runtime Lock-In: 400+ Snowflake procedures, ~1,000 Trino queries in Metabase, and 280 Klipfolio queries all required dialect translation, ruling out a lift-and-shift approach.
- Multi-Source Ingestion Complexity: CDC pipelines from MySQL (432 tables), MongoDB (22 collections), Postgres (10 tables), and Cassandra (900 GB/day) operated through separate, non-standardized ingestion paths.
- Tenant-Scale Processing: The SRF pipeline served approximately 1,000 tenants across 110 topics each on EMR, with no native BigQuery equivalent, requiring a complete replatform of the consolidation logic.
The Impact
- 40%+ Cost reduction versus the prior AWS stack
- 1,937+ SQL objects migrated
- ~1,680 Queries translated across all workstreams
The Solution
Hoonartek executed the migration across 9 structured workstreams, consolidating the client’s entire analytics estate onto BigQuery as the primary analytical store, with Cloud Composer 3 managing all orchestration, Dataflow and Dataproc handling processing, and Debezium on GKE with Pub/Sub managing CDC ingestion. The architecture was restructured around a domain-oriented GCP project model, spanning Orders, Shipments, Tracking, Finance, Customer, and WMS domains, with shared services projects for networking, security, monitoring, and CI/CD.
Hoonartek’s proprietary MigFac accelerator automated compatibility checks across 26 rules, classified objects, and generated the rewrite inventory. A 140+ rule SQL transpiler ruleset handled dialect translation across Snowflake SQL and Trino to BigQuery, reused consistently across four workstreams. The Composer DAG library contributed 64 reusable pipeline patterns. 432 ELT pipelines, 40 reverse ETL jobs, and the full SRF batch operation now run under a single orchestration layer. All 180 Metabase and 30 Klipfolio dashboards were cut over with zero business disruption, and output contracts to downstream consumers were fully preserved.
Key Benefits
- Predictable Cost Model: Snowflake’s variable MAR pricing replaced by BigQuery slot-based consumption; EMR cluster costs eliminated entirely, delivering 40%+ cost reduction versus the prior stack.
- Unified Analytics Estate: 11 disparate AWS tools retired or replatformed onto a single GCP-native architecture with end-to-end governance and lineage.
- Reporting Continuity: 210 dashboards across Metabase and Klipfolio cut over with zero business disruption; ~1,680 SQL queries translated and validated.
- Scalable Orchestration: All pipelines consolidated under Cloud Composer 3, 432 ELT pipelines, 40 reverse ETL jobs, and the SRF batch workload running under a single orchestration layer.
- Reusable Migration Practice: MigFac toolkit, Composer DAG library, and SQL transpiler ruleset are now baseline accelerators for all of Hoonartek’s GCP migration engagements.