Blazeclan’s Data Ingestion Framework Helps Globe Telecom, Inc. Migrate its Data Platform to Snowflake on AWS within 3 Months

About Globe Telecom, Inc.

Globe Telecom, Inc. is a leading Philippines-based full-service telecommunications company. It offers a suite of services and products, including mobile, broadband, internet, managed services, and data connectivity, to meet the technology and telecommunications needs of organizations. The company spans a wide presence across digital marketing solutions, financial technology, virtual healthcare, and venture capital funding for startups.

The Need for Data Platform Migration within Strict Timeline

Globe had its data platform on Teradata, which embodied 3 data marts with historical data of nearly 500 TB having 300-400 tables in total. The customer sought to decommission Teradata and replace it with a cloud-based data warehouse due to support-related issues that directly impacted its data management and eventually, the business. A key challenge in migrating Globe’s data platform was that it was a stringent time frame of 3 months.

Snowflake as a Solution

Keeping Globe’s requirement in mind, Blazeclan proposed using Snowflake as a solution while moving its data platform to AWS. Considering the size of the platform, the Blazeclan team suggested using our data ingestion framework. The reason behind using this framework was its high compatibility in meeting Globe’s requirements and facilitated fast-tracking migration by 60-70% compared to its existing Apache Spark for Amazon EMR operations. 

Initially, the solution was designed using AWS Glue, but considering the cost and budget of Globe, we chose to go with the more cost-effective Snowflake stored procedures. Here, Globe’s responsibility was to transition the data from on-premises to Amazon S3, from where we have moved it to Snowflake. Hence, the data migration would flow in the manner of Teradata to Amazon S3 to Snowflake. Post-migration, Globe wanted support to manage its new data platform on the AWS-Snowflake environment.

Solution Approach

The Blazeclan team implemented an ELT-based solution using the below frameworks on Snowflake for every given stage of the data flow:

• Snowflake loading Framework:  It is a customizable framework to load data from S3 in any given format. It can load multiple hidden tables at a time.
• DQ Framework: It helps validate the Data Quality (DQ) rules concerning null and length checks.
• SCD Framework: It’s the configuration-based dynamic framework to prepare the merge statements based on the conditions.

For data reconciliation, we maintained audit logs at every stage of data flow with record counts. Moreover, the team created operational & monitoring dashboards on Snowsight for visualization and connected Tableau dashboards to Snowflake as a downstream consuming application.

Solution Framework

Benefits Achieved

• Data reconciliation became effortless for Globe due to the implementation of the audit and logging framework.
• Direct loading into Snowflake enabled high availability and faster access to the data.
• Automated view creation enabled scanning raw tables and generating appropriate views, allowing only specific roles to view the data and determine how RBAC must be applied.

Tech Stack