Miroglio

The Challenge

• Clean and normalize the vast amount of unstructured and heterogeneous end-customer transactions data, tracked through fidelity cards, to make them ready for advanced Data Analytics flows.
• Implement an ever-improving ML model to effectively identify purchase habits and recurring buying patterns to cluster customers based on specific parameters (e.g., spending, buying frequency, last purchase date).
• Automate the entire ML process from data extraction and data lake creation to the implementation of an integrated flow for feature selection, model training, and inference.

The Solution

• Setup of an automated ETL Pipeline based on AWS Step Functions. AWS Glue crawlers and jobs are used to extract the dataset from an existing legacy database and transform it into a modern data lake based on Amazon S3. The aggregated and normalized Data lake on S3 is then explored using Amazon Athena for high-performing queries and Amazon Quicksight for data visualization.
• Creation of a complete ML workflow based on the Amazon Sagemaker suite: SageMaker Notebook instances for exploratory data analysis (EDA), feature selection, and ML model testing and SageMaker Training and Hyperparameter tuning for model training and tuning.
• Setup of an ML pipeline orchestrated by AWS Step Functions, which executes the ETL workflow to fetch new data from the original data store and train/test the ML model on the new dataset. Sagemeker Processing is then used to infer the clusters, while AWS Glue writes the results both in the data lake and in the original database for further integrations with existing legacy systems.

The Benefits

Performances

beSharp migrated the customer dataset from an EC2 backed database to Amazon S3. Leveraging a scalable and extensible storage service like Amazon S3, beSharp put in place a Data Lake as a single source of truth for model training. Furthermore, the same S3 Data lake can be used to feed an arbitrary number of workloads, meeting any possible future business need. Amazon S3 perfectly integrates with several data visualization services, and in this particular case, Amazon Quicksight was used for dashboarding and BI. Thanks to the use of Amazon Athena, the customer could also implement powerful, precise, and cost-optimized file-level queries into the Data Lake.

Automation

Automation and orchestration goals were achieved thanks to automatic ETL – MLOps pipelines orchestrated through AWS StepFunctions. This enables the continuous update of the data lake so that Data Analysts are constantly relying on a fresh dataset. Besides, beSharp succeeded in integrating each pipeline into the customer’s internal legacy workflow control tool.

Agility

Thanks to the combination of SageMaker Studio, Amazon CloudFormation, and AWS Cloud9, we provided the customer Dev team with a secure and agile process to test, tune and train the SageMaker Notebooks and finally bring them into production.

About beSharp and AWS

beSharp supported Miroglio during the implementation of a customized infrastructure optimized to support Data Analytics and Machine Learning workloads, designed according to the best practices of the Well-Architected Framework and in line with the DevOps and MLOps principles. By leveraging the AWS Cloud and the AWS services for ETL, Machine Learning, and pipeline orchestration, beSharp has enabled Miroglio’s Data Analysts team to integrate with development and operations allowing growth in the number of releases in production of increasingly performing models. Thanks to the new infrastructure, specifically designed to support the development, testing, and training of ML models, the customer was able to implement a highly targeted marketing strategy based on customer clustering performed with the most advanced Machine Learning techniques. Furthermore, the beSharp Cloud Experts have made Miroglio’s technical teams independent in managing the new architecture through an on-the-job training approach.