Ibrahima Matar Gueye

Benchmarking of Data Platforms (Databricks, Snowflake)

• Definition and validation of evaluation criteria for the benchmark
• Selection of a large and representative dataset for business use cases (complex joins, filtering, aggregations)
• Design and implementation of reproducible test scenarios for each platform:
• ingestion, transformation (ELT), aggregation, analytical querying, load testing.
• Writing of a detailed comparative report including summary tables and graphs

Migration from Teradata to Azure Databricks - SIC Intermarché France

• Development of ingestion pipelines triggered by file drops from source applications into the DataHub (Blob Storage), using a feeding framework designed with Databricks and orchestrated via Azure Data Factory
• Writing of interface contracts defining technical specifications and mutual commitments for data exchange between sources and the DataHub
• Creation of business table DDLs on Databricks based on existing Teradata DDLs
• Migration of business tables from Teradata to Databricks for reporting needs (cash register receipts, revenue, customers, cardholders, Intermarché France points of sale)
• Development and population of calculated tables on Databricks, based on Teradata feeding scripts
• Orchestration of Databricks notebooks via Azure Data Factory

Project 1: Migration of Digicompta (Cloudera on-premise to Databricks)

• Resource creation: keyvaults, premium Databricks workspace, ADLS gen 2,
• Upgrade of Spark 2 code to Spark 3 to ensure compatibility with Databricks Runtime
• Migration of Airflow DAGs to Azure Data Factory to orchestrate our job pipelines, thus replacing the use of Airflow
• Implementation of a post-migration testing strategy:
• Comparison of results between source (Cloudera) and target (Databricks) environments on representative samples.
• Validation of volumes, business rules, and aggregates
• Creation of non-regression reports and gap analysis.

Project 2: C3S (development of indicators to evaluate the effectiveness of the call system by factors in the delivery of signed or taxed mail)

• Development of an ingestion pipeline on Databricks for daily processing of flat files deposited in a storage account
• Implementation of a medallion architecture (Bronze / Silver / Gold):
• Bronze: For ingesting raw files as-is into Delta Lake for archiving and traceability.
• Silver: For processing, cleaning, and normalizing data.
• Gold: Calculation of performance indicators (signed delivery rate, failure rate, average response time), aggregation by geographical area and period.
• Development of modular PySpark jobs for each layer
• Storage optimization (partitioning, compaction, Z-Ordering) to speed up downstream queries.