Mohammed T.

Within the PCM (Professional Coffee Machine) team, I am responsible for designing and implementing data pipelines and Spark jobs to feed the data warehouse and build data visualization dashboards/reports for the various clients using these professional coffee machines.

👉Data Engineering Tech Lead

• Interface with the backend team to manage changes at the data source level and anticipate technical impacts

• Support for new incoming Data Engineers

• Writing technical stories

• Pair programming

• Pull request reviews

• Release delivery (branch

👉Data Pipelines

• Design and development of Spark ingestion jobs for telemetry, twins, and reference data

• Design and development of Spark jobs (full-process mode) for transformation according to business rules and exposure via SQL/Synapse tables for PBI Dashboards

• Design and development of Spark jobs for the management entity

• Design and Development of transformation Spark jobs (delta-process mode) (initialization pipeline + delta-process pipeline)

• Design and Development of data pipelines for aggregated tables: golden data preparation pipeline, aggregation calculation pipeline (Daily, Monthly, Weekly), aggregated table exposure pipeline

• Optimization of transformation jobs

👉Data Stack

• Integration of Azure Synapse with ARM on the data platform

• Upgrade of Spark versions (Spark project + Spark run-time)

• Sizing of the Spark pool according to the use case for the run-time of data pipelines

• Configuration of Spark log transfer to Log Analytics

• Implementation of monitoring for data/Spark pipelines

• Integration of Delta Lake at the Spark jobs level and implementation of the Vacuum pipeline

I acted as a data professional to evaluate the final project defenses (Master's degree) in Data and Artificial Intelligence.

Data Engineer within the DataLake team

👉Data Pipelines

👉👉Batch Processing:

• Design and development of Spark/Scala ingestion jobs

• Design and development of Spark/Scala jobs for GDPR (key generation, encryption, right to be forgotten)

• Development of Airflow DAGs for ingestion jobs (GDPR compliant)

• Development of Salt formulas for creating Hive Tables and Views

• Development of Salt formulas for creating Phoenix/Hbase Tables

• Development of Spark job for compacting small HDFS blocks

👉👉Streaming Processing:

• Development of Nifi workflows for collecting events (sports reference data) then buffering in Kafka topics and processing in Spark streaming as well as restitution in Hbase

• Development of a Spark streaming job for enriching sports alerts with reference data

👉Data Stack

Modeling and implementation of the DataLake batch layer on the DEV environment

• Modeling and implementation of the data science platform (Jupyter, Hue) on the DEV environment

• Modeling and implementation of the speed layer (nifi/kafka/elastic) of the DataLake on the DEV and Pre-prod environments

• Study and Migration of the Prod DataLake (batch layer) to a new VLAN

• Development/Implementation of Salt Formulas for stopping and starting all batch layer services

• Development/Implementation of Salt formulas for unit testing of DataLake batch layer services

• Troubleshooting and Correction of Anomalies and Incidents

👉Data Science Industrialization

• Development of an industrialized pyspark data science project model (modular project + unit testing + CI/CD)

• Demo for data scientists on data science industrialization.