Wael Hilati

The project involves migrating an existing application called OSCAR using Spark as the data processing engine and Power BI as the visualization tool. Most of the data provides information about Enedis's electrical grid, such as component locations or outages and their reasons.

Main Tasks:

• Transformations of business specifications into Scala Spark jobs for data aggregation and enrichment (Mapping, transformation, and joins).

• Sizing the Kafka service (number of vCPUs, RAM…)

• Ensuring cluster security through IAM roles

• Monitoring the Kafka cluster using CloudWatch and associated metrics such as (Global Topic Count, Offline Partition Count, Messages In Per Sec, Bytes In/S….)

• Developing a Kafka producer to send part of the data to the cloud

• Developing a Kafka consumer to retrieve data and save it in S3 buckets

• Exposing external and internal Hive tables on the processed data.

• Code review and optimization of developed jobs.

• Definition of architecture and implementation of a data virtualization platform

• Data migration between the internal CDP cluster and the new platform shared by European central banks

Main Tasks:

• Market study on different virtualization technologies

• Participation in the design of the final architecture

• Implementation of the Denodo virtualization solution

• Migrating data between the cluster and S3 buckets

• Migrating Spark jobs to AWS Glue

• Developing Lambdas and Step Functions for pipeline orchestration

• Implementing queues for event exchange with AWS EventBridge

• Implementing infrastructure through a Terraform project

Technical Environment: AWS Lambda, Step Function, EventBridge, Cloud9, Terraform, Scala 2.14, Python 3.8, Spark 2.4, Hadoop, Hive, REST, MYSQL, Github, Gitlab, Docker, Ansible, JIRA, SonarQube, Artifactory.

1) Ingestion and processing of the group's massive data.

2) Design and development of Spark jobs for enriching and aggregating data representing the group's contracts.

3) Calculation of functional KPIs to help management make decisions regarding their client portfolios.