Achraf Ben Salem

Galileo is an application built on the AWS platform around the SCALA SPARK development framework and various AWS components (Lambda, Glue, DynamoDB, Redshift, Kinesis).

This platform is interconnected with Microsoft and Dataiku services.

✓ ACHIEVEMENTS

➢ AWS Cloud:

• Needs analysis and participation in identifying technical implementation scenarios on AWS.

• Design and implementation of high-performance ETL jobs on AWS Glue, using Scala/Spark, for processing large volumes of data from different sources (S3, JDBC, API).

• Formalization of technical specifications and evolution of the existing system.

• Implementation of Data Quality checks and use of Glue Catalog to facilitate data governance.

• Design and development of a flexible data ingestion module written in Scala/Spark, capable of handling multiple types of data sources (REST API, S3, JDBC).

• Analysis of CloudWatch logs for incident investigation and continuous improvement of processing.

➢ Windmill/Trading Auto:

• Facilitation of workshops with traders to gather business needs and understand the functional aspects related to configuring trading strategies.

• Implementation of the Python ETL: integration of multiple data sources using the boto3 framework for interaction with AWS S3 Cloud service.

• Automation of the implementation, monitoring, and execution of trading strategies via the Windmill platform.

Technical environment: AWS (Lambda, DynamoDB, S3, AWS Glue, Glue Catalogue, CloudWatch, CloudFormation, Athena), REST API, Scala Spark, Python, Azure DevOps, Windmil, Github, skills=[SkillTranslatableContent(id=Amazon Web Services, type=GLOBAL, name=Amazon Web Services), SkillTranslatableContent(id=Scala, type=GLOBAL, name=Scala), SkillTranslatableContent(id=Spark, type=GLOBAL, name=Spark), SkillTranslatableContent(id=AWS Glue, type=GLOBAL, name=AWS Glue), SkillTranslatableContent(id=API REST, type=GLOBAL, name=API REST)]

Development from scratch and implementation of the BING project to process data and populate various BI cubes to rationalize and energize reporting.

✓ ACHIEVEMENTS

• Design and development of data ingestion applications into the Datalake using Spark/Python.

• Data preparation: collection and transformation of ingested data (PySpark).

• Implementation of pipelines to load and transform data.

• Orchestration of pipelines with Azure Data Factory (ADF).

• Data validation with Databricks.

• Planning and execution of workflows with ADF.

• Participation in the drafting of specifications and writing of technical documentation.

• Contribution to the refactoring of PySpark application code by applying Spark best practices.

• Development of YAML code for continuous integration and deployment (CI/CD) in AZURE DEVOPS to optimize the integration and deployment of developments.

• Review and approval of Pull Requests to merge them into the master branch.

• Monitoring and analysis of production incidents via Azure Data Factory.

• Advancing teams and contributing to the validation of the schedule, skills=[SkillTranslatableContent(id=Microsoft Azure, type=GLOBAL, name=Microsoft Azure), SkillTranslatableContent(id=Azure DevOps, type=GLOBAL, name=Azure DevOps), SkillTranslatableContent(id=PySpark, type=GLOBAL, name=PySpark), SkillTranslatableContent(id=Databricks, type=GLOBAL, name=Databricks), SkillTranslatableContent(id=Docker, type=GLOBAL, name=Docker)]

Consolidate, orchestrate PMU data, and then make this data available for analytical uses such as BI, Data Science, Exploration, etc.

 ACHIEVEMENTS

 On-Premise:

• Design and creation of Spark/Scala applications from scratch and integration into the CI/CD chain to deploy projects in production via Jenkins.

• Implementation of pipelines to load and transform data.

• Deployment of applications in pre-production and production environments.

• Scheduling and supervision via ControlM.

• Massive data validation via Impala.

• Production support: Monitoring and analysis of production incidents.

• Maintain and evolve the functionalities of existing Big data projects.

 AWS Cloud:

• Redesign and migration of Hadoop On-Prem projects to AWS Cloud.

• Writing queries and validating massive data on AWS Athena to exploit tables.

• Continuous integration and deployment of programs via Gitlab CI/CD.

• Development and supervision of Lambda functions.

• Orchestration and monitoring of pipelines via Airflow.

• Data extraction from Amazon Simple Queue Service (Amazon SQS).

Technical environment: Cloudera, Spark, Scala, Hive, Hue, Impala, Jenkins, Foreman, Control-M, Dataiku, GITLAB, Jira, AWS (Airflow, Lambda, SQS, Athena, S3 Bucket).