Romain Duval

Data architecture design, including:

Database and storage system design.

Definition of data ingestion and processing pipelines.

Efficient integration of company data sources by developing:

Data flows.

ETL (Extract, Transform, Load) processes.

Data quality management:

Implementation of data cleaning and validation processes.

Definition of rules to ensure data accuracy and reliability.

Data security:

Implementation of security measures such as data encryption.

Access and authorization management.

Compliance with data protection regulations.

Optimization of data infrastructure performance, ensuring that:

Queries and analysis processes run efficiently and quickly.

Cross-functional collaboration with other teams:

Software development teams.

Data analysts.

Business managers, to understand needs and design appropriate solutions.

Implementation of the SAFe methodology

Evangelizing teams on Craftsmanship

PROJECT CONTEXT 1:

Within the BI department on the project to redesign and migrate the on-premise datamart of TV advertising data to the cloud. This datamart allows tracking indicators on advertising revenue evolution. Additionally, it involves implementing a data quality control process.

ACTIVITIES:

Design of Azure Data Factory pipelines for data ingestion into the different layers of the Data Lake.

- Implementation and deployment of Apache Spark data transformation jobs with Azure Databricks and ADLS Gen2.

- Implementation of data quality controls with Spark Scala.

- Code review and quality control with SonarQube.

- Writing technical and operational architecture documents.

- Design of a monitoring and supervision dashboard for components (Azure Databricks cluster, Data Factory pipeline, Azure SQL Database, Azure Blob Storage) with Datadog.

METHODOLOGY: Agile (SCRUM)

TECHNOLOGIES: Azure Data Lake Storage Gen 2, Apache Spark, Scala, Azure Databricks, Azure Data Factory, Azure DevOps, Azure SQL Database, Azure Key Vault, Microsoft PureView

PROJECT CONTEXT 2: Implementation of a BI data quality management process

ACTIVITIES:

- Modeling of an advertising data catalog.

- Proposal of a plan for defining key data by functional area.

- Business workshops for defining data quality rules.

- Definition of data quality KPIs

Data engineer within the Collateral and Risk entity of the Transverse Markets department on the Data Initiative project, which aims to establish a common cloud data platform for all collateral data consumers. This data platform ingests data from multiple sources, transforms it according to a repository, and then exposes it via APIs. Main tasks include:

- Proposal of an efficient platform architecture (in terms of cost and performance).

- Implementation of APIs for extracting and loading data from Oracle DB to ADLS Gen2 using Spring Batch and Apache Camel.

- Design of datalake datasets and cross-functional data lineage. Business workshops for building the data catalog.

- Supervision and monitoring of APIs and Azure HDInsight, Azure Kubernetes Services clusters with the Elasticsearch Logstash and Kibana suite.

- Implementation and deployment of Apache Spark data transformation jobs with Spark Scala, Azure HDInsight, and Azure Kubernetes Services and ADLS Gen2.

- Implementation of data quality controls with Spark SQL.

- Pipeline orchestration with Apache Airflow and continuous delivery with Jenkins, SonarQube, Gitlab.