Rafik B.

In an environment marked by strong competition and accelerating digital uses, the ambition was to implement a strategic and sustainable data platform on Microsoft Fabric.

This platform was intended to be a value creation lever, by offering a 360° view of policyholders, beneficiaries, and prospects, in order to strengthen customer knowledge, optimize commercial steering, and support a more targeted offer strategy.

Tasks:

• Close collaboration with the Data Architect for the definition of the platform's logical and technical architecture (Lakehouse, Workspaces, domain separation, data ingestion and exposure patterns, capacity sizing).

• Participation in data modeling workshops with the Data Manager and Data Architect (domain scoping, business object definition, etc.).

• Development of data orchestration and integration pipelines using Microsoft Fabric Pipelines and Spark Notebooks.

• Implementation of the Medallion architecture (Bronze / Silver / Gold) in a Data Mesh oriented foundation, with separation of Lakehouses and Workspaces by business domain.

• Definition and implementation of a data governance strategy:

• Setup of automated data quality controls and reporting via Soda Core,

• Data lineage and traceability management via Microsoft Purview, Access security via separation of business Workspaces and Row-Level Security (RLS) implementation.

• Data historization via implementation of the Data Vault 2.0 model at the Silver level.

• Industrialization and automation of deployments via Fabric Deployment Pipelines and CI/CD integration with Azure DevOps.

As a Databricks partner and member of the Delivery Partner Program, I participated in a mission to migrate the gold layer from Big Query and Amazon RDS to Databricks.

Tasks:

• Study of the technical and functional architecture.

• Participation in data layer modeling – Gold.

• Migration of data from Big Query and Amazon RDS to Databricks SQL with Lakeflow Connect.

• Design and development of data pipelines via Lakeflow Declarative Pipelines.

The Cerba Group initiated a hybrid data analysis platform project based on a data mesh architecture to improve data sharing, quality, and security within its entities. This project also aimed to ensure compliance with CNIL (GDPR) regulations for health data.

My intervention took place in two phases:

1. Development and production deployment of the first functional batch: implementation of an operational version of the platform.

2. Extension of capabilities and integration of new data sources: expanding analysis to financial and HR data to obtain a 360° view of the business.

Missions performed:

• Gathering technical requirements and participating in architecture workshops.

• Design and modeling of data layers (Bronze, Silver, Gold).

• Development of Big Data pipelines on Databricks (PySpark, SparkSQL).

• Pseudonymization of health data in PySpark.

• Implementation of a quality and compliance control framework (Pydeequ, Great Expectations).

• Development of right-to-be-forgotten mechanisms and Delta Sharing between Databricks.

• Performance optimization (ZOrder, partitioning).

• Workflow orchestration via Azure Data Factory.

• Implementation of data governance with Unity Catalog.

• Technical documentation and operating manuals.

Technical Environment:

• Azure (Data Factory, Databricks, DataLake, DevOps)

• Spark (PySpark, SQL)

• Databases (Oracle)

• Data Governance and Quality (Pydeequ, Great Expectations)

• Networks (Virtual Networks)