Freelancer profile translated to English.
Back to original languageAbout Chafiq
đ Welcome to my Malt profile! đ
I am a Senior Data Engineer / Tech Lead (8+ years), expert in Databricks, Spark, Kafka, Airflow, and CI/CD (GitLab/Jenkins). I help teams deliver reliable, observable, and cost-effective pipelines, from batch to streaming, on-prem or cloud (Azure/AWS). Areas: banking/insurance, risk, fraud, regulatory, KPIs.
đ§âđ» Data Engineering & DevOps
âą Design & build of end-to-end ETL/ELT pipelines (Delta/SQL/APIs)
âą Streaming & event-driven with Kafka (Connect/Streams/Schema Registry)
âą Performance & cost optimization (partitions, cache, AQE, Z-Order)
âą Data quality, TDD tests, observability (metrics, SLA/SLO)
âą Airflow orchestration, CI/CD, Docker packaging
âą Kubernetes deployments, secrets management
âą Evolutionary/corrective maintenance & debugging of critical jobs
đ» Software Engineering
âą Python/Scala/Java backends (FastAPI/REST/Spring Boot)
âą Database & model integration (Delta Lake, Hive, Elasticsearch, Postgres)
âą Data exposure: APIs, SQL, dashboards & KPIs
đŻ Key Achievements
âą đŠ CrĂ©dit Agricole Assurance: x30 performance increase on Spark (hours â minutes) + Spark 3.5 & Airflow migration, automations via GUI.
âą đŠ SociĂ©tĂ© GĂ©nĂ©rale: on-prem market fraud migration â Azure/Databricks, execution time Ă·30.
âą đŠ Natixis: optimized risk engine (VaR/CVaR), 95% TDD, jobs 8 min â 1 min.
𧰠Stack
Apache Spark 2/3, Databricks (Delta/Unity Catalog), Kafka, Airflow, Hadoop/Hive/HBase, Snowflake, DBT, Python, Scala/Java, SQL, GitLab/Jenkins/Sonar, Docker/Kubernetes, Azure & AWS, Cloudera/MapR.
đ My requirement: robust, scalable, tested, and observable solutions, with quick wins from the 1st week.
Need a Data Engineering/DevOps/CI expert?
Write to me with use case, volume, stack: I'll come back with a concrete action plan.
I respond quickly (often < 1 hour).
Agencies/Consulting firms:specific conditions, please contact me.
English
Fluent
German
Basic
Spanish
Basic
Arabic
Native or bilingual
French
Native or bilingual
Can work on-site
Paris (up to 50km), Paris (up to 100km)
Experience
- Crédit Agricole AssurancesTech Lead - Senior Data EngineerBANKING AND INSURANCEJanuary 2023 - Today (3 years and 5 months)Paris, FranceContext:Refactoring, migration, and evolution of several monthly and annual regulatory declarations (FICOVIE, IER, EAI CDC, EAI, FATCA) in a critical and regulated Big Data environment.Actions:
- Coaching and mentoring of a Data Engineers team on Spark/Hadoop.
- Implementation of development standards (Git, TDD, CI/CD, SonarQube).
- Management of the Data platform migration: Oozie â Airflow orchestration, Spark 2.4 â Spark 3.5 migration, Java 8 â Java 17, management of compatibility and performance constraints.
- Active participation in technical and functional specification workshops, as well as migration monitoring committees.
- Development of a business GUI to automate end-to-end declarative processes.
- Release management.
Results:- Reduction of Spark processing time from several hours to a few minutes (x30 performance).
- Successful migration to Spark 3.5 and Airflow, with securing of critical jobs and minimization of regression risks.
- Acceleration of team skill development (Spark 3.x, Airflow, CI/CD training).
- Improvement of software quality and pipeline reliability thanks to the implementation of CI/CD and TDD best practices.
- Automation of regulatory declarations via the GUI â reduction of manual tasks and increased reliability of business processes.
Technologies:Spark 2.4 & 3.5, Hadoop, Hive, HBase, Kafka, Python, Airflow, Java 8, Scala 2.12, MAPR, Spring Boot, Angular, PostgreSQL, SQL, Kubernetes, JFrog, Jenkins, SonarQube, GitLab, Github Copilot (GPT, Claude, Gemini), IntelliJ, Windows. - Société GénéraleSenior Data EngineerBANKING AND INSURANCEAugust 2021 - December 2022 (1 year and 4 months)Fontenay-sous-Bois, FranceContext:Development of a market control and fraud detection platform on massive volumes, in a cloud migration context.Actions:
- Management of the On-Premise â Azure Cloud ecosystem migration (HDInsight, Azure Storage).
- Spark 2.x â Spark 3.x migration, with performance optimization (AQE, DPP, adaptive joins).
- Coaching and mentoring of the Data Engineering team (Paris, London, Bangalore).
- Implementation of development best practices (Git, TDD, CI/CD, SonarQube)
- Close collaboration with business and support teams to secure production deployments.
- Release management.
Results:- Reduction of Spark processing time from several hours to a few minutes (x30 performance).
- Acceleration of fraud analysis thanks to optimized and scalable pipelines in Azure.
- Successful adoption of Spark 3.x and Azure Cloud â improved platform robustness and flexibility.
- Significant improvement in the stability and operational costs of risk calculations.
- Improvement of software quality and pipeline reliability thanks to the implementation of CI/CD and TDD best practices.
Technologies:Microsoft Azure (HDInsight, Azure Storage Explorer, Databricks), Java 8, Scala 2.11, Apache Spark 2 and 3, Kerberos, Spring Boot, REST API, HDP 2.6, Hive, Hadoop, Windows, IntelliJ. - BPCE SAData EngineerBANKING AND INSURANCEFebruary 2019 - June 2021 (2 years and 4 months)Paris, FranceSubject: Software for calculating risk indicators and P&L (Profit & Loss).
- Ensure maintenance and evolution of existing components.
- Implement calculation processes for HVAR, Sensi x Shocks, Stress Tests.
- Implementation of the TDD methodology in the team (from 0% to 95% code coverage by UT).
- Optimization of algorithms and Spark code for greater efficiency in time and memory usage (from 8 mins to 1 min in calculation processes).
- Analysis, design, and development of solutions and new features.
- Participation in the development of technical and functional specifications.
- Participation in deliverable costing meetings.
- Ensure production deployments.
- Coordination with support and infrastructure teams.
Technologies: Windows, IntelliJ, Scala 2.11, Apache Spark 2, Apache Kafka, Kerberos, Spring Boot, Java 8, REST API, HDP 2.6, HBase, Hive, Hadoop.
Recommendations
These freelancer profiles also match your criteria
Agatha Frydrych
Backend Java Software Engineer
4.7
(3)
2
Baptiste Duhen
Fullstack developer
4.6
(4)
5
Amed Hamou
Senior Lead Developer
4
(2)
7
Audrey Champion
Web developer
4.3
(3)
4
Education
- Master MIAGE - Big Data for FinanceUniversité Paris-Dauphine2017- Big Data & Data Engineering - Machine Learning & Data Analytics
- Engineering Degree - Software Engineering (Big Data Option)ENSEIRB-MATMECA2017- Développement Logiciel - Architecture des SystÚme d'information - Développement Web - Bonnes pratiques et méthode Agile - Big Data
Certifications
- Big Data Analysis with Scala and SparkĂcole polytechnique fĂ©dĂ©rale de Lausanne (EPFL)2017
- Apache Kafka Series - Core & InternalsUdemy - Stephane Maarek2019