Freelancer profile translated to English.

Description

Data Engineer, specializing in Big Data solutions and data pipeline development. With over 8 years of experience, I have designed and optimized complex data architectures, integrating tools such as Hadoop, Spark, Scala, and Apache Hive. My expertise covers the creation and industrialization of data ingestion, transformation, and loading processes, as well as workflow automation. I am also proficient in continuous deployment using tools like GitLab CI/CD and work in cloud environments and Big Data clusters. My approach focuses on performance optimization and adaptation to business needs through Agile methodologies.

Industry field of expertise

Languages

French
Native or bilingual
English
Conversational

Workplace preferences

Can work on-site

Paris (up to 50km)

Ville de Paris
Senior Big Data Consultant / Data Engineer
PUBLIC SECTOR
February 2024 - June 2025 (1 year and 4 months)
Paris, France
🔧 Maintenance and improvement of a data ingestion system with Apache NiFi, Python & Spark 🌐

As part of a project to centralize and process diverse urban data (geospatial, parking, bikes, green spaces, Seine water quality, etc.), I implemented a data ingestion system based on Apache NiFi, initially in standalone mode, then evolved to a NiFi cluster leveraging the distributed computing power of the Spark cluster.

🔍 Context:
- **Data Sources**: Integration of data from multiple sources (API, local files, S3, ELS, etc.).
- **Storage**: Centralization of data in MinIO, an S3-compatible storage solution, enabling efficient and secure management.
- **Processing**:
- Python & Spark: Use of Pandas for data processing and PySpark for large-scale analysis. Spark's distributed and parallel computing optimizes performance by efficiently processing large volumes of data.
- **Infrastructure**: Deployment on Linux machines with container management via Docker images to ensure optimal scalability and portability.
- **Exposure**: Processed data is accessible via Trino, facilitating its exploration and use by stakeholders.

📈 Maintenance:
- Implementation of data flow monitoring and incident management to react quickly to anomalies.
- Continuous performance optimization to ensure system reliability and scalability.
Apache Nifi Spark Scala PySpark DBT
URSSAF
Senior Big Data Consultant / Data Engineer
PUBLIC SECTOR
June 2022 - June 2023 (1 year and 1 month)
Paris, France
🌟 Key achievements:
🛠️ Development of data ingestion scripts for the data lake with HDFS and Hive, using Scala and SparkSQL in Parquet format.
🔄 Transformation and normalization of data with Scala and SparkSQL to ensure consistency and accuracy.
⚙️ Industrialization of processing by designing robust data pipelines with Scala.
🧹 Thorough data cleaning, significantly improving data quality through effective search and correction techniques.
✔️ Implementation of unit tests to ensure script reliability and proper functioning.
🚀 Optimization of distributed processing script performance on Spark, improving efficiency and execution speed.
🔄 Management of continuous deployment of scripts via GitLab and Jenkins, streamlining the development workflow.
🐞 Bug fixing and script optimization for better performance and maintainability.
📈 Automation of workflows with Oozie, increasing process efficiency.
Oozie Spark Scala PySpark Gitlab CI/CD
ORANGE
Senior Big Data Consultant / Data Engineer
TELECOMMUNICATIONS
June 2018 - April 2022 (3 years and 11 months)
Douala, Cameroon
🛠️ System Optimization: Collaboration in optimizing the Oracle data warehouse, significantly reducing errors and notably improving overall performance.
🌐 Data Lake Architecture: Participation in establishing a modern Data Lake architecture using HDFS, NiFi (cluster mode), MapReduce, Hive, Sqoop, and YARN technologies for efficient data processing and management.
📊 Monitoring Solutions: Implementation of Ambari for comprehensive cluster monitoring and Grafana for real-time tracking of key performance indicators and incident management.
⚠️ Error and Incident Management: Implementation of proactive error detection and incident resolution processes, ensuring minimal disruption to data operations.
🏆 Agile Methodologies: Working in an Agile environment applying the Scrum methodology to enhance team collaboration and accelerate project delivery.
📈 Data Visualization: Creation of impactful dashboards in Power BI to visualize key metrics and provide stakeholders with actionable insights.
👩‍🏫 Team Leadership: Management of junior staff and supervision of interns, fostering a collaborative atmosphere conducive to professional development.
🎓 Training Young Professionals: Design and delivery of training sessions for young professionals to equip them with skills in Big Data and related technologies.
🤝 Client Collaboration: Assisting clients in drafting functional specifications and requirements documents to ensure project alignment with business needs.
Talend Apache Nifi Nifi Hive HDFS