Ayoub F.

•Member of the Data Engineering and Data Platform team, at the heart of the Algolia organization, with the ingestion and provision of petabytes of data each month.

•Design and maintenance of large-scale ingestion pipelines using Kafka and Kinesis, integrating multiple sources, including external cloud providers and third-party tools such as Salesforce.

•Led the migration from Stitch to Meltano and created a reusable framework to automate ingestions via API and databases, orchestrated by Airflow and executed in deferred tasks on ECS.

•Development and optimization of data processing with Spark on EMR and AWS Glue.

•Implementation of a structuring framework for Analytics Engineers using dbt, to standardize transformations and analytical workflows.

•Conducted feasibility studies and PoCs for migrating Redshift to Databricks and Snowflake, in direct coordination with the publishers.

•Contributed to the design and implementation of a Lakehouse architecture based on Delta Lake and Databricks.

•Migration of the entire platform from Redshift to Databricks in production with 100% uptime

Joined as a Lead Data Engineer part of the PerfECO project to move from Talend and Redshift to a Spark/Scala based Data Pipeline.

Data validation at scale (batch and streaming) using FP Techniques and Frameworks (Cats/ZIO) => POSLog.

Train team members to get up to speed with Functional Programming in Scala and Distributed Programming.

Work closely with the “Data Factory” team, to improve the overall quality of Decathlon's Data Pipelines and Data Architectures accross teams.

Implementation of an agent-based distributed streaming system for Data Ingestion.

As part of the core team of the "Harmony" team, my responsibilities included setting up HPE's global data platform (complex event-driven architecture, based on Kubernetes and Knative), allowing teams to use a unified framework for all data processing jobs on the platform (Spark, Golang and Python).