Maxime Loyez

In an agile and cloud context (Databricks on AWS), participation in the development of robust data pipelines for a strategic project. Design of development notebooks, creation of Pyspark data flows orchestrated via Airflow, and automation of deployments via GitHub Actions.

• Implementation of tests with Great Expectations to ensure data quality.

• Creation of Makefiles to simplify local tests and checks.

• Implementation of CI/CD via GitHub Actions to ensure reliable deployments.

• Creation of data flows in pyspark within Airflow

• Addition of Github Action to automate deployments + code control

The goal was to provide data expertise and support teams in setting up data flows in Spark (Scala and Pyspark) as well as in automating and optimizing data flows (via Airflow).

- Adaptation of SQL queries (under Dremio) to Spark

- Deployment and automation of data flows via Airflow

- Development of data flows in Spark (pyspark/scala) / Python

- Use of APIs for data retrieval / implementation of probes for automated monitoring

- Dremio / Tableau / Dataiku / Jupyter

– Git / Jira

- Airflow / Spark (scala / pyspark) / Python / SQL

– Bash / Hadoop / Hive

- Agile

Intervention within the framework of a Dremio-Tableau POC

- Skill development for the team that would be in charge of the Dremio application.

- Load testing

- Implementation of Dashboards in Tableau based on different volumes.

- Implementation of VDS in Dremio

Support for data analysts on Pyspark to optimize their data processing.