Sathiya Prabhu Kumar

Project: Lakehouse - Transformation of the Data Lake platform to a LakeHouse Architecture

• Migration of data and services from the Horton Works Data Platform (HDP) cluster to Trunk Data Platform (TDP).
• Redesign of data pipelines: transition from HDFS to S3 object storage.
• Optimization and tuning of Spark jobs (Batch and Streaming) as part of the HDP → TDP migration.
• Modernization of cold chain services: migration from Akka to containerized applications on OpenShift.
• Redesign of the supervision system: update of monitoring and alerting.
• Replacement of the Oozie scheduler with Apache Airflow: design, deployment, and automation of DAGs.
• Implementation of Airflow pipelines for data centralization and cross-referencing.
• Development and deployment of a CI/CD chain dedicated to Airflow DAG management.
• Implementation of Data-Aware Scheduling via Airflow's Inlets and Outlets features.
• Centralization of metadata and data lineage with OpenMetadata.
• Creation of alerting dashboards on Grafana for resource monitoring.
• Implementation of application monitoring dashboards with Kibana and Dynatrace.
• Level 3 support for the MCO (Maintenance in Operational Condition) team.
• Active participation in Agile ceremonies: SCRUM and SAFe.

Project: Naomi – Processing of sales and after-sales data for transport tickets

• Implementation of a data processing platform for sales and after-sales data of travel cards and tickets.
• Development of business data valuation pipelines using Spark on Databricks.
• Data ingestion into Azure Synapse (Data Warehouse).
• Storage, merging, and querying of data in DeltaLake format.
• Data archiving in Azure Data Lake.
• Development of Spark job scheduling workflows via Rundeck.
• Implementation of a CI/CD chain for continuous deployments and support through to production (MEP).
• Supervision and monitoring of applications via Datadog.

Project: Météo – Meteorological API adapted to SNCF needs

• Participation in the design of a project for accessing observation and forecast weather data provided by Météo France, with management of complex filters and parameters.
• Drafting of the Interface Contract for technical and functional documentation.
• Construction of a reference database for formatting meteorological data in Ariane format (railway standard).
• Design of the data model for the application.
• Definition and implementation of the data processing pipeline in the Azure environment.

Project: DATALAKE IT – Centralization of EDF data on a HortonWorks Big Data platform

• Implementation of a Data Lake on Big Data clusters based on the HortonWorks distribution.
• Design and development of real-time ingestion workflows via Akka APIs and Kafka topics.
• Real-time data processing with Spark Streaming.
• Development of Akka APIs for secure data exposure by authorization.
• Implementation of data valuation pipelines with Spark job scheduling via Oozie.
• Data archiving in structured files stored on HDFS.
• Deployment of a CI/CD chain managed by the service provider for production (MEP).
• Implementation of a supervision solution based on Fluentd and ELK: installation and configuration of Fluentd agents, code adaptation for sending logs to the collector.
• Creation of Grafana dashboards for log analysis and alert generation in case of incidents.
• Active participation in Agile ceremonies (SCRUM and SAFe).