Description

I help European companies build, migrate, and optimize Databricks Lakehouse platforms on Azure — with full overlap of your working day (CET) despite being remote.

What I do best:

• Lakehouse builds & migrations — medallion architecture (Bronze/Silver/Gold), legacy ETL → Delta Lake, ADF metadata-driven ingestion frameworks handling 20–40 GB/day incremental loads

• Unity Catalog governance — catalog design, fine-grained access control, lineage and audit-readiness (increasingly relevant for EU AI Act compliance)

• Performance & cost optimization — Spark tuning, cluster right-sizing, job consolidation; cut pipeline processing time by 35% for a Fortune 500 insurance client

• GenAI on the Lakehouse — RAG pipelines, LangChain/LangGraph agents. I built a production MCP-based observability agent that monitors Databricks job runs and cluster health and surfaces remediation suggestions automatically

Background: 7 years in data engineering across enterprise clients in insurance, retail, and the public sector. Currently a Senior Data Engineer at a global analytics consultancy.

Certifications: Databricks Certified Data Engineer Professional · Microsoft Fabric Data Engineer Associate

Working with me: Remote from India with full availability during European afternoons and evenings (12:30–21:30 IST = 08:00–17:00 CET). Available 15–20 hrs/week — ideal for ongoing platform support, a well-scoped migration workstream, or an optimization audit. Fluent English, async-friendly, documented handovers.

Let's talk about your Databricks challenge — happy to start with a short scoping call.

Industry field of expertise

Languages

English
Native or bilingual

Workplace preferences

Remote only

Primarily works remotely

Fractal Analytics,
Senior Data Engineer
BANKING AND INSURANCE
September 2025 - Today (11 months)
Noida, Uttar Pradesh, India
Project: 360
• • Architected and deployed scalable ETL pipelines using Azure Data Factory (ADF) and Databricks, ingesting large-scale policy datasets into ADLS Gen2.
• • Leveraged Claude Code CLI to generate automated PySpark unit-test suites, cutting syntax refactoring time by 20%.
• • Automated job orchestration through Databricks Workflows, resulting in a 40% reduction in manual operational overhead.
• • Enhanced data reliability by 30% through robust PySpark and SQL-based data quality frameworks.
• • Optimized Spark performance via partitioning, caching, and join strategies, reducing overall processing time by 35%.
• • Executed storage optimization via Parquet compression and lifecycle policies, cutting cloud storage costs by 20%.

Project: Chubb-Ontology
• • Built a Generative AI-powered observability agent, integrated via a custom MCP (Model Context Protocol) server, to autonomously monitor Databricks job runs and cluster health.
• • Exposed Databricks Jobs and Clusters APIs as MCP tools, enabling the LLM agent to query job status, fetch failure logs, and trigger automated cluster restart and recovery on job failure.
• • Engineered an LLM-based root-cause analysis layer that parses failure logs and generates actionable remediation
suggestions.
• • Delivered real-time, context-rich failure alerts to developers with root cause and suggested fix, reducing mean time to resolution (MTTR) and on-call triage effort.
Apache Spark Databricks RAG Langchain AI Agent
Publicis Sapient (TLG India Pvt Ltd),
Data Engineer
DIGITAL AND IT
May 2023 - August 2025 (2 years and 3 months)
Gurugram, Haryana, India
• • Developed high-volume data pipelines in the finance domain using Databricks and ADF, processing 8GB+ of daily transactional data.
• • Optimized existing PySpark ETL transformations and analytical queries, improving pipeline execution efficiency by 25%.
• • Integrated automated monitoring and alerting to proactively flag data SLA breaches, maintaining 99.9% platform availability.
• • Ensured data privacy and compliance by implementing GDPR-compliant processing pipelines and governance controls.
• • Engineered 20+ automated reconciliation scripts, improving data validation accuracy by 35%.
Databricks Apache Spark SQL Data Quality GitHub
Dunnhumby,
Data Engineer
BANKING AND INSURANCE
April 2022 - May 2023 (1 year and 1 month)
Gurugram, Haryana, India
• • Implemented cloud data warehouse solutions for a UK pharmaceutical client using DBT and Google Cloud BigQuery.
• • Managed complex workflow orchestration with Cloud Composer for reliable pharmaceutical analytics delivery.
• • Automated deployment by designing and maintaining Jenkins CI/CD pipelines, and used Terraform (IaC) to provision BigQuery tables and schemas.
• • Integrated datasets from 10+ vendors (~50GB) into a centralized analytics platform while reducing operational costs by 20%.
BigQuery GitHub Terraform Jenkins DBT