About Deepak
I help businesses build reliable and scalable data pipelines using Azure Data Factory, Azure Databricks, PySpark, SQL, and Delta Lake.
I specialize in Azure Data Engineering, ETL/ELT pipeline development, Data Lakehouse architecture, Medallion Architecture (Bronze, Silver, Gold), incremental data loading, and data transformation using PySpark and SQL.
As a Databricks Certified Data Engineer Associate, I have hands-on experience with Apache Spark, Delta Lake, Auto Loader, Structured Streaming, Delta MERGE, and Spark performance optimization.
I can help with:
• Azure Data Factory (ADF) pipeline development
• Azure Databricks and PySpark development
• Apache Spark data processing and optimization
• SQL development and data transformations
• Delta Lake and Medallion Architecture
• Incremental data loading and MERGE/upsert logic
• ETL/ELT pipeline development
• Data Lake and Lakehouse solutions
• Spark job troubleshooting and performance optimization
If you need help building, troubleshooting, or optimizing an Azure data pipeline or Databricks solution, feel free to contact me.
English
Fluent
Can work on-site
Krakow (up to 50km)
Experience
- L&T Technology ServicesSenior Software Engineer – Data EngineerAugust 2023 - September 2025 (2 years and 1 month)Krakow, Poland• • Designed and implemented an Azure Lakehouse platform based on Medallion Architecture using Azure Data Factory, Azure Databricks, ADLS Gen2, and Delta Lake to support scalable analytics and reporting.• • Developed and maintained 20+ Azure Data Factory pipelines to ingest and orchestrate data from SQL databases, REST APIs, ERP systems, CRM platforms, and vendor supplied CSV/JSON files.• • Built scalable ELT pipelines processing approximately 300–400 GB of data daily across 40+ business tables, delivering Bronze, Silver, and curated Gold datasets.• • Reduced daily data processing by 85–90% by implementing watermark-based incremental loading and Delta MERGE instead of full refreshes.• • Implemented SCD Type 1 and Type 2 for dimensional models.• • Improved Spark job execution time by 35–60% through partitioning, caching, broadcast joins, Adaptive Query Execution (AQE), and Delta Lake optimization.• • Improved data reliability and recoverability by leveraging Delta Lake features including ACID transactions, schema enforcement, schema evolution, Time Travel, and Delta MERGE.• • Managed governance with Unity Catalog and deployments using Azure DevOps, Git and CI/CD.• • Monitored and troubleshot production pipelines using Azure Data Factory Monitor, Spark UI, Databricks Job Runs, and Azure Monitor to ensure reliable data processing.
- JSDSolutionsSenior Software EngineerMay 2022 - July 2023 (1 year and 2 months)Krakow, Poland• • Technical Lead for automotive diagnostics.• • Led customer discussions to gather, analyze, and clarify vehicle diagnostics requirements, and coordinated task creation and assignment across the team.
- AptivExpert Software EngineerMarch 2020 - April 2022 (2 years and 1 month)Krakow, PolandDiagnostics stack development for automotive software.Developed and configured Watchdog manager to monitor health of the system
Recommendations
Be the first to recommend Deepak
Help this freelancer shine by sharing your experience working together.
These freelancer profiles also match your criteria
Agatha Frydrych
Backend Java Software Engineer
4.7
(3)
2
Baptiste Duhen
Fullstack developer
4.6
(4)
5
Amed Hamou
Senior Lead Developer
4
(2)
7
Audrey Champion
Web developer
4.3
(3)
4
Education
- Bachelor of EngineeringK.L.E. Institute of TechnologyBachelor of Engineering
Certifications
- Databricks Certified Data Engineer AssociateDatabricks2026