Yangfan Xiang

- Microsoft Fabric

- airflow

- dbt

- postgresql

- python

with my team to build from scratch a new data warehouse

Total Energies Digital Factory as a Senior Data Engineer

Technical stack:

- azure

- databricks

- pyspark

- python

Tinyclues, specializing in predictive targeting for marketing campaigns, has over 300 clients, ranging from large corporations to SMEs. Its platform allows for campaign planning while targeting each consumer with personalized content. The tool also integrates a CDP (Customer Data Platform) component that allows campaign managers to better understand their consumers and have a macro/micro view. The technical challenges involve serving the 300 clients with a single codebase using the multi-tenant concept. This naturally leads to the following issues that we were able to resolve:

- GDPR Compliance: implementation of data masking, data retention, data deletion requests...

- Onboarding new clients: managing clients solely through YAML configuration files, auto-generation of configurations, tooling...

- Template and multi-tenant concept: no code redundancy, change management = configuration modification

- Efficiency, robustness, readability, and extensibility: designed with the Medallion architecture, all our code (ELT workflow, Airflow DAG, and SQL queries) is tested (unit tests + integration tests) and optimized (FinOps is a daily task)

- Mature CI/CD: every change = configuration modification = production deployment in minutes and automatically. Technologies: Data: Python, DBT, PostgreSQL Platform: Cloud Native on GCP, BigQuery, Airflow (Composer)... CI/CD: Git (Gitflow), GitHub, GitHub Actions, CircleCI, Terraform NoCode: Retool, Airbyte