Hamid Zatri

Context:

As a Cloud Data Engineer, my mission is to successfully migrate the Data Warehouse from Teradata to the BigQuery solution, which is a data warehouse platform offered by Google Cloud Platform.

My responsibilities include designing and implementing the data architecture in BigQuery, ensuring that schemas, tables, and data pipelines are properly configured and executed. I am also responsible for extracting, transforming, and loading (ETL) existing data from Teradata to BigQuery, ensuring data integrity and optimizing performance.

During the migration process, I work closely with business teams and data analysts to understand specific reporting and analysis needs. I ensure that SQL queries are optimized and that data is securely accessible to authorized users.

Tasks:

o Definition, transcription of schemas and creation of tables on BigQuery.

o Development of orchestration workflows using Airflow.

o Creation and industrialization of ETL pipelines to extract data from Teradata, transform it according to specific needs, and load it into BigQuery in a reliable and optimized manner.

o Creation of environments and assignment of authorizations on GCP.

o Implementation of business and monitoring repositories on BigQuery.

o Implementation of business and monitoring repositories on BigQuery.

o Support of feature Teams in the use of GCP tools, pipelines, and resolve problems or technical questions encountered.

o Copying data to GCP.

o Improving the tools and processes used in the context of the migration, identifying opportunities for automation, performance optimization and task simplification.

Context description:

Migration of data warehouses to GCP (google cloud platform)

Mission description:

• Transfer of files from the server to Google Cloud Storage

• Feeding the data warehouse in BigQuery

• Transfer of data in batch from cloud storage to BigQuery

• Creation of table, pre-processing and transformation of data with BigQuery

• Creation of routines to include the calculation of business KPIs in new tables

Tools: Bigquery, Cloud Storage, Dataproc

Subject: Fraud detection based on dynamic rules Tasks accomplished: Building a prototype capable of modifying the static rules and machine learning parameters while the fraud application is running. Tools: Java, Drools, Spring