Juan Carlos Gomez

Developed a Proof of Concept (POC) to test Snowflake's various ingestion types (batch, near real-time, and real-time) along with an analysis of costs, latency, and performance for storing large volumes of information. This involved configuring a development environment in Snowflake with the creation and configuration of warehouses, roles, permissions, and database objects. A stage was defined to access data from S3, S3 buckets were created along with AWS roles and policies to be visible from Snowflake, and S3 notifications were configured for near real-time ingestion using Snowflake's Snowpipe. Column-level masking policies and row-level data access policies were created. For batch ingestion, COPY was used; for near real-time ingestion, Snowpipe was used; and for real-time ingestion, a Java program was used that ingested at a rate of 60,000 records per second and bursts of 2,000 records per second via the Snowflake API. The information was stored in both tables and dynamic tables to avoid ETL processes that move data from the BRONZE to SILVER and GOLD layers. This last layer was created to enable the exploitation of the ingested information.

Data Architect at Singular Bank

- Development and implementation of the bank's corporate Data Lake using its various layers.

- Development and implementation of the bank's corporate Data Warehouse on Snowflake.

- Analysis and development of the entire Data Lake and Data Warehouse architecture (security, role definition, permissions, layer folder structure, ELT process structure and analysis, data lifecycle, table and field types, data quality rules, backup policies, definition and setup of data processing framework, etc.).

- Automation of ETL processes using SSIS, Data Factory, Python, SP technologies.

- Migration of SSIS 2016 to 2019 ETL processes from file system to catalog.

- Technologies: SSIS, Snowflake, SQL Server, Talend, Data Factory, Python, Shell scripts, Azure, AWS.

- Leading the design and implementation of DataOps policies, including naming standards for tables in Data Lake and Data Warehouse, quality rules definition, process naming, nominal and service user definition, permissions, CI/CD, definition of policies and standards for moving between environments, etc.

- Supporting the design and implementation of the company's Data Governance policies according to DAMA methodology.

- Development of policies, guidelines, and strategies that ensure homologation, standardization, scalability, and adequate attention to business needs.

- Definition of batch, near real-time, and real-time data architectures for data analysis and exploitation through various tools like Tableau, Power BI, Metabase.

Development of ingestion processes for S3 files in near real-time into Snowflake's SILVER layer.

Development of quality control module processes in the SILVER layer and configuration of rules for each file.

Development of traceability (logs) module processes.