Irving Diaz

Strategic data technical architect, responsible for the design and implementation of enterprise-scale data architectures for international logistics operations. Specializing in Microsoft Fabric, Azure AI Search, and the implementation of RAG solutions with vector databases, aiming to improve operational efficiency through advanced data integration and AI-powered solutions.

Architect of a Microsoft Fabric solution processing over 20 TB of daily logistics data, improving data search times by 40%, while reducing operational costs and manual processing time.

Implementation of a RAG and vector database-based Azure AI Search system, enabling semantic search and analysis of logistics documents with high accuracy.

Developed a real-time data streaming architecture capable of processing over 50,000 logistics events per minute.

Optimization of Fabric SQL database performance through advanced vector indexing and AI query tuning.

Seasoned professional specializing in creating data ecosystems and leading digital transformation initiatives, expert in Lakehouse architecture to foster innovation and data-driven decision-making. Application of Data Mesh principles to establish decentralized, scalable, and federated governance data management, enhancing business efficiency.

Development of a data strategy aligned with digital transformation objectives, defining key areas and the data action plan.

Design of data integration plans promoting continuity and availability of inter-system flows, with mastery of complex pipelines and real-time analytics.

Leadership in the selection, evaluation, and design of scalable cloud solutions, aligned with cloud adoption frameworks to ensure flexibility and cost control.

Determination of architectures and data models for migrating the monolithic data warehouse to the Databricks Lakehouse, adopting Data Mesh for multiple business flows.

Introduction of federated and collaborative governance on Azure platforms, fostering a democratized data culture.

Implementation of robust data management practices: ownership, quality standards, lifecycle management, and exhaustive catalogs ensuring accuracy and usability.

Security reinforcement: access control, anonymization, and encryption via Databricks governance, ensuring regulatory compliance.

Detailed documentation.

Expert in designing and deploying data integration solutions in complex ecosystems (on-premises & cloud). End-to-end integration management in Palantir Foundry, ensuring enterprise-scale performance and reliability. Supervision of platform migrations, including the massive transition from Cloudera CDP to Azure Databricks.

Implementation of real-time and batch ingestion pipelines from various sources: SAP, MDM, IoT, RDBMS, DWH, BW, REST API, via Palantir connectors, sync, HyperAuto2, and Apache Kafka.

Design of data lineage pipelines automating data flow between systems, with integrity verification and continuous monitoring (health checks, data expectations).

Creation of business ontologies for semantic and interoperable data integration.

Mapping of needs, source-target mapping, design of transformation processes ensuring consistency and quality, strong expertise in real-time data and analytics connectors/protocols.

Maintenance of ETLs (PySpark on Databricks, Azure Data Factory, Apache Nifi) aligned with the Medallion architecture (Bronze, Silver, Gold).

Data quality and integrity validation procedures, automatic error management to ensure reliability.

Global integration of operational data (PostgreSQL, SQL Server CDC) into Databricks Lakehouse.

Agile Scrum project management practice on Azure DevOps for data/architecture projects.