Ramzi Chebbak

Context: The project aimed to migrate an application from On-premise cloud (OVH) to Microsoft Azure to benefit from managed services, improve resilience, and reduce operational costs.

Build & Run:

• Analysis of the existing infrastructure (code, databases, files, dependencies) to assess needs and dependencies.

• Definition and implementation of the target architecture on Microsoft Azure, adhering to best practices (security, scalability, cost).

• Development of Terraform files to provision Azure resources such as:

• Network: Virtual Network, Subnets, Network Security Groups (NSG).

• Compute: App Services, Azure Kubernetes Service (AKS), ACR.

• Database: Azure SQL Database, Cosmos DB, Storage Accounts.

• Configuration of Terraform states in a secure Azure Blob Storage for infrastructure tracking.

• Use of Azure DevOps and GitHub Actions to automate infrastructure deployment via CI/CD pipelines.

• Implementation of secret management mechanisms with Azure Key Vault and their integration into Terraform.

• Creation of reusable Ansible playbooks to manage configuration and multi-environment deployments (Dev, Test, Prod).

• Data migration and validation of service integrity after deployment.

• Post-migration performance optimization and monitoring with Azure Monitor and Application Insights.

• Collaboration with development teams to adapt applications to the cloud environment.

• Documentation of deployment processes and training of internal teams.

Context: Supervise and manage a BIGDATA production platform (Hortonworks & Cloudera), deploy Project applications, Configure and set up CI/CD chains and automation scripts to deploy applications on environments (dev, homologation, prod).

• Perform daily Morning-Check and communicate on Alerts.

• Supervise and monitor the production infrastructure (KAFKA, HDFS, ELASTIC, COUCHBASE …).

• Handle Level 2 and 3 incidents (node failures, HDFS block loss, YARN queue saturation, …).

• Analyze bottlenecks and tune configurations (memory, CPU, partitions, etc.) to improve processing times and scalability.

• Monthly on-call duties for critical production chains.

• Implementation of an ELK monitoring, supervision, and alerting system to monitor cluster metrics (latency, throughput, resource utilization).

• Create Jenkins CI/CD pipelines and deploy jobs via Ansible Tower.

• Develop Ansible Roles for automating environment creation (GITHUB, JENKINS, AWX…).

• Automate the production deployment of projects/applications (Spark, HIVE, …).

• Automate application deployment via Github action.

• Installation and deployment of KAFKA / COUCHBASE / ELK & OPENSEARCH clusters via Ansible Playbooks / Roles.

Context: The project's objective is to design and implement an auto-scaling system on AWS capable of automatically adjusting resources based on application load while optimizing operational costs.

Build & Run:

• Needs analysis to manage application load variations during peak traffic periods.

• Creation of custom AMI images with Packer integrating software prerequisites (Java, Python, etc.).

• Automation of cluster deployment with Ansible, including the installation of HDFS, YARN, Spark, and Kafka components.

• Deployment of a highly available architecture via Terraform based on:

• AWS EC2 with Auto Scaling Groups for dynamic scaling.

• Application Load Balancer (ALB) for traffic distribution.

• Integration of Amazon RDS for the database and S3 for static resource storage.

• Implementation of auto-scaling policies based on CloudWatch metrics (CPU, memory, etc.).

• Use of tools like Apache JMeter to simulate massive loads.

• Adjustment of thresholds and minimum/maximum instance sizes as needed.

• Deployment of the ELK stack to centralize and analyze HDP cluster logs.

• Collection of scaling logs (AWS Auto Scaling, CloudWatch) and application logs via Filebeat and Logstash.

• Real-time visualization of scaling events in Kibana (number of active instances, scaling delays).

Technical Environment AWS (EC2, Auto Scaling Groups, S3, CloudWatch, ALB), Hortonworks Data Platform (HDFS, YARN, Spark, Kafka), ELK (Elasticsearch, Logstash, Kibana, Filebeat), Packer, Ansible, Terraform, Apache JMeter.