Abdelkabir B.

Collection and analysis of Trading operations.

The project involves collecting large volumes of data from multiple sources, consolidating, enriching, and saving them in a local database. This was the batch sourcing/processing part, aimed at managing large data volumes from various sources (API, SFTP, databases).

My tasks included:

• Developing and optimizing ETL pipelines using Python and Pandas. The code architecture followed object-oriented programming principles and SOLID principles.

• Designing and maintaining SQL queries on SQL Server and Oracle. Using indexes to speed up data updates.

• Implementing and promoting Behavior-Driven Development (BDD) principles and Unit Tests. Test coverage was 100%, and BDD covered almost all scenarios.

• Using Git for version control, actively participating in code reviews and collaborative development workflows to maintain high-quality standards.

• Analyzing several Python code components and SQL queries to identify bottlenecks; several improvement avenues were proposed, adopted, and added to the backlog.

• Managing and deploying database changes using Liquibase (via Jenkins and XLR).

• Actively participating in backlog population while also engaging in Backlog Refinement ceremonies.

• Participating in the continuous improvement of agile practices.

• Creating documentation for developed components and deployment processes.

• Guided the team in technical choices, application design, and architecture.

• Transformed an application running via a terminal into an Event-Driven application with Kafka and Faust-Streaming:

- Transformed the project's backend into a Faust-streaming application to allow for efficient application scalability when needed.

- Faust agents listen to and consume Kafka messages, initiating processing.

- Generated data is then sent to an S3 bucket.

• Migrated and containerized several Django projects from Python 2 to Python 3.8 and from Windows to CentOS.

• Contributed to the setup and maintenance of CI/CD processes on GitLab.

• Implemented GIT flow to make code review and delivery more flexible and minimize production incidents.

• Collaborated with DevOps teams to set up API authentication configurations (using KeyClock).

• Created 'Health Checks' to monitor the availability of Python applications; these 'Health Checks' were subsequently integrated into Grafana Dashboards.

• Mentored junior developers.

• Led Daily Meetings and Retrospectives within an Agile methodology framework (for 1 year).

Technical Stack: Celery, Django, FastApi, Flask, Pandas, Pydantic, duckdb, RabbitMQ, Kafka, Faust Streaming, Prefect, CI/CD (Gitlab), CentOS, Docker, Unit, Integration, and Acceptance Tests, Oracle, Sqlite, Postgresql, Memgraph, ELK, KeyClock, Jira, Confluence

Within the INFRA department, I hold the position of Senior Python Developer, contributing to the design and implementation of internal projects aimed at simplifying the deployment and operation of FTTH telecommunication networks. The projects I have implemented:

• Setup of an industrialization process for a series of Python scripts, orchestrating their sequential execution with dependency and log management. The application was developed with Django, using Celery as a worker.

• Creation 'from scratch' of an ETL for extracting progress deliverables: design and development of components for extracting and transforming attribute and geometric data from Oracle databases, primarily using pandas & geopandas: analysis of cable capacities, tracing optical routes, creating fictitious infrastructures, verifying optical continuity, and generating deliverables.

• Creation of a data migration project from an Oracle database to another Oracle database using FastAPI and Pydantic for data validation and error report generation.

• Implementation of a graph database with Memgraph to simplify the creation of optical routes.

• Creation of several REST APIs with FastAPI and Flask