About Hanbo
Data Engineer specializing in scalable data pipelines, analytics automation, and ETL systems using Python, SQL, and Rust. Developed open-source data infrastructure in Rust, featuring a Parquet-backed time-series table format with append-only commit log, DataFusion SQL support, and bitmap-based coverage tracking. Contributions include published benchmarks and methodology.
English
Native or bilingual
Remote only
Primarily works remotely
Experience
- Arizona DiamondbacksData EngineerHEALTH AND WELLNESSJanuary 2025 - Today (1 year and 7 months)Phoenix, AZ, USA- Built resilient ingestion services in Rust to pull from MLB REST APIs and TrackMan FTP, writing Hive-partitioned Parquet to S3; achieved ~20x faster extraction vs. prior client, with pagination/retry/backoff and schema-driftguards.- Automated daily batch pipelines primarily on Databricks using dbt formodeling and Databricks Asset Bundles for IaC/deploy.- Productionized with Terraform and GitHub Actions (CI/CD), plus Step Functions / EventBridge for orchestration and scheduling.- Cut Spark compute ~50% by preferring hash over sort aggregations, tuning partitioning/skew, and pruning; improved runtime stability and cost.- Implemented observability through Avro structured logs, data-quality checks (row-count deltas, dedupe, schema validation), and Slack alerts for jobs and SLAs.- Built a React + deck.gl field-view dashboard to visualize player/ball tracking with filters (game/inning/position), enabling fast, self-serve analysis for ops and R&D.- Operational backfills & hotfixes: Used Bash on WSL for targeted backfills/hotfixes and ad-hoc reprocessing; parameterized scripts + simple runbooks enabled same-day recovery when late/missing games occurred.- Partnered with analysts, data scientists, coaches, and R&D to define schemas and refresh SLAs, conducting demos and integrating feedback into dbt models and React/deck.gl dashboard.
- Stealth StartupData EngineerDIGITAL AND ITFebruary 2024 - December 2024 (10 months)Skills: Pandas, Extract, Transform, Load(ETL), SQL, Python- Developed data models to analyze educational trends and outcomes.- Collaborated with stakeholders to refine reporting processes and enhance data accuracy.- Automated data extraction and transformation processes using SQL andPython.
- Louisiana Department of EducationData Analytics EngineerJune 2023 - March 2024 (9 months)Baton Rouge, LA, USA- Skills: Pandas, Extract, Transform, Load(ETL), SQL, Python- Developed data models to analyze educational trends and outcomes.- Collaborated with stakeholders to refine reporting processes and enhance data accuracy.- Automated data extraction and transformation processes using SQL andPython.
Recommendations
Be the first to recommend Hanbo
Help this freelancer shine by sharing your experience working together.
These freelancer profiles also match your criteria
Agatha Frydrych
Backend Java Software Engineer
4.7
(3)
2
Baptiste Duhen
Fullstack developer
4.6
(4)
5
Amed Hamou
Senior Lead Developer
4
(2)
7
Audrey Champion
Web developer
4.3
(3)
4
Education
- Master's degreeTulane University2023Master's degree
- Bachelor's degreeEast China Normal University2021Bachelor's degree
Certifications
- Academy Accreditation - Databricks Lakehouse FundamentalsDatabricks2023
- Google Advanced Data Analytics CertificateCoursera2023