• Primary on-call engineer and release manager for 300+ Airflow pipelines and Spark workloads, coordinating weekly multi-environment deployments and resolving hundreds of production incidents to sustain 97% SLA adherence for business-critical analytics.
• Designed and implemented new Airflow patterns (custom operators and API integrations such as PagerDuty), standardising ELT workflows for 5+ stakeholder teams, boosting Airflow adoption, and reducing incident detection time by up to 90%.
• Led critical components of the platform-wide Apache Spark 2→3 upgrade, automating test configuration generation with Python and Bash (50+ hours saved), migrating 30+ PySpark applications, and unblocking releases by resolving 300+ unit-test failures caused by breaking changes.
• Automated manual processes in weekly release cycles via Python-driven Jenkins pipelines, integrating Bitbucket (Stash), Confluence, and Jira APIs; eliminated 5+ hours of manual effort per week and streamlined the release process.
• Led an end-to-end SOx audit automation initiative with Tech Risk and KPMG, delivering a config-driven ELT framework that consolidated change metrics into a canonical Oracle source and Power BI reporting — saving ~2 days of manual audit effort per service, per audit.

#airflow #pyspark #jenkins #oracle-database #sql #bash #etl/elt #power-bi #api-integration #git

Kafka Platform Modernisation (Auckland Transport)

• Built an IaC-driven CI/CD pipeline using Terraform and Azure DevOps to automate Kafka cluster provisioning and deprovisioning; supported migration from on-prem Kafka to Confluent Cloud, improving platform reliability and maintainability.

#terraform #azure #azure-devops #confluent-cloud #bash

Cloud Data Warehouse Accelerator (Internal project)

• Developed a reusable dbt package to accelerate Snowflake and BigQuery warehouse deployments using Kimball dimensional modelling; presented the solution to senior leadership, driving adoption across consulting teams.

#dbt #snowflake #bigquery #docker #sql #jinja #azure-devops #kimball-dimensional-modelling

• Developed an end-to-end product segmentation pipeline for 40,000+ SKUs using OpenAI GPT-3, improving inventory planning accuracy and delivering a Streamlit-based self-service UI for stakeholders.
• Applied GPT-3 to summarise, extract keywords, and perform sentiment analysis on historic customer interaction logs, surfacing actionable insights for marketing and service teams.
• Built an internal chatbot using Azure Bot Service powered by the company's knowledge base, reducing information search time for staff from minutes to seconds.

#openai-gpt-3 #streamlit #python #sql

• Developed ETL pipelines to ingest and process semi-structured data across multiple formats (CSV, XML, HTML), ensuring schema validation and data quality prior to loading into Elasticsearch using Python.
• Built automation scripts in Python and Bash to support pipeline operations and ongoing maintenance.
• Designed and performance-tested Apache Spark jobs to transform datasets into big data formats (Parquet, Avro, ORC), significantly improving processing efficiency and storage performance.
• Created Kibana dashboards to monitor pipeline health and performance.
• Performed exploratory data analysis to extract domain insights and support downstream analytics use cases.

#apache-spark #elasticsearch #kibana #hadoop #python #bash #etl