Join us as an experienced Senior Data Engineer, you will be part of a team who design, build and maintain scalable, secure, and high-performance data solutions on Microsoft Azure.

• You will support mission-critical operations across oil trading, refining, and retail fuel distribution by building robust data pipelines, integrating real-time telemetry, and ensuring compliance with regional and international regulations.
• This role sits within the Oil Assets IT team, reporting to the Data Engineering Team Lead, with occasional travel to affiliates in South Africa and Brazil.

Responsibilities:

• Design and manage scalable data pipelines and ETL/ELT processes using Azure Data Factory, Synapse Analytics, and emerging platforms such as Microsoft Fabric and Databricks.
• Collaborate with peers to create and maintain data models and databases in Azure SQL DB and Azure Data Lake.
• Ensure data quality, lineage, and availability through rigorous validation, testing, and monitoring practices.
• Integrate data from ETRM systems, business applications, refinery control systems, and retail station point-of-sale networks.
• Implement real-time analytics solutions using Azure Event Hubs, Stream Analytics, and IoT Hub.
• Liaise with multiple functional groups across Glencore Group, the Oil Department and the industrial assets, to provision and deploy infrastructure within Azure.
• Implement data governance and security policies using Microsoft Purview and Azure RBAC.
• Ensure data quality, lineage, and availability for business-critical applications.
• Deliver high-velocity solutions supported by strong coding practices and automation.
• Build and maintain multiple pipelines in parallel, managing context switching effectively.
• Stay current with modern data technologies and trends, including AI/ML, and advise on their responsible adoption.
• Operate independently, driving individual workstreams while contributing to team-wide initiatives

The ideal candidate will have:

• Bachelor’s Degree in Computer Science, Information Technology, or equivalent experience.

Certifications

• Microsoft Certified: Azure Data Engineer Associate.
• Microsoft Certified: Azure Solutions Architect Expert.
• Microsoft Certified: Azure Fundamentals (AZ-900).
• Microsoft Certified: Azure AI Engineer Associate.
• Apache Spark Developer Certification.

Competency:

• Data Engineering and Modelling: Strong grasp of core data modelling concepts and techniques, experience designing and managing ETL/ELT pipelines using Azure Data Factory and Synapse Analytics.
• Programming Languages: Expert proficiency in SQL, Python, and PySpark for data transformation, validation, and analytics.

Cloud Technologies (Azure):

• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Azure Databricks
• Azure Data Lake Storage Gen2
• Azure SQL
• Azure Functions
• Version Control and CI/CD: Experience with Git/GitHub and CI/CD pipelines using GitHub Actions.

Experience

• Seasoned data engineer with 7+ years of experience, and a strong foundation in designing and managing cloud-native data pipelines, data models, and analytics solutions.
• Have a hands-on expertise in Azure-based technologies including Synapse Analytics, Data Factory, and Azure SQL, with forward-looking experience in Microsoft Fabric and Databricks.
• Operates in a fast-paced, business-facing environment, delivering high-quality solutions across multiple parallel workstreams.
• Self-driven and adaptable, with a deep understanding of the downstream oil and gas domain and a commitment to continuous learning and innovation.

Desirable

• Visualisation and Reporting: Skilled in building dashboards and visual narratives using Power BI.
• Infrastructure and Automation: Familiarity with infrastructure-as-code tools such as Terraform; experience provisioning resources in Azure.
• Data Governance and Security: Experience implementing data lineage, cataloguing, and access controls using Microsoft Purview and Azure RBAC.
• Real-Time and Streaming Data: Exposure to Azure Event Hubs, IoT Hub, and Azure Stream Analytics for real-time data processing.

Advanced Tools and Platforms:

• Apache Airflow for workflow orchestration
• Microsoft Power Platform for low-code solutions
• Azure DevOps for collaborative development and deployment
• NoSQL/Big Data technologies (e.g., Cosmos DB, Hadoop, or similar)
• Machine Learning Engineering: Understanding of ML workflows and integration into data pipelines.