Senior Data Engineer - Databricks (3rd Party Contractor)

Job description

to Position: Senior Data Engineer (Databricks & Cloud Data Platforms) Reporting to: Practice Head: Data and Automation ROLE OVERVIEW As a Senior Data Engineer, you will play a critical consulting role in designing, building, and modernising enterprise data platforms, with a strong focus on Databricks lakehouse implementations across cloud environments. You will work closely with solution architects, analysts, data scientists, and client stakeholders to deliver secure, scalable, and high-performing analytics platforms that drive real business impact. This is a hands-on technical role suited to someone passionate about solving complex data challenges and delivering best-in-class cloud-native solutions. responsibilities RESPONSIBILITIES Design and implement scalable Databricks-based data platforms Build robust ETL / ELT pipelines for batch and streaming workloads Lead data migration and modernisation initiatives from legacy platforms to cloud-based lakehouse architectures Develop high-performance data processing solutions using PySpark, Python, and SQL Implement Delta Lake, Unity Catalog, and modern governance frameworks including Purview Design enterprise-grade lakehouse and medallion architectures Optimise Spark workloads for performance, scalability, and cost efficiency Build orchestration solutions using Databricks Workflows, ADF, Airflow, or similar tools in Azure/AWS Collaborate with technical and business stakeholders to define solution requirements Mentor junior engineers and contribute to technical best practices within the Data & AI practice Produce technical documentation, architecture artefacts, and reusable delivery assets IDEALLY YOU ARE: A strong technical leader who can own complex delivery outcomes Comfortable engaging directly with clients and stakeholders A proactive problem-solver with excellent analytical skills Able to clearly communicate technical concepts to both technical and non-technical audiences Passionate about mentoring and uplifting engineering teams Delivery-focused, quality-driven, and adaptable in consulting environments MINIMUM EXPERIENCE 8+ years in Data Engineering projects 3+ years of hands-on Databricks delivery experience Deep expertise in Python and PySpark Strong understanding of the Medallion Architecture and Lakehouse Architecture Hands-on experience building and maintaining live Databricks AI/BI Governance Dashboards using SQL Strong experience with Azure data services including Synapse, ADF, OneLake, Purview Practical exposure to AWS data platforms including Glue, S3, Redshift Proven track record delivering data migration and modernisation projects Experience working in consulting or client-facing environments Bachelor’s degree in computer science, Engineering, Information Systems, or equivalent practical experience TECHNICAL REQUIREMENTS Databricks Expertise (Essential) Cloud Data Platform Experience (Azure Fabric, AWS) (Essential)