Senior Manager - Principal Data Engineer (London)

Job description

Grade: B Job Title: Principal Data Engineer Team: Clara Technology & Solutions Locations: UK Capability: Audit Service line: Audit Central Contract type: Permanent Summary of role purpose: The Principal Data Engineer will be accountable for shaping the technical direction for data engineering initiatives and services within Technology & Solutions. This role combines deep technical expertise in Databricks and cloud data engineering with proven team leadership capabilities and a passion for driving innovation. The successful candidate will be accountable for delivering enterprise-scale data engineering solutions, championing modern engineering practices including AI-assisted development, and fostering a high-performing team culture focused on excellence, collaboration, and continuous growth. Description of the role: Data Engineering & Delivery Excellence • Design and deliver enterprise-scale data solutions on cloud platforms (Azure preferred), leveraging Databricks as the core data engineering platform • Build and optimize sophisticated ETL/ELT data pipelines using PySpark, SQL, and Python, orchestrating complex data workflows through Databricks Workflows, Delta Live Tables, Azure Data Factory • Implement and maintain Delta Lake storage architectures and Unity Catalog governance frameworks, ensuring data quality, security, and compliance across the data estate • Design data solutions for scalability, performance, resilience, and operational excellence, embedding enterprise-grade standards from inception through production deployment • Lead technical discovery and requirements gathering with senior stakeholders, translating business needs into actionable data platform strategies and technical roadmaps • Integrate data platforms with business intelligence and analytics tools including Databricks AI/BI and Power BI to enable self-service analytics and data-driven decision making Engineering Standards & Modern Data Practices • Establish, evolve, and enforce data engineering standards, coding practices, and quality gates that enable safe, scalable, and maintainable data platform delivery • Champion modern software engineering practices within data engineering contexts, including Git version control workflows, automated testing frameworks, and CI/CD deployment pipelines for data workloads • Drive comprehensive observability, monitoring, and alerting for data pipelines and platforms, ensuring operational readiness and rapid incident response • Promote AI-augmented data engineering practices, leveraging GitHub Copilot, Claude Code, and other approved AI coding assistants to enhance productivity while maintaining code quality and standards Technical Leadership & Team Development • Provide technical direction and engineering accountability for data engineering initiatives, leading teams through hands-on contribution and solving key business challenges • Coach and mentor data engineers at various career levels, fostering a culture of technical excellence, continuous learning, and innovation adoption • Manage delivery of data engineering projects using Agile methodologies (Scrum, Kanban), balancing technical execution with people leadership and stakeholder management • Foster collaboration across engineering, architecture, and product teams, breaking down silos and promoting knowledge sharing across the organization • Represent data engineering in portfolio planning discussions, technical governance forums, and architectural review boards The candidate Experience and knowledge requirements: Required Experience in data engineering disciplines, with advanced, hands-on expertise in Databricks platform including PySpark framework, Delta Lake storage format, and Unity Catalog governance Extensive experience in designing advanced data models, including medallion architectures, data lakes, and enterprise data warehouses Expert-level proficiency in SQL for complex data transformation, optimization, and analytical workload development Advanced proficiency in Python for data engineering, pipeline automation, and integration development Deep expertise in PySpark for data processing and transformation Proven mastery of cloud platforms with strong preference for Microsoft Azure Expertise in ETL/ELT development and data pipeline orchestration (e.g. Databricks Workflows, DLT, ADF) Extensive experience with Git version control, GitHub collaboration workflows, and modern CI/CD practices including automated testing frameworks and deployment pipeline automation for data engineering workloads Hands-on expertise with Agile software development methodologies (Scrum, Kanban) and DevOps practices applied to data engineering contexts Demonstrated capability to balance technical execution with people management responsibilities and product accountability Proven analytical thinking and advanced problem-solving skills with demonstrated ability to tackle complex, enterprise-scale technical challenges Strong decision-making capabilities with proven ability to evaluate technical trade-offs and make pragmatic choices under pressure and ambiguity Strong track record in managing and coaching engineering teams (minimum 3+ years in Tech Lead roles) Experience with AI-native software delivery and specification-driven development, including the use of AI-assisted engineering tools, automated quality assurance, modern testing practices, and highly automated CI/CD workflows within enterprise-scale environments Desirable Experience implementing data governance frameworks, data lineage tracking, and role-based access control (RBAC) using tools like Apache Atlas, Microsoft Purview Experience building event-driven architectures and real-time data streaming pipelines using tools like Apache Kafka, AWS Kinesis, or Azure Event Hubs Familiarity with agentic AI frameworks and architectures such as Microsoft Agent Framework or equivalent frameworks in Python, TypeScript, or other languages Experience architecting data solutions for scalability, performance optimization, security hardening, and operational excellence in production environments Knowledge of Infrastructure as Code (IaC) frameworks including Terraform or Azure Bicep for data platform provisioning and management Background in professional services, consulting, or client-facing technical delivery environments Active contribution to open-source data engineering projects or participation in data engineering and cloud communities Qualifications (optional): Relevant associate-level or expert-level professional certifications in Microsoft Azure (e.g., Azure Data Engineer Associate, Azure Solutions Architect Expert), Databricks (e.g., Databricks Certified Data Engineer Professional), or modern development frameworks Bachelor's degree in Computer Science, Data Science, Engineering, or related technical discipline (or equivalent practical experience) KPMG overview: KPMG in the UK is part of a global network of firms that offer Audit, Legal, Tax and Advisory services. Through the talent of over 16,000 colleagues, we bring our creativity, insight and experience to solve our clients’ and communities’ biggest problems. We’ve been doing this for more than 150 years. We aim to be universally recognised as a place for great people to do their best work. A firm known for our collaborative and inclusive culture, using technology to empower and equip our people to deliver outstanding work with real flexibility – through inspiring workspaces, innovative ways to collaborate and hybrid ways of working. With offices across the UK, we work with everyone from small start-ups and individuals to major multinationals, in virtually every industry imaginable. Our work is often complex, yet our mission is simple: To support the UK in a connected world. It guides everything we do, underpinned by our values: Courage, Integrity, Excellence, Together and For Better. KPMG are proud to be an inclusive, equal opportunity employer and we seek to attract and retain the best people from the widest possible talent pool. As a member of the Business Disability Forum we’re committed to ensuring that all candidates are treated fairly throughout the Recruitment Process. We pride ourselves on being a place where your individuality is valued; you can be yourself and still achieve your potential. We believe that your individuality helps us to deliver the best results to our clients. Diversity of background, diversity of experience, diversity of perspective - that’s the KPMG difference.