Senior Data Engineer

Job description

Entity: Technology Job Family Group: IT&S Group Job Description: Let me tell you about the role At the heart of bp's technological endeavors is the Technology organization. We are the driving force behind the technology that fuels bp's operations, from upstream energy production to downstream delivery to our esteemed customers. Our diverse teams cater to a broad spectrum of interests, encompassing everything from infrastructure and backend services to customer-centric web and native applications. With a keen emphasis on agility, we foster using native AWS and Azure services as our infrastructure foundation, including serverless architectures. Our ethos is not just about using the best tech for the task but also encouraging an environment where our software and platform engineers thrive, learn, and continuously evolve. Key Responsibilities Drive engineering best practices through code reviews, technical standards, CI/CD practices, and automation Lead source control, release management, and deployment processes for data engineering solutions. Facilitate technical discussions, architecture reviews, and technology decision-making across teams. Mentor engineers and contribute to building a high-performing engineering culture through knowledge sharing and technical leadership. Ensure compliance with information security, governance, and operational standard methodologies. Support the reliability, resilience, and continuous improvement of data platforms and services. Design, build, and operate scalable, secure, reliable, and efficient data products, pipelines, and platforms. Own the end-to-end lifecycle of data solutions, ensuring operational accountability, supportability, and continuous improvement. Collaborate with business stakeholders, engineers, architects, analysts, and data scientists to deliver high-quality data solutions. Design and maintain data models, ingestion frameworks, and transformation pipelines that ensure data quality, integrity, and performance. Implement DataOps best practices including CI/CD, Infrastructure as Code, automated testing, monitoring, and deployment automation. Build observability, alerting, and automated data quality controls to ensure platform reliability and rapid issue detection. Maintain metadata, lineage, documentation, runbooks, and support processes to improve governance and operational readiness. Lead incident resolution, root cause analysis, and reliability improvements to reduce recurring issues and improve service outcomes. Ensure compliance with security, privacy, governance, and enterprise architecture standards. Provide technical leadership through architecture guidance, mentoring, code reviews, and engineering best practices, including good team work What you will need to be successful (experience and qualifications) 8+ years of experience in Data Engineering, Software Engineering, or related disciplines. 5+ years of hands-on Python development experience. 3+ years of experience with PySpark and large-scale distributed data processing. Shown experience delivering and operating enterprise-scale data platforms and pipelines. Experience working in DataOps, DevOps, SRE, or production support environments. Technical Skills Strong expertise in SQL, database design, and data modelling. Experience designing end-to-end data architectures including ingestion, transformation, storage, governance, and consumption layers. Strong AWS experience, including services such as S3, Glue, Lambda, Athena, Redshift, and EventBridge. Hands-on experience with Databricks and modern Lakehouse architectures. Experience with Git, CI/CD, Infrastructure as Code, and deployment automation. Operational Excellence Strong understanding of observability, monitoring, alerting, and platform reliability. Experience implementing data quality controls, lineage, metadata, and governance practices. Experience with incident management, root cause analysis, and continuous service improvement. Knowledge of security, privacy, compliance, and operational risk management. Leadership Excellent stakeholder management and communication skills. Shown ability to influence technical decisions, mentor engineers, and drive engineering excellence. Demonstrated ownership mindset with a focus on reliability, quality, and business outcomes. About bp At bp, we provide the following environment and benefits to you: A company culture where we respect our diverse and unified teams, where we are proud of our achievements and where fun and the attitude of giving back to our environment are highly valued. Possibility to join our social communities and networks Learning opportunities and other development opportunities to craft your career path Life and health insurance, medical care package And many other benefits. We are an equal opportunity employer and value diversity at our company. We do not discriminate based on race, religion, colour, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, perform crucial job functions, and receive other benefits and privileges of employment. Travel Requirement No travel is expected with this role Relocation Assistance: This role is not eligible for relocation Remote Type: This position is a hybrid of office/remote working Skills: Cloud Platforms, Cloud Platforms, Collaboration, Communication, Configuration management and release, Continuous deployment and release, Creating a high performing team, Database Design, Digital Project Management, Documentation and knowledge sharing, Emerging technology monitoring, Facilitation, Information Security, Mentoring, Metrics definition and instrumentation, NoSql data modelling, Problem Solving, Relational Data Modeling, Risk Management, Scripting, Secure development, Service operations and resiliency, Software Design and Development, Solution Architecture, Source control and code management {+ 5 more} . Legal Disclaimer: We are an equal opportunity employer. We do not discriminate on the basis of protected characteristics like race, religion, color, sex, national origin, sexual orientation, veteran status or disability status. Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process, please contact us . If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks.