Senior Enterprise Technology Engineer

Job description

Entity: Technology Job Family Group: IT&S Group Job Description: We are looking for a Senior Enterprise Tech Engineer to establish and lead the Data Engineering squad within our global data team, supporting the Trading Analytics and Insights (TA&I) organisation within the Trading and Shipping (T&S) Entity. You will play a pivotal role in building and leading a high-performing, global team of engineers responsible for designing and implementing robust, enterprise-scale data pipelines, while mentoring and coaching a team of 8-10 engineers. Operating at true enterprise scale across multiple environments and subject area communities, you will bring deep expertise in Python and SQL data engineering, ETL, cloud data platforms and orchestration at scale. Working with TA&I sponsors and Technology product owners, you will define, design and implement the data engineering roadmap, balancing the demands of safe and reliable daily operation with the need to continually explore and bring new capabilities — including Generative AI — to market. This role requires expertise across: data engineering, platform engineering, software engineering, capability analysis and design and forensic problem diagnosis. You will be able to define and apply meticulous principles of automation, CI/CD and DevOps, and to lead a demanding front-office trading organisation in fully exploiting the potential of its data platform. In This Role You Will Strategic Leadership: Establish and lead a high-performing, global data engineering squad, fostering a collaborative, results-oriented environment and owning the technical direction for the team's deliverables. Design, develop and implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow. Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time streams). Leverage Generative AI platforms and frameworks — agents, RAG, prompt and context engineering, evaluations and AI SDLC — to deliver novel capabilities for the trading organisation. Tap into expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) to build, deploy and manage data pipelines in a secure and cost-effective manner, with particular focus on cloud cost optimisation and Total Cost of Ownership (TCO). Coach and mentor a team of 8-10 data engineers, providing guidance on standard processes, technical development and career growth, and fostering a culture of knowledge sharing and continuous learning. Lead and collaborate effectively within a global, cross-functional team, building strong relationships across borders and time zones. Drive the adoption of Agile methodologies (Kanban/Scrum) and use Azure DevOps (ADO) to plan, track and manage team workload efficiently, refining operational and capability backlogs. Create compelling data visualisations using advanced tools (e.g. Tableau, Power BI) to communicate insights to both technical and non-technical audiences. Champion automation tools and standard processes to deliver reliable, repeatable results, ensuring all configuration is maintained under source control with automated CI/CD deployment and test. Develop and maintain clear, concise technical documentation for data pipelines and procedures to ensure knowledge transfer and maintainability. Embody the spirit of continuous improvement through agile sprint retros, RCAs and design review workshops; coaching junior team members and using data to drive where the team invests. Essential Experience and Requirements Bachelor/Master's degree or equivalent experience in Computer Science, Engineering, Information Systems or other numerate fields. Demonstrated ability of 10 years as a Data Engineer with a proven track record of success in leading and mentoring teams. Experience of building market data solutions within a trading or financial services environment. In-depth expertise in Python (Pandas, web scraping libraries such as Selenium, Beautiful Soup and Requests, and machine learning libraries such as TensorFlow and PyTorch). Generative AI techniques including agents, skills, prompt engineering, RAG, AI SDLC, context engineering and evaluations. Mastery of SQL (including advanced optimisation techniques) and experience with relational databases (PostgreSQL preferred), including data modelling and design skills. Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data. Proven experience with Apache Airflow for data orchestration and scheduling at scale. Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must. Definition and design of the full SDLC (standards and tooling): development, test and deployment, using Git source control and ADO (or Jenkins) pipelines. Experience working in a Linux environment and proficiency with Git version control. Excellent written and verbal communication skills, with the ability to present complex technical concepts clearly to both technical and non-technical audiences. Demonstrated experience working in a global environment and building strong relationships with teams across borders. A passion for building high-performing teams and fostering a culture of innovation and continuous learning. Desirable Criteria Experience with Databricks for large-scale data processing. Experience with advanced data visualisation tools (e.g. Tableau, Power BI) with a focus on insightful storytelling. Knowledge of Microsoft SQL Server administration and query performance tuning. Familiarity with core cloud concepts: subscriptions, resource groups, IAM. Knowledge of the commodities front office domain e.g. financial and physical instruments, market data prices. Why Join Us? At BP, we support our people to grow in a diverse and exciting environment. We believe that our team is strengthened by diversity. There are many aspects of our employees’ lives that are significant, so we offer benefits to enable your work to fit with your life. These benefits can include flexible working options, a generous paid parental leave policy, excellent retirement benefits, among others. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform crucial job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation. Reinvent your career as you help our business meet the challenges of the future. Travel Requirement No travel is expected with this role Relocation Assistance: This role is not eligible for relocation Remote Type: This position is a hybrid of office/remote working Skills: Cloud Platforms, Cloud Platforms, Collaboration, Communication, Configuration management and release, Continuous deployment and release, Creating a high performing team, Database Design, Digital Project Management, Documentation and knowledge sharing, Emerging technology monitoring, Facilitation, Information Security, Mentoring, Metrics definition and instrumentation, NoSql data modelling, Problem Solving, Relational Data Modelling, Risk Management, Scripting, Secure development, Service operations and resiliency, Software Design and Development, Solution Architecture, Source control and code management {+ 5 more} . Legal Disclaimer: We are an equal opportunity employer. We do not discriminate on the basis of protected characteristics like race, religion, color, sex, national origin, sexual orientation, veteran status or disability status. Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process, please contact us . If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks.