Infrastructure Tooling & Observability Engineer( UK)

Job description

ABOUT US We’re a fast-growing GPU-as-a-Service provider, delivering scalable, high-performance compute infrastructure purpose-built for AI and HPC workloads. Operating across global data centres, we run mission-critical environments where uptime, throughput, and ultra-low latency are non-negotiable. ROLE OVERVIEW We are seeking an Infrastructure Tooling & Observability Engineer to act as a key engineering force within our global Infrastructure Operations organisation. Working closely with our SRE teams, you will translate high-level reliability objectives into scalable, production-ready systems that directly improve the resilience, efficiency, and performance of our global infrastructure. This role goes beyond traditional monitoring. You will help design and build the internal control plane that enables operations at scale across a rapidly growing GPU fleet. Your work will focus on transforming complex, high-volume telemetry—spanning logs, metrics, and events across HPC, networking, and platform layers—into actionable insight that drives operational excellence and proactive reliability. A core part of your responsibility will be developing intelligent observability and automation systems, including advanced alerting strategies, anomaly detection, and AI-driven tooling that reduces L1/L2 escalations and removes operational toil. You will also contribute to Continual Service Improvement (CSI) initiatives by building frameworks for reliability measurement, automated remediation, and system health evaluation. In addition, you will play a central role in turning SRE reliability initiatives into scalable engineering solutions. This includes designing and delivering capabilities such as inventory management systems, performance testing frameworks, and automated performance result collection. You will also help eliminate manual workflows involved in onboarding new regions, facilities, and clusters, embedding automation and standardisation into every stage of infrastructure deployment. As the organisation scales, you will act as a critical interface between operations and engineering teams. You will evaluate and mature internally built tooling—from capacity planning systems to autonomous remediation pipelines—and help integrate these capabilities into core infrastructure platforms to ensure consistent, high-performance, and highly reliable global operations. WHAT’S IN IT FOR YOU? Join a team building the internal platforms that enable large-scale infrastructure to operate reliably, efficiently, and at speed. As an Infrastructure Tooling & Observability Engineer, you will design and develop the systems that power visibility, automation, and operational intelligence across complex distributed environments. This role goes beyond traditional monitoring. You will build the internal control plane that transforms high-volume telemetry—logs, metrics, and events—into actionable insight for engineering and operations teams. Your work will improve observability across infrastructure systems, strengthen signal quality, and help teams understand and respond to system behaviour in real time. Working closely with SRE and infrastructure engineering teams, you will translate reliability goals into scalable, production-grade tooling. This includes frameworks for observability, alerting, anomaly detection, capacity planning, and service health tracking. A key focus of the role is automation. You will help eliminate manual processes across infrastructure operations, including environment provisioning, cluster onboarding, inventory management, and recurring operational workflows. You will also contribute to performance engineering initiatives, building tooling for testing, benchmarking, and automated results collection at scale. You will play a central role in turning SRE reliability initiatives into reusable engineering solutions, including automated remediation systems and tooling that reduces operational toil while improving system resilience. You can also expect: Exposure to large-scale distributed infrastructure systems Opportunities to shape foundational internal platforms A collaborative, engineering-led culture with strong ownership High-impact work spanning observability, automation, and reliability Close partnership with SRE and infrastructure engineering teams A fast-moving environment where tooling directly improves operational performance KEY RESPONSIBILITIES - Design, build, and evolve internal tooling and observability platforms that support large-scale infrastructure operations across distributed environments. - Develop systems that turn high-volume telemetry (logs, metrics, events) into actionable insight, improving visibility, alerting quality, and operational decision-making. - Translate SRE reliability requirements into scalable, production-ready software solutions, including automation for incident detection, prevention, and remediation. - Drive automation across infrastructure operations, reducing manual effort in areas such as environment provisioning, cluster onboarding, inventory management, and lifecycle workflows. - Build tooling for capacity management, performance testing, benchmarking, and automated collection and analysis of results. - Contribute to Continual Service Improvement (CSI) initiatives by identifying operational inefficiencies and delivering durable engineering solutions. - Work closely with SRE and infrastructure engineering teams to embed observability and reliability into core platform workflows. - Interface with Platform Engineering teams to ensure tooling aligns with broader orchestration and infrastructure strategy. - Integrate and extend existing systems written in Ruby/Rails and Go, contributing to a consistent and maintainable engineering ecosystem. - Develop and maintain automation workflows using Ansible and AWX. - Support CI/CD-driven operational tooling, including GitHub Actions and self-hosted runners. ESSENTIAL SKILLS & EXPERIENCE - Degree in Computer Science/Software Engineering, or equivalent experience - 6–8 years of experience in infrastructure engineering, DevOps, SRE, and/or software engineering roles, with a strong focus on operational systems. - Proven experience in at least one recent DevOps or software engineering role, building or maintaining production infrastructure tooling or platform systems. - Experience working in large-scale or distributed infrastructure environments (hyperscale, enterprise, or similarly complex systems). - Strong programming ability in at least one of: Ruby (Rails), Go, or similar systems languages, with willingness and ability to work across multiple languages and codebases. - Hands-on experience with infrastructure automation tools such as Ansible and orchestration platforms such as AWX. - Strong experience with observability systems, including the Grafana stack (Prometheus, Loki, Mimir, and Grafana Alloy). - Familiarity with low-level telemetry and infrastructure protocols such as SNMP and syslog. - Experience working with Kubernetes or similar orchestration platforms in production environments. - Understanding of API design and integration patterns, particularly REST-based services and service-to-service communication. - Experience building and maintaining CI/CD pipelines, including GitHub Actions and self-hosted runners. - Strong understanding of operational reliability concepts, including monitoring, alerting, capacity planning, and incident response. - Comfortable working closely with SRE, Platform Engineering, and infrastructure teams to translate operational needs into maintainable software systems. Preferred Qualifications - Kubernetes Certified Administrator - Cloud-native observability training courses, attendance at industry conferences in this field - CompTIA+ Security Qualifications - LPI/LPIC certification