Technical Lead, AI Cluster Commissioning

Job description

## ROLE SUMMARY Firmus Technologies is seeking a Technical Lead, AI Cluster Commissioning, to join our team. You will work closely with internal teams, vendors, and partners and lead the technical commissioning of some of the world's most advanced AI computing platforms, ensuring they meet performance, reliability, and operational readiness requirements. You will provide day-to-day technical oversight and leadership to efficiently execute the commissioning plan, prioritise and coordinate parallel work streams, provide input and technical advice, escalate issues with the appropriate engineering teams, and ensure the project stays on schedule. You will also work with the teams to keep the tools, trackers, and technical documentation up-to-date, ensure a consistent commissioning process, implement suitable improvements, and make it repeatable for future projects. ## KEY RESPONSIBILITIES ### Commissioning Process Work with internal teams, partners, and vendors on component, systems, rack, and cluster-level testing. Apply your expertise to guide the compute, network, and storage teams through a thorough and efficient commissioning process from initial power-on through component and system testing, benchmarking, and acceptance testing to handover to operations. This also needs to consider data-center readiness, power and cooling availability, resources, and effective tools and communication. ### Team coordination The technical lead plays a pivotal role in coordinating the work between data center infrastructure teams, internal teams, partners and vendors. Apply your experience to ensure efficient communication and coordination between the teams, work with project managers to plan and execute the commissioning process and ensure appropriate resourcing. ### Leadership - Provide technical leadership and mentorship during commissioning activities. - Set technical priorities and coordinate execution across multiple workstreams. - Drive decision-making during critical incidents and escalations. ### Issue tracking and resolution Work with partners and vendors to efficiently triage, track, diagnose, and resolve technical issues arising during the commissioning process. Regularly review the issue trackers, ensure accurate and timely updates, provide input to the teams, liaise with engineering teams to escalate when required, collaboratively work on resolutions, and guide the project to a successful conclusion. ### Tools and Documentation Efficient, reliable, and repeatable processes require suitable tools and appropriate documentation. As we accelerate the pace of our projects, we need to constantly evolve and improve our processes, tools, and documentation. The technical lead has a central role in ensuring that the existing processes and tools are applied consistently and efficiently, they are documented suitably, but also continue to evolve and improve. This happens in close collaboration with internal and external teams who are responsible for various parts of the overall process, and work with and contribute to a variety of tools and sources of documentation. ### Partnership and Escalation Work with partners, vendors, as well as internal teams in close collaboration throughout the commissioning process. Identify risks, challenges, recurring issues, and bottlenecks and work in partnership with those teams to move the process forward towards successful on-time delivery of the platform. Escalate challenges, which put the project timeline at risk, and work with the teams involved to resolve them in a timely manner. Provide regular updates to project management and leadership on progress, risks, and challenges. ### Success Measures Success in this role is measured by: - Safe and on-time completion of commissioning activities. - Achievement of cluster performance and reliability targets. - Efficient triage and resolution of technical issues. - Consistent execution of commissioning processes. - Smooth transition of validated systems into production operations. ## SKILLS AND EXPERIENCE - Bachelor's degree in computer science or a related technical field, or equivalent practical experience. - 5+ years in HPC/AI/hyperscale infrastructure environments, including deployment, commissioning, monitoring and diagnostics. - Experience with large-scale cluster infrastructure including compute, networking, storage, and supporting management systems. - Strong Linux and server fundamentals. You can debug on the machine and reason across GPU, CPU, memory, PCIe, power and cooling. You automate in bash, Python or similar. - Familiarity with Ethernet fabrics, RoCE, NVLink, NVSwitch, and distributed AI workload validation. - Hands-on with GPU diagnostics and validation such as DCGM, NCCL/collective tests, stress testing and you've turned them into checks other people run. - Experience analysing hardware, network, and storage diagnostics and alerts. - Experience executing cluster validation, performance benchmarking, scale testing, and acceptance testing. - Experience leveraging AI-assisted troubleshooting, knowledge management, automation, or operational workflows to improve engineering productivity and incident response. - Willingness to undertake international and/or domestic travel for on-site deployments and commissioning as required. - Clear and effective written and verbal communication in English. ## Highly Desirable Experiences - Production experience on large GPU systems with tight GPU-to-GPU interconnect. - Background at an AI cloud, GPU cloud or HPC centre with involvement in system commissioning and testing. **Diversity** At Firmus, we are committed to building a diverse and inclusive workplace. We encourage applications from candidates of all backgrounds who are passionate about creating a more sustainable future through innovative engineering solutions. Join us in our mission to revolutionise the AI industry through sustainable practices and cutting-edge engineering. Apply now to be part of shaping the future of sustainable AI infrastructure.