Manager, Supercomputer Site Reliability Engineering - Memphis

Job description

SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this possible, with the ultimate goal of making life multiplanetary and expanding consciousness to the stars. SpaceXSI is building Super Intelligence to help humanity understand the universe. Our teams are building the full Super Intelligence stack: from our Colossus supercomputers, to Grok models, to our suite of applications, including Grok Bot, Grok Build, Grok Imagine, Grok Voice, and X. We're creating a powerful flywheel where more compute trains better models, better models power more useful products, and what we learn from people using those products shapes what we build next. ### ABOUT THE ROLE: SpaceXSI operates multi-campus, hyperscale GPU infrastructure where cluster availability directly determines training throughput, customer handover commitments, and capital efficiency of the fleet. The Manager of Supercomputer Site Reliability Engineering will be responsible for both compute and infrastructure performance, and will be responsible for significantly increasing the SRE team size. On the compute side, focus will be put on continuous site monitoring, cross-discipline incident command, deep hardware root-cause analysis (RCA), and emphasis on software systems that orchestrate repair velocity. Additional focus will be on physical infrastructure reliability as well. Understanding system margins and resiliency will be key. Site Reliability engineering manager owns >99% compute uptime outcome, drives unplanned facilities compute losses to 0, ensures efficient allocation of > 3GW site power, upgrade robust fault-free megapack failover systems capable of supporting all halls for >4 hours (160 MWhr per hall), drives robust hardware inventory systems and repair velocity to eliminate hardware chokepoints. ### **RESPONSIBILITIES:** - Serve as the senior operational authority for the SRE organization and represent production reliability concerns in engineering and leadership discussions. - Lead integrated Supercompute systems engineering reliability team responsible for reactive and increasingly proactive cluster and facilities health and reliance initiatives - Build out the site reliability engineering sub-teams: compute hardware failure analysis, software tools, cluster monitoring, and integrated site reliability engineering Drive operational superiority - Define SRE organizational structure and develop culture of 24/7 operational excellence - Physics limit anomaly response times and robust proactive health processes - Define on-call responsibilities, incident management protocols, leadership structure, and communication discipline - Develop training schemes for tough, competent, and disciplined monitoring and response teams - Establish clear operational interfaces with Datacenter Operations, engineering teams, vendors, and other service owners - Own vendor oversight and accountability mechanisms - Drive mean time to repair toward physics limits - Define and continuously improve operational procedures and analysis documentation mechanisms with strong emphasis on intelligent automation - Lead robustification of documentation workflow - Ensure efficient signal to noise in execution - Define operational targets, health indicators, alerting standards, and incident severity definitions that facilitate measurable team performance insights and accountability mechanisms Cultivate engineering excellence - Drive high technical standard for hardware root cause analysis and remediation teams ensuring industry-leading hardware failure diagnostic expertise - Integrate with platform engineering teams to develop sophisticated hardware and system understanding to drive high operational reliability - Drive repeat hardware issues to a minimum and lead automation efforts to eliminate manual overhead for unavoidable remediation needs ### **BASIC QUALIFICATIONS:** - Bachelor’s degree and 3+ years of experience working in a large scale operations with 2+ years leading people leaders of technical teams or 10+ years of experience working in a large scale operations with 5+ years leading people leaders of technical teams. ### **PREFERRED SKILLS AND EXPERIENCE:** - Proven ability to lead large, multi-site, 24/7 operations organizations in fast-paced, high-responsibility settings. - Deep expertise in server hardware, cluster reliability, and data center technologies, from deployment through lifecycle management. - Experience supporting compute-heavy environments like AI, machine learning, or high-performance computing at scale. - A track record of owning uptime, Service Level Agreements, or reliability metrics for large compute clusters. - Experience leading site reliability engineering or equivalent reliability-focused teams, including root cause analysis and procedure ownership. - Strong analytical skills and the ability to explain technical concepts clearly to diverse audiences, from technicians to executive and tenant partners. - A history of partnering with vendors at scale, driving mean time to repair down, and scaling operations across multiple sites. - Familiarity with tooling and automation (e.g., Jira, Python, Bash) used to monitor cluster health and improve team efficiency. - Enthusiasm for SpaceXAI's mission to accelerate human discovery and unravel the universe. - Ability to thrive in a dynamic, mission-focused environment with on-call ownership of cluster-impacting events. ### **ADDITIONAL REQUIREMENTS:** - Willingness to travel frequently to data center locations to support operations across sites. - Physical capability to handle data center tasks, including lifting up to 50 lbs unassisted, standing for long periods, and occasional ladder use. - Must be willing to work extended hours and/or weekends as needed. *SpaceXSI is an equal opportunity employer. For details on data processing, view our**[Recruitment Privacy Notice](https://x.ai/legal/recruitment-privacy-notice).*