Tech Lead Manager, GPU Cluster Infrastructure

Job description

**This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Tech Lead Manager, GPU Cluster Infrastructure based in the United States.** We are seeking an experienced technical leader to shape and scale the infrastructure powering cutting-edge AI research. This role combines hands-on systems engineering with team leadership, offering the opportunity to influence the architecture, reliability, performance, and security of a rapidly expanding GPU cluster platform. You will oversee infrastructure spanning multiple providers, including compute, networking, storage, scheduling, and automation. Working closely with researchers and engineers, you will ensure large-scale experiments run efficiently and reliably. You will also establish technical standards, strengthen operational practices, and build a high-performing infrastructure team. This is an opportunity to make foundational engineering decisions in a fast-moving research environment where infrastructure directly enables scientific progress. ### Accountabilities - Define the platform's technical direction, architecture, and roadmap in collaboration with research teams and engineering stakeholders. - Design and oversee GPU cluster scheduling, storage, networking, and resource allocation across multiple infrastructure providers. - Remain actively involved in hands-on engineering, including architecture decisions, system implementation, troubleshooting, and resolving complex failures involving GPU hardware, network fabrics, node health, and distributed jobs. - Recruit, mentor, and develop a small team of senior infrastructure engineers, establishing clear ownership, priorities, project scopes, and feedback practices. - Lead security strategy for shared GPU infrastructure, including identity and access management, workload isolation, admission controls, and sandboxing for AI agents. - Establish and improve operational practices covering observability, incident response, fault tolerance, postmortems, and future on-call processes. - Monitor and improve platform performance through metrics such as GPU utilization, queue wait times, failure rates, and workload reliability. - Serve as the primary escalation point for research teams and infrastructure providers, turning recurring operational challenges into sustainable platform improvements. - Collaborate with researchers and engineers to ensure infrastructure evolves alongside changing research requirements and increasingly demanding experiments. ### Requirements - Experience leading engineers as a manager, technical lead, or project lead, including setting technical direction, defining project scope, prioritizing work, and providing constructive feedback. Previous direct people-management experience is not required, but a strong interest in managing people is essential. - At least five years of experience in systems or infrastructure engineering within production Linux environments, particularly GPU infrastructure, high-performance computing (HPC), or large-scale batch platforms. - Demonstrated ownership of production systems from initial design through deployment, operation, and continuous improvement. - Deep expertise in at least one of the following areas: workload scheduling, distributed storage, networking, infrastructure security, or GPU systems, alongside sufficient breadth to review designs across the other areas. - Hands-on experience operating production Kubernetes environments for GPU workloads with a batch scheduling layer such as Slurm, Kueue, Volcano, or an equivalent, including quotas, prioritization, preemption, and node health management. - Experience managing infrastructure as code and observability for production environments using tools such as Terraform, Ansible, Helm, ArgoCD, and Prometheus, or equivalent technologies. - Strong programming skills in at least one infrastructure-oriented language, such as Python, Go, Rust, or C++, with experience maintaining shared automation and services. - Excellent written and verbal communication skills, including the ability to prepare technical roadmaps, design documents, incident reports, and clear escalations for engineers, researchers, and providers. Additional skills that would be an advantage: - Experience with distributed training infrastructure, including multi-node PyTorch and NCCL debugging, NVIDIA drivers, GPU Operator, DCGM, InfiniBand or RoCE networking, and topology-aware workload placement. - Familiarity with distributed storage technologies such as VAST, Weka, Lustre, Ceph, or large-scale object storage, including checkpoint I/O optimization. - Expertise in cluster security, including RBAC, admission control, node and container hardening, sandboxed runtimes such as gVisor, Kata, or Firecracker, and the isolation of autonomous agents on shared infrastructure. - Knowledge of Kubernetes scheduler internals, custom controllers, gang scheduling, fair-share allocation, quotas, and the trade-offs between utilization, fairness, and latency. - Experience building multi-provider platforms that unify scheduling and storage across separate clusters, including environments with limited network connectivity or uneven data locality. - Previous experience establishing a new infrastructure team, hiring senior engineers, and implementing operational practices for incident management, on-call coverage, and technical reviews. Candidates who meet most, but not necessarily all, of these requirements are encouraged to apply. ### Benefits - Compensation: $225,000–$325,000, with the final package determined by experience, level, and location-based salary adjustments. - Health insurance: For eligible full-time US employees, the organization pays 94% of the insurance premium, with coverage commencing within one month of the start date. - Retirement plan: 401(k) with an employer match of up to 2% for eligible US employees. - Paid time off: 25 days of annual paid time off, accrued weekly, plus up to 10 days of paid sick leave per year for eligible US employees. - Paid leave: Bereavement, family, medical, and pregnancy disability leave, subject to applicable employment arrangements. - Remote-work support: A work computer and work-from-home stipend for eligible employees. - Office meals: Catered lunches and dinners on workdays for employees working from the Berkeley office. - Remote and in-person flexibility: Remote work is available in most countries, with in-person options in Berkeley, California, and Singapore. Visa sponsorship is available for eligible in-person employees. - International employment arrangements: Employees based in the United States or Singapore are employed directly by the organization. Candidates elsewhere may be employed through an employer-of-record provider or engaged as contractors, with benefits depending on the arrangement. - Full-time position: Approximately 40 hours per week, with working-hour overlap with Berkeley preferred. - Collaborative technical environment: Work closely with AI researchers and infrastructure engineers on challenging, large-scale systems where engineering decisions directly support research outcomes.