Forward Deployed Infrastructure Engineer - Eastern US

Job description

# **Who We Are** Hyperbolic Labs is on a mission to democratize AI by breaking down the barriers to computing power with our Open-Access AI Cloud. By making better use of idle computing resources across the globe, we offer an innovative GPU marketplace and AI inference service that promise affordability and accessibility for all. As pioneers at the intersection of AI and open-source technology, we believe in an open future where AI innovation is limited only by imagination, not by access to resources. We're looking for forward-thinking individuals who share our passion for making AI universally accessible, secure, and affordable. Join us in building a platform that empowers innovators everywhere to turn their visionary AI projects into reality. NOTE: This role is focused on EASTERN TIME ZONE ## **About the Role** Our reserved customers run large multinode GPU clusters, and when those clusters misbehave the problem is rarely simple. You are the engineer embedded with those customers: you stand their cluster up, you hand it over, and you stay with it. You do not own tickets. You own environments. Technical Support Engineers own the ticket lifecycle and pull you in when an issue needs real depth: multinode collective performance, hardware faults, fabric problems, or a provider who needs to be told what is wrong with their hardware. One thing we will be straight about, because it shapes the job. We aggregate capacity from suppliers rather than owning most of the hardware ourselves. That means a real part of this role is technical liaison work: proving where a fault actually lives, taking it to the provider with evidence, and coordinating the fix on the customer's behalf. The engineers who enjoy this role are the ones who find that interesting rather than frustrating. **Who You Are** - Cluster stand-up and handoff. Build, validate, and benchmark new customer clusters, then hand them over with documentation the customer's own engineers can work from. - Deep escalations. Multinode and NCCL performance debugging, GPU and hardware faults (XID and ECC errors, lspci, dmesg), driver and fabric issues, container and scheduler problems. - Provider escalation and coordination. Maintenance windows, RMAs, hung nodes, and disputed fault attribution. You bring the evidence that makes the provider act, and you keep the customer informed while it happens. - Embedded ownership of named accounts. You are the engineer your customers know by name. You learn their workload, not just their infrastructure, and you tell them what to change before they hit the wall. - Proactive monitoring. Own the monitoring and alerting we put in front of customer clusters (Grafana, Prometheus) so we find faults before the customer reports them. - Tooling and pushing work down. Automate the repeat work and turn your own escalations into runbooks the L1 tier can run. Anything you fix three times should stop reaching you. - Deep Linux experience and total comfort in the CLI, including in someone else's broken environment. - Hands-on multinode GPU experience: NCCL, InfiniBand or RoCE, collective performance debugging, topology and placement. - Hardware fault triage on GPU nodes: XID and ECC errors, dmesg, lspci, nvidia-smi, thermal and power faults. - Experience provisioning and operating GPU clusters with Kubernetes, Slurm, or both. - Genuinely comfortable customer-facing, including delivering bad news and saying “this is ours” or “this is the provider’s” with confidence. - Sound judgment on when to keep digging and when to escalate. **Preferred Qualifications** - Experience with parallel filesystems (Weka, Lustre, GPFS) and high-performance storage. - Grafana, Prometheus, or similar observability stacks in production. - Prior forward deployed engineer, solutions architect, or technical account manager experience. - Infrastructure as code (Terraform, Ansible) and CI for cluster provisioning. - Exposure to model training or inference workloads from the practitioner side. *Hyperbolic is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.*