GPU Cluster Engineer, Systems & Platform

Job description

Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million-dollar funding and direct sponsorship from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack powering frontier AI models and real-time applications. ABOUT THE ROLE We are looking for a GPU Cluster Engineer to own the entire software stack of our GPU clusters — from kernel tuning and GPU drivers up through schedulers, containers, and ML frameworks. While our Hardware Operations team keeps the physical machines healthy and connected, you define what a production-ready node looks like in software: you author the images, playbooks, and pipelines that take a freshly provisioned server to a fully validated GPU node, and you keep the fleet consistent, upgradable, and fast. You will serve two demanding customer groups — our foundation model training teams and our model serving/product teams — ensuring both run on correctly configured, well-managed, high-performance infrastructure. Key Responsibilities - OS Bring-Up & Node Lifecycle Engineering - Golden Images & Automated Bring-Up: Own the node software definition — versioned OS images, kernel tuning (NUMA, hugepages, IRQ affinity, cgroups), GPU/NIC driver stacks — and the automated pipeline that takes a node from base OS to production-ready. - Validation & Burn-In: Build automated acceptance suites (DCGM diagnostics, nccl-tests/RCCL tests, bandwidth and topology checks, HPL) that gate every node before it enters a scheduler pool. - Fleet Maintenance: Execute rolling kernel/driver/toolkit upgrades with minimal disruption to running workloads; enforce configuration consistency, detect drift, and maintain the driver ↔ CUDA/ROCm ↔ framework compatibility matrix across the fleet. - Self-Healing Operations: Automate detection of unhealthy nodes (Xid/ECC errors, link flaps, thermal throttling), with cordon/drain/reboot/re-image workflows and clean handoff to Hardware Operations for physical repair or RMA. - Configuration Management & Automation - Infrastructure as Code: Manage all node and cluster configuration through Ansible/SaltStack playbooks in Git, with peer-reviewed changes, CI validation, and canary rollouts before fleet-wide deployment. - Provisioning Pipelines: Build and maintain image/provisioning tooling (PXE, MaaS, Packer, or similar) so new or re-imaged nodes are reproducible, not hand-crafted. - Operational Tooling: Develop Python/Bash tooling for cluster operations, health reporting, and workflow automation. - Orchestration & Scheduling (Kubernetes & Slurm) - Kubernetes for Serving: Deploy and operate GPU-enabled Kubernetes for inference workloads — NVIDIA GPU Operator, device plugins, node feature discovery, topology-aware scheduling, and MIG/MPS partitioning where appropriate. - Training Schedulers: Operate Slurm (or Run:AI) for multi-node training — partitions, QoS, preemption, accounting, and container integration (enroot/pyxis). - Container Platform: Maintain base images, registries, and the NVIDIA Container Toolkit / ROCm container stack; keep training and serving images lean, current, and reproducible. - GPU Driver & ML Stack Engineering - Driver & Runtime Lifecycle: Build, deploy, and debug the full accelerator stack — NVIDIA (CUDA toolkit, cuDNN, NCCL, Fabric Manager) and AMD (ROCm, RCCL) — including kernel modules (DKMS), GPUDirect RDMA/Storage, and the RDMA software stack (MOFED/DOCA). - Framework Environments: Maintain curated, optimized PyTorch and JAX environments with sane dependency and version management for researchers and production services. - Distributed Performance: Tune NCCL/RCCL across NVLink/NVSwitch and InfiniBand/RoCE fabrics, ensure topology-aware job placement, and run continuous communication/throughput benchmarks to catch regressions. - Advanced Debugging & Observability - Escalation Point: Own the hard problems — NCCL hangs and timeouts, CUDA memory leaks, ROCm kernel crashes, straggler nodes, and unexplained throughput drops. - Observability: Own software-layer monitoring (DCGM exporter, Prometheus/Grafana, alerting) plus job-level GPU utilization and cluster efficiency reporting. Qualifications - Must-Haves: - 5+ years in systems/infrastructure engineering with significant GPU cluster, HPC, or large-scale ML infrastructure experience. - Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field. - Deep Linux internals expertise: kernel modules/DKMS, systemd, cgroups, NUMA, and system performance tuning. - Hands-on experience with NVIDIA (CUDA) and/or AMD (ROCm) driver and runtime stacks on modern accelerators (H200/B200, MI325x/MI355x class), including kernel-level debugging. - Production Kubernetes experience with GPU workloads, plus working knowledge of HPC schedulers (Slurm/Run:AI) — or the reverse (deep Slurm, working K8s). - Strong configuration management experience (Ansible or SaltStack) with Git-based, code-reviewed infrastructure workflows. - Provisioning and image tooling experience (Packer, MaaS, Foreman, Terraform, or similar) for automated, reproducible node builds. - Client-side experience with distributed filesystems (Lustre, GPFS, Weka) and checkpoint I/O optimization. - Container fluency: Docker/containerd and the NVIDIA Container Toolkit or ROCm equivalent. - Proficiency in Python and Bash for automation and tooling. - Working knowledge of NCCL and RDMA networking (InfiniBand/RoCE, GPUDirect) and of PyTorch/JAX runtime behavior. - Nice-to-Haves: - Experience directly supporting foundation model training teams — multi-node job failure debugging, checkpoint pipeline tuning, and framework-level performance triage — ideally in a startup or research-heavy environment. - Experience deploying and tuning inference/serving stacks (vLLM, Triton Inference Server, TensorRT-LLM) for latency and throughput targets. - GPU/system profiling tools: Nsight Systems/Compute, rocprof, perf, eBPF. BENEFITS INCLUDE - Medical, dental, and vision insurance - 401k plan - Daily lunch, snacks, and beverages - Flexible time off - Competitive salary and equity EQUAL OPPORTUNITY Sciforium is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.