Member of Technical Staff — Pretraining Infra (Experienced)

Job description

## About Nuance Labs Nuance Labs is building photorealistic, real-time AI avatars with emotional intelligence: a full-duplex audiovisual system that can listen, speak, react, interrupt, and respond like a real person. We're a research company, with PhDs from MIT, UW, Oxford, CMU, and Johns Hopkins, and industry experience from Apple, Meta, Amazon AGI, and more. Backed by Accel, Lightspeed, South Park Commons, and NVIDIA, we combine frontier research with ruthless engineering needed for consumer-grade, real-time systems. The team is small, the work is real, and the problems are unsolved. ## How Nuance Differentiates Most conversational AI avatars today are hacks — a face slapped on a speech-to-speech pipeline, stuck in the uncanny valley: emotionless, mechanical, one-turn-at-a-time. Current systems take 2–5 seconds to respond; natural conversation requires sub-500ms. That's a 10x improvement, and it demands rethinking the entire stack. That rethinking starts with full-duplex: an AI that listens and speaks simultaneously, perceives emotion in real time, and responds with a face that actually reflects it. It's an extremely hard problem, and we're developing foundation models designed for it from the ground up. ## About the Role We're looking for a deeply technical MTS to own distributed training infrastructure for large-scale omni model pretraining. This role sits at the intersection of research, systems, and GPU-scale execution — building the training stack from 0→1 and scaling it: distributed execution, parallelism, GPU communication, data loading, checkpointing, observability, and debugging. Our models are omni from the ground up (audio, video, language, real-time full-duplex), which introduces systems challenges beyond standard LLM training: multimodal synchronization, long temporal context, variable sequence lengths, and tight memory/throughput constraints. High ownership. Direct impact on what models we can train, how fast research can iterate, and how reliably we scale. ## What You’ll Own - Own the distributed training stack for omni model pretraining, from 0→1 system design to 1→10 scaling across large GPU clusters. - Build and operate the core training runtime: job orchestration, distributed execution, checkpointing, recovery, monitoring, and debugging for long-running training jobs. - Optimize large-scale training performance across parallelism strategy, GPU communication, memory usage, data throughput, MFU, step time, and end-to-end training efficiency. - Build infrastructure for omni training workloads: high-throughput audio/video/text data loading, temporal alignment, variable sequence handling, multimodal synchronization, and memory-efficient training. - Evolve the platform as model architectures, training recipes, data mixtures, sequence lengths, hardware constraints, and research directions change. ## What We’re Looking For - Hands-on experience running large-scale distributed training jobs across large GPU clusters; experience at hundreds of GPUs minimum, 1,000+ GPUs a strong plus. - Deep understanding of distributed training mechanics: data/tensor/pipeline/sequence parallelism, gradient communication, collectives, mixed precision, activation checkpointing, optimizer state, memory pressure, and framework-level tradeoffs. - Strong understanding of GPU communication and performance debugging: NCCL, all-reduce/all-gather/reduce-scatter, communication-computation overlap, topology, synchronization, stragglers, low MFU, OOMs, checkpoint bottlenecks, and data starvation. - Practical experience with at least one major large-scale training stack such as Megatron, PyTorch FSDP, DeepSpeed, or equivalent internal infrastructure. - Understanding of omni or multimodal training challenges, especially audio/video/language data, long temporal context, variable sequence lengths, modality-specific bottlenecks, and high-throughput dataloading. - Strong software engineering fundamentals, curiosity, and adaptability to new model architectures, training frameworks, hardware constraints, and research ideas. ## Bonus Points - Prior 0→1 experience building large-scale training infrastructure or deeply modifying core training frameworks, runtimes, checkpointing, or debugging systems. - Experience training large omni or multimodal models involving audio, video, text, or long-context temporal data. - Experience with adjacent infrastructure areas such as RL/post-training, data infrastructure, synthetic data generation, evaluation, or serving. - Publications or substantial open-source contributions in ML systems, distributed systems, HPC, GPU performance, or training infrastructure. ## Compensation $300,000 – $400,000 base salary, plus meaningful equity. We think long-term ownership matters and structure equity accordingly. ## Logistics - **Location:** In-person in Seattle, five days a week — we believe in the compounding value of working shoulder-to-shoulder. - **Visa sponsorship:** We sponsor visas (O-1, H-1B, green card, etc.) from day one. - **AI-native tooling:** Do your best work with the best tools, including unlimited tokens. ## Benefits - **Health:** HSA plan with ~$2,000 in annual company contributions — roughly 2x what most big tech companies put in. - **Time off:** 15 days of PTO plus public holidays, and we close the office for a full week at year-end. - **Food:** Lunch, drinks, and snacks on us every workday — the small thing that quietly makes the day better. - **Commuter benefits:** We help cover the cost of getting to the office. - **401(k)** *Nuance Labs is an equal opportunity employer. We believe diverse teams build better AI.*