Human Evaluation Researcher

Job description

## About Nuance Labs Nuance Labs is building photorealistic, real-time AI avatars with emotional intelligence: a full-duplex audiovisual system that can listen, speak, react, interrupt, and respond like a real person. We're a research company, with PhDs from MIT, UW, Oxford, CMU, and Johns Hopkins, and industry experience from Apple, Meta, Amazon AGI, and more. Backed by Accel, Lightspeed, South Park Commons, and NVIDIA, we combine frontier research with ruthless engineering needed for consumer-grade, real-time systems. The team is small, the work is real, and the problems are unsolved. ## How Nuance Differentiates Most conversational AI avatars today are hacks — a face slapped on a speech-to-speech pipeline, stuck in the uncanny valley: emotionless, mechanical, one-turn-at-a-time. Current systems take 2–5 seconds to respond; natural conversation requires sub-500ms. That's a 10x improvement, and it demands rethinking the entire stack. That rethinking starts with full-duplex: an AI that listens and speaks simultaneously, perceives emotion in real time, and responds with a face that actually reflects it. It's an extremely hard problem, and we're developing foundation models designed for it from the ground up. ## Why this role exists "Does this avatar feel human?" is the question our whole company is organized around — and no automated metric can answer it. Lip-sync error and video quality scores say nothing about whether a smile landed as sincere or unsettling, whether a conversation felt warm or hollow, or whether someone would want to talk to our avatar again tomorrow. Your job is to turn human judgment into a reliable signal our researchers can train and ship against. You'll take the most ambiguous problems in our field (naturalness, emotional resonance, trust, presence) and design studies whose results people actually agree on. When two models differ, your study is the tiebreaker. When a model "feels off" and nobody can say why, your investigation finds the cause. This is a hands-on IC role and our first hire dedicated to human evaluation. You'll own it end to end (what to measure, how to measure it, who rates it, and what the results mean) working directly with the founders and the modeling team. Your findings will decide which models ship and what we train next. ## What you'll do - Design and run qualitative and quantitative studies of our AI avatars: side-by-side comparisons, controlled rating experiments, in-depth interviews, think-alouds, diary studies, and longitudinal panels. - Turn ambiguous judgments into instruments people can align on (rubrics, anchored scales, annotation guidelines) then measure and improve inter-rater agreement without flattening real signal. - Use ethnographic techniques (observation of live conversations, contextual inquiry, field work) to understand how people actually experience face-to-face AI, not just what they report in a survey. - Build the human-eval pipeline itself: participant panels, rater training and calibration, tooling, and an evaluation cadence tied to model releases. - Calibrate automated and model-based metrics (including LLM-as-judge) against human judgment, so the team knows when to trust them and when not to. ## You may be a good fit if you have - 5+ years designing and running human-subjects research in industry or academia (UX research, HCI, experimental psychology, behavioral science, or a related field). - Examples of study designs you can walk us through, especially ones where you got humans to converge on an ambiguous judgment (tone, emotion, quality, trust), including the rubrics, anchors, and protocols that made alignment possible. - Strong grounding in both qualitative methods (interviews, ethnography, contextual inquiry) and quantitative methods (survey and psychometric design, experimental design, statistics for rating and pairwise-comparison data). - Fluency with agreement and reliability. For example: You know your Cohen's kappa from your Krippendorff's alpha, and more importantly, how to raise them. - A bias toward running a scrappy, sound study this week over a perfect one next quarter, and the ability to explain findings crisply to ML researchers. ## Strong candidates may also have - Experience evaluating generative AI: avatars or digital humans, speech or video generation, conversational agents, or emotion expression and recognition. - A background in perceptual science or psychophysics (how people perceive faces, voices, motion, and emotion). MS/PhD in a related field welcome. - Familiarity with human evaluation at scale: crowdsourcing platforms, annotation tooling, golden datasets. - Enough statistics and scripting (Python or R) to analyze your own data. No candidate checks every box. If the "good fit" list sounds like you but your background is unconventional, we'd like to hear from you anyway. ## Compensation $160,000 – $190,000 base salary, plus meaningful equity. We think long-term ownership matters and structure equity accordingly. ## About the Role Model quality is ultimately a data problem. The best architecture and the best training run can't outrun bad, slow, or poorly curated data — and at the scale we're operating, the difference between a good data pipeline and a great one shows up directly in the model. We're looking for someone who lives and breathes data at scale. You know how to build pipelines that are fast, reliable, and maintainable — and you're just as comfortable taking a researcher's messy processing script and turning it into something that runs on petabytes as you are designing a new pipeline architecture from scratch. Research moves fast here, and the ability to productionize quickly without losing fidelity is the core skill. Our data is multimodal — video, audio, and text — and the processing requirements are demanding: high throughput, low error rates, and strict quality filters. There's a lot of interesting engineering work here, and the impact is direct and measurable. ## What You'll Do - Design, build, and operate large-scale data pipelines for ingestion, processing, filtering, and curation of multimodal training data (video, audio, text) - Take research-grade data processing code and turn it into robust, production-level pipelines — quickly and without losing correctness - Optimize pipeline throughput and efficiency at scale; identify and eliminate bottlenecks across compute, I/O, and storage - Build and maintain data quality systems — deduplication, filtering, validation, and quality scoring at scale - Manage petabyte-scale datasets: storage architecture, versioning, lineage tracking, and cost efficiency - Work closely with researchers to understand data requirements and translate them into scalable processing systems - Build tooling and infrastructure that makes the research team faster — efficient data access, reproducible processing, and fast iteration loops ## What We're Looking For - Proven experience building and operating large-scale data pipelines in production — you've processed data at a scale where naive approaches break - Strong proficiency with distributed data processing frameworks — Spark, Ray, Dask, or similar — and a clear sense of when to use each - Solid software engineering fundamentals: you write clean, testable, maintainable code and understand why that matters when pipelines run unattended at scale - Experience with multimodal data (video, audio) is a strong plus — understanding of formats, codecs, and processing libraries (FFmpeg, decord, etc.) - Familiarity with ML data pipelines specifically — understanding of how data quality and format affect model training - Ability to move fast: you can take a prototype script from a researcher and ship a production version in days, not weeks ## Bonus Points - Experience building data pipelines for large-scale model training (pre-training or fine-tuning) - Familiarity with data versioning and lineage tools (DVC, Delta Lake, Apache Iceberg, etc.) - Experience with streaming data pipelines or online data processing - Prior work at an AI lab, video platform, or other data-intensive company - Contributions to open-source data tooling ## Compensation $200,000 – $300,000 base salary, plus meaningful equity. We think long-term ownership matters and structure equity accordingly. ## Logistics - **Location:** In-person in Seattle, five days a week — we believe in the compounding value of working shoulder-to-shoulder. - **Visa sponsorship:** We sponsor visas (O-1, H-1B, green card, etc.) from day one. - **AI-native tooling:** Do your best work with the best tools, including unlimited tokens. ## Benefits - **Health:** We offer a variety of plans that meet your needs, including an HDHP with ~$2,000 in annual HSA contributions by the company (roughly 2x what most big tech companies put in). - **Time off:** 15 days of PTO, 10 public holidays, and we close the office for a full week at year-end. - **Food:** Lunch, drinks, and snacks on us every workday. We observe boba tea Tuesdays and Thursdays. - **Commuter benefits:** Utilize pre-tax money (up to $340/month) for parking and transportation. - **401(k):**4% match (100% of 1st 3% + 50% of next 2% contributions). *Nuance Labs is an equal opportunity employer. We believe diverse teams build better AI.*