Program Manager, Human Evaluations (Recommendations)

Job description

**This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Program Manager, Human Evaluations (Recommendations) based in the United States.** The Program Manager, Human Evaluations (Recommendations) will lead the design and delivery of human evaluation programs that measure and improve personalized content recommendations. This role combines strategic program management, evaluation design, and operational execution to ensure recommendation systems deliver relevant, diverse, and meaningful experiences. Working closely with Product Management, Machine Learning, Data Science, UX, and Engineering teams, you will translate product priorities into scalable evaluation frameworks and measurable success criteria. You will oversee external vendors, manage rater onboarding and quality assurance, and establish consistent guidelines for evaluating subjective content recommendations. Through data-driven process improvements and strong cross-functional collaboration, you will enhance evaluation accuracy, efficiency, and reliability. This is an opportunity to influence how audiences discover films and television content while helping advance innovative personalization technologies in a dynamic, fast-paced environment. ### Accountabilities - **Program Strategy and End-to-End Delivery** - Lead human evaluation and data operations initiatives from initial scoping and planning through execution, quality validation, and final delivery. - Partner with Product Management to translate recommendation strategies and roadmap priorities into evaluation designs, success metrics, project milestones, and timelines. - Coordinate cross-functional stakeholders across Product, Machine Learning, Data Science, UX, Engineering, and external vendor teams to ensure alignment and successful execution. - Track project progress, manage dependencies, anticipate risks, and proactively resolve bottlenecks to maintain delivery schedules and quality standards. - Ensure evaluation programs support continuous improvements in recommendation relevance, diversity, consistency, and overall member experience. - **Evaluation Frameworks and Quality Assurance** - Develop clear, scalable evaluation guidelines, scoring rubrics, and assessment protocols that support consistent human judgment across recommendation use cases. - Design evaluation approaches for both objective criteria and subjective, taste-based judgments, accounting for rater variability and potential bias. - Oversee rater onboarding, training, calibration, and quality assurance workflows to improve consistency and reliability. - Pilot and refine evaluation tasks, instructions, and scoring methodologies to improve clarity, feedback quality, and inter-rater reliability (IRR). - Collaborate with product and technical teams to establish shared definitions of recommendation quality and translate evaluation findings into actionable recommendations. - **Vendor and Operational Management** - Manage external human annotation and evaluation vendors, overseeing capacity planning, service-level agreements (SLAs), performance monitoring, and issue escalation. - Optimize operational metrics, including cost per task, throughput, turnaround time, and time required to achieve acceptable inter-rater reliability. - Monitor vendor performance and ensure evaluation activities meet established standards for accuracy, consistency, efficiency, and timely delivery. - Identify opportunities to improve workflows, strengthen quality controls, and scale evaluation operations while managing costs effectively. - Synthesize operational feedback and evaluation results into clear recommendations for continuous process improvement. ## Requirements - **Relevant Experience:** At least 4 years of experience in human evaluation, data collection, data labeling, annotation operations, or related program management within machine learning environments. - **Program Management:** Demonstrated success leading complex, cross-functional programs from planning through delivery, with strong accountability for timelines, quality, and outcomes. - **Vendor and Team Management:** Experience managing external annotation vendors, human raters, or data-labeling teams, including performance monitoring, capacity planning, quality management, and cost optimization. - **Evaluation Design:** Strong understanding of evaluation methodologies, scoring rubrics, annotation guidelines, and quality control systems for human-generated data. - **Operational Excellence:** Proven ability to improve annotation program efficiency, including throughput, turnaround time, cost per task, and inter-rater reliability. - **Content Knowledge:** Working familiarity with film and television content, including genres, formats, talent, and the ability to assess whether groups of titles are coherent and appropriate together. - **Technical Tools:** Experience working with human labeling or annotation platforms and coordinating evaluation workflows in technology-driven environments. - **Analytical and Communication Skills:** Excellent written and verbal communication skills, with the ability to synthesize complex feedback, communicate findings clearly, and recommend practical improvements. - **Judgment and Quality:** Understanding of how subjectivity and bias can affect taste-based assessments, with the ability to design evaluation processes that encourage consistency and reliable results. - **Organizational Skills:** Strong attention to detail, prioritization, and problem-solving abilities, with the capacity to manage multiple workstreams while maintaining a broader strategic perspective. - **Collaboration and Adaptability:** Ability to build credibility with product managers, data scientists, and machine learning engineers, navigate ambiguity, and turn evolving requirements into actionable plans. **Preferred Qualifications** - Experience with personalization, recommendation systems, search, or ranking evaluation. - Familiarity with how offline human evaluations complement online experimentation, including A/B testing and engagement metrics. - Experience evaluating recommendation lists or entire page-level experiences rather than assessing individual items in isolation. - Experience collaborating with Product Managers and Data Scientists to define metrics and establish evaluation criteria. ## Benefits - **Competitive compensation:** Annual salary range of $230,000–$360,000, depending on location, experience, skills, and relevant market benchmarks. - **Flexible compensation structure:** The opportunity to choose the annual mix of salary and stock options. The compensation structure does not include bonuses. - **Comprehensive healthcare:** Health insurance plans and mental health support. - **Retirement benefits:** 401(k) retirement plan with employer matching. - **Equity opportunities:** Access to a stock option program. - **Financial protection:** Disability coverage, health savings accounts (HSAs), flexible spending accounts (FSAs), and life and serious injury benefits. - **Family support:** Family-forming benefits and paid leave programs. - **Flexible time off:** Full-time salaried employees have access to flexible paid time off. - **Paid leave:** Additional leave programs to support employees through different life circumstances. - **Remote work:** A remote position available to eligible candidates based in the United States. - **Meaningful impact:** The opportunity to help improve personalized content discovery through rigorous human evaluation and innovative machine learning technologies. - **Collaborative environment:** Work alongside multidisciplinary teams focused on product innovation, data-driven decision-making, and continuous improvement.