Evaluation Research Manager

Job description

## About Aaru Aaru builds simulations of human behavior. Each simulation contains a population of AI agents, each representing a person who could plausibly exist in the real world and capable of making decisions within a modeled environment. Companies and institutions use these simulations to test consequential choices before committing—from product launches and pricing decisions to strategic communications and policy changes. Building a useful simulation requires more than generating plausible text. Populations must represent real people and groups; predictions must be calibrated; simulations must remain coherent as conditions change; and the product must make the resulting evidence legible enough to support real decisions. We are a small, in-person team in New York. We work with urgency, high ownership, and intellectual honesty. We expect people to surface inconvenient evidence, change their minds quickly, and carry important work all the way to a result. ## About Evaluation Research Evaluation Research determines whether Aaru's populations, predictions, and end-to-end simulations correspond closely enough to the real world to support consequential decisions. The team defines what should be measured, develops the methods for measuring it, and produces the evidence Aaru uses to improve its systems and describe their capabilities. The function builds both **rails** and **carts**. Rails are reusable evaluation infrastructure: datasets, harnesses, libraries, experiment standards, leaderboards, reporting systems, and ways to translate technical evidence into decisions. Carts are the specific evaluations that run on those rails: a historical backtest, a prospective forecast study, a population-coherence test, a reproduction of an observed behavioral pattern, or an end-to-end comparison with a resolved outcome. Evaluation Research is not conventional QA and it is not an internal approval service. It is an independent research function that works closely with the teams building Aaru's systems while preserving the ability to reach and communicate inconvenient conclusions. ## The role As Evaluation Research Manager, you will lead a focused team of Evaluation Researchers and research engineers. You will translate Aaru's evaluation charter into a coherent portfolio of studies, datasets, and shared infrastructure, and you will be accountable for the quality, pace, and usefulness of the team's work. Managers at Aaru remain researchers. You will design studies, write analysis code, inspect individual failures, review statistical and measurement choices, and directly contribute to the hardest evaluations. You will also set priorities, hire exceptional people, develop the team, provide candid feedback, and create the operating mechanisms that keep protected evidence independent while making diagnostic evidence available quickly. You will work across Population Research, Prediction Research, Simulation Engineering, Product Engineering, Research Product, Deployment, and company leadership. The job requires both scientific independence and practical judgment: an evaluation must be rigorous enough to support a real claim and diagnostic enough to help a team improve the system. ## What you will do - Build, lead, and develop a high-performing team of Evaluation Researchers and research engineers. - Turn broad questions about realism, accuracy, calibration, usefulness, and decision quality into measurable constructs, decisive experiments, and explicit decision criteria. - Set a focused evaluation agenda across population construction, predictive systems, individual agent behavior, group dynamics, and end-to-end simulations. - Decide which evaluation infrastructure should become a reusable organizational rail and which questions require a purpose-built study. - Establish standards for baselines, temporal holdouts, prospective testing, contamination control, statistical power, uncertainty, subgroup analysis, and reproducibility. - Build tests of population quality that assess individual coherence, joint and conditional distributions, representation of rare but plausible profiles, and whether a profile induces behavior consistent with the person it represents. - Evaluate forecasts and other predictive outputs using calibration, proper scoring rules, ranking quality, selective prediction, temporal validity, subgroup performance, and the real cost of different errors. - Compare simulations with transactions, product usage, behavioral traces, operational outcomes, market movements, resolved events, and longitudinal decisions. - Design end-to-end studies that determine whether improvements to a component actually improve the decision-relevant output customers receive. - Find failures hidden by aggregate metrics, especially those concentrated in important subgroups, rare cases, changing environments, or ambiguous ground truth. - Create diagnostic evaluations that help researchers localize why a system failed and distinguish a real general improvement from benchmark-specific optimization. - Partner with Simulation Engineering to make evaluations repeatable, versioned, scalable, and integrated into development and release workflows without compromising protected holdouts. - Convert production incidents, customer surprises, and deployment failures into durable test cases and better measurement methods. - Review evidence used in product, customer, or public claims and ensure that conclusions are reproducible, appropriately scoped, and honest about uncertainty and limits. - Communicate negative, null, and inconclusive results with the same precision and urgency as positive findings. - Recruit exceptional researchers, set clear expectations, provide direct feedback, develop independent research judgment, and address performance problems early. ## Representative research and leadership problems You might be responsible for situations such as: - Two teams disagree about whether a system improved because they use different metrics and test sets. Identify the underlying construct, choose the right evidence, and create a shared evaluation that resolves the disagreement. - A population looks plausible one profile at a time but fails to reproduce important real-world relationships. Build measurements that expose the gap and help Population Research identify its cause. - A prediction method is well calibrated overall but systematically overconfident for a high-value subgroup. Determine whether the issue is data coverage, model structure, selection, condition shift, or the evaluation itself. - An offline benchmark has become a development target and is beginning to leak into decisions. Redesign the evaluation system so teams can iterate quickly without exhausting the integrity of the final holdout. - A component metric improves, but customer decisions do not. Determine whether the metric is invalid, the effect is too small, downstream components erase the gain, or the product is presenting the result incorrectly. - Ground truth is delayed, noisy, incomplete, or open to multiple interpretations. Design a study that remains useful without pretending that the label is cleaner than it is. - A customer outcome contradicts the simulation. Reconstruct the information available at decision time, identify the relevant comparison, and convert the failure into a fair and reproducible test. - Evaluation work is becoming a collection of bespoke notebooks. Choose the common abstractions, data contracts, and reporting systems that should become durable rails without freezing the research too early. - A favorable result is strategically important but methodologically weak. Communicate the limitation clearly, resist pressure to overclaim, and propose the fastest credible path to stronger evidence. ## How we work A useful evaluation measures something consequential and helps the company learn. We compare systems with strong alternatives, protect held-out data, quantify uncertainty, and separate exploratory findings from evidence used to support a claim. When ground truth is imperfect, the quality and limits of the outcome data are part of the research problem rather than a footnote. Evaluation should accelerate research by giving teams clear, diagnostic signals about what improved, what did not, and why. It should also make Aaru more trustworthy by exposing failures early and keeping product and external claims aligned with the available evidence. Management in this function requires independence without isolation. The team must understand the systems deeply enough to measure them well, collaborate closely enough to make the results useful, and remain willing to conclude that an attractive idea did not work. ## You might thrive in this role if - You have led evaluation, measurement, or empirical research in machine learning, behavioral science, computational social science, statistics, economics, psychometrics, or a comparably rigorous environment. - You have built evaluations that changed a research direction, model capability, product decision, or scientific conclusion. - You can define a difficult construct precisely enough to measure it without reducing away the underlying question. - You are comfortable with experimental design, observational data, sampling, statistical power, uncertainty, causal threats, leakage, and condition shift. - You can write code, analyze large datasets, design studies, inspect individual failures, and review the technical work of researchers and engineers. - You can work closely with builders while reaching independent conclusions about the quality of their systems. - You care more about an accurate result than a favorable one and are willing to revise your own evaluation when evidence shows it is inadequate. - You can prioritize a research portfolio and choose which uncertainty is most important to resolve next. - You have managed or technically led strong researchers, give clear feedback, and can develop independent judgment rather than creating dependence on your review. - You can explain technical evidence clearly to researchers, engineers, product teams, customers, company leadership, and non-specialists. - You want to work in person in New York with a team that moves quickly and takes truth-seeking seriously. ## Strong candidates may also have - Work in ML evaluation, model behavior, forecasting, econometrics, psychometrics, causal inference, experimental economics, survey methodology, or measurement theory. - Experience evaluating LLM agents, multi-agent systems, synthetic populations, recommender systems, probabilistic models, simulations, or decision-support tools. - Experience with longitudinal records, transaction data, product analytics, field experiments, prospective studies, backtesting, or validation against operational outcomes. - Experience building evaluation platforms, regression suites, experiment-tracking systems, shared research datasets, model scorecards, or scientific reporting tools. - A record of finding an important failure that standard metrics missed and developing a better measurement method. - Experience communicating scientific results in customer-facing, public, policy, legal, or regulatory settings. - Experience hiring and leading a small, high-talent research team through ambiguous work with short feedback cycles. ## Candidates need not have - A career spent exclusively in AI evaluation or prior experience with Aaru's exact domain. - A PhD, provided you have equivalent evidence of rigorous empirical work and research leadership. - Managed managers or a large organization; this role is about leading a focused team and remaining directly involved in the research. ## What success looks like - Aaru has clear, widely trusted definitions of quality across populations, predictions, and end-to-end simulations. - Core evaluations are grounded in observed outcomes and reveal whether performance generalizes across time, domains, populations, and subgroups. - Researchers receive fast, diagnostic measurements while protected holdouts preserve the integrity of final results. - Important failures are found early, explained clearly, and converted into durable datasets, tests, or research questions. - Reusable evaluation rails reduce duplicated work and make results comparable across projects and versions. - Product and company decisions rely on evidence that is reproducible, appropriately uncertain, and connected to real-world value. - Customers and external audiences can understand what Aaru has demonstrated, where the evidence is limited, and how confidence should be interpreted. - The team develops a reputation inside and outside Aaru for evaluation work that is scientifically rigorous, practically useful, and unusually honest. - Evaluation Researchers grow into independent owners of important measurement areas, and the team maintains a high bar as it expands. ## Location and benefits This role is based in New York City. Aaru is an in-person company, working five days a week in the office. Candidates should be located in the New York metropolitan area or open to relocation. Aaru offers a competitive base salary, equity participation, comprehensive medical, vision, and dental coverage, visa sponsorship and relocation support, and other benefits and perks. Final compensation depends on level and experience and is set within Aaru's internal bands.