Software Engineer, ML Infrastructure

Job description

**About Arc Institute** Arc Institute is an independent nonprofit research organization at the interface of artificial intelligence and biology, working to accelerate scientific progress and understand the root causes of complex diseases. Founded in 2021 and based in Palo Alto, Arc partners with Stanford University, UC Berkeley, and UC San Francisco. Unlike academia, our scientists have long-term funding and industry-like resources. Unlike industry, they're free to pursue high-risk, long-term research without commercial pressures. Arc's Technology Centers and Core Investigator labs work side by side, integrating experimental and computational biology under one roof to tackle problems neither could solve alone. Our two Institute Initiatives reflect this model in action: - [**Virtual Cell Initiative**](https://arcinstitute.org/virtual-cell-initiative)**:** Building a full-stack virtual cell model to identify disease mechanisms and nominate drug targets, accelerating the path from biological insight to clinical trials. - [**Alzheimer's Disease Initiative**](https://arcinstitute.org/alzheimers-disease-initiative)**:** Mapping the genes, pathways, and environmental factors behind Alzheimer's disease to develop drug candidates that address root causes. More than 300 Arconauts work together at our Palo Alto headquarters, backed by substantial long-term philanthropic funding. **About the position** You'll join Arc's Infrastructure Engineering team and own the data and evaluation of ML infrastructure for Arc's virtual cell models. You'll start with two systems: our Model Performance Dashboard, which benchmarks those models, and the pipeline that turns single-cell data into their training sets. You'll set the technical direction for both and connect them, so every new dataset version is judged by what it does to the models. Later in your first year, the role widens into the engineering that makes the ML team's training and evaluation run fast and at scale, lab-in-the-loop infrastructure, validation frameworks for agent-written software, and Arc's rules for what research data we bring in, keep and share. The role consolidates ownership that is split across several teams today. **Why this position could be the best job you’ve ever had** - You will contribute to building a virtual cell. Your evals show which data makes Arc's models better, and the answer shapes which experiments Arc's labs run next. - You will own the platform end to end, and you work directly with the ML researchers and scientists who rely on it. **About you** - You finish what you own, and you fix what's broken without being asked. - You're curious about the biology, enough to learn single-cell perturbation data and argue about what an eval should measure. - You're steady when priorities shift. Research is high-risk, so you test ideas early and change course when the evidence says to. - You don't wait on scarce expert time. If proving a dataset's value means running the model evals yourself, you run them. **In this position you will** - Own how Arc measures the performance of its virtual cell models, starting with the backend of our Model Performance Dashboard, which you'll take over within your first two months. - Decide with the ML team which evaluation tools they need, then design and build them so researchers can evaluate and compare models themselves, and measure what each dataset version does to model performance. - Redesign the pipeline that turns single-cell data into training sets for our virtual cell models, then run it with AI agents: agents find, process and curate datasets, and automated quality checks, not manual review, decide whether a dataset version is ready for model training. - Own the lab-in-the-loop infrastructure that turns model performance results into the case for what Arc's labs should test next, working directly with the science leads who design those experiments*.* - Build or buy validation frameworks (tests, evals, QC gates, policies) that let agent-written code and apps ship without line-by-line human review. Start with our team, where you'll also own how we work with coding agents, then extend them to apps that scientists across Arc build themselves. - Own the engineering that makes the ML team's training and evaluation run fast, reliably and at scale. The ML team owns the research and the models. Once a model is ready to push, it enters the train-and-eval CI system you own, which feeds our Model Performance Dashboard. You'll also make multi-node training a supported path, track down distributed-training failures, and measure and raise throughput and GPU utilization (MFU) on clusters that Arc IT operates. **Requirements** - Bachelor’s degree with 8 years of relevant experience, or Master’s degree with 6 years of relevant experience. - You've owned a production data or ML platform that researchers depend on, end to end (pipelines, storage, versioning), and set its technical direction. - You design and build systems with AI tools and agents, and you make their output verifiable with evals, QC gates or tests before anyone relies on it. - You've learned an unfamiliar technical domain well enough to make design calls in it. - You work directly with stakeholders to find requirements, with no PM in between. - You make the people around you better through code review, pairing and writing patterns down. - You've diagnosed and fixed performance problems in Python or PyTorch systems with profilers and measurements: GPU utilization, memory, data loading. - Work onsite in our Palo Alto office at least 3 days a week. **Preferred Qualifications** - You've worked with single-cell or perturbation data (AnnData/h5ad). - You've built or run model evaluations: benchmarks, eval harnesses, PyTorch/GPU inference. - You've run multi-node distributed training (DDP/FSDP, NCCL, SLURM) or raised a training job's throughput. This is a plus, not a requirement: we care more that you learn it fast with agents and show the numbers moved. - You've led an effort with one or two other engineers. The base salary range for this position is $191,000 to $237,000. These amounts reflect the range of base salary that the Institute reasonably would expect to pay a new hire or internal candidate for this position. The actual base compensation paid to any individual for this position may vary depending on factors such as experience, market conditions, education/training, skill level, and whether the compensation is internally equitable, and does not include bonuses, commissions, differential pay, other forms of compensation, or benefits. This position is also eligible to receive an annual discretionary bonus, with the amount dependent on individual and institute performance factors. In addition to the Base Salary and Annual Bonus above, you will be eligible to participate in the Institute’s Compound Cash Grant Program (the “CCGP”). The CCGP is a program designed to offer multi-year cash grants providing for a fixed dollar amount paid out in semi-annual installments to reward employees for staying and contributing to the Institute. As soon as reasonably practical after your start date, you will receive a grant letter describing the terms of your award under the CCGP.