Full-stack Software Engineer (Product)

Job description

## **THE OPPORTUNITY** We are building [Watcher](https://watcher.apolloresearch.ai/landing/), a coding agent security product. Watcher is deployed in production and monitors billions of agent tokens per month across engineering teams at agent-building scale-ups and enterprises. We are looking for a product engineer to build and scale Watcher end to end. This role includes backend and frontend components. You will own features across the full stack, from agent hooks and ingestion pipelines to monitoring dashboards and enterprise integrations. You can expect to own features from customer conversation to production deployment. This is truly a “start-up role”: you will own big chunks of the product, make decisions fast, and ship at high volume. You will join a small team with significant ability to shape the product and tech, and you can earn more responsibility quickly. This is an individual contributor role but could lead to management responsibilities eventually, if desired. ### KEY RESPONSIBILITIES ### **Ship product end to end (~45%)** **Feature development** - Design, build, and ship Watcher features across the stack: agent-side hooks and CLI tooling, ingestion and grading pipelines, real-time monitoring UI (Watcher Live), and the organization-wide Analyzer dashboard. - Own features end to end, from ambiguous requirements to production: talk to users, scope the work, make the technical decisions, put that into production, and iterate based on usage. - Build Watcher integrations: We currently support Claude Code and Codex, but we’re planning on increasing our coverage to other coding agents, LLM gateways, and anywhere that could benefit from agent monitoring. - Build the policy and configuration layer that lets security teams set org-wide guardrails for engineers to operate safely within. **Productionize research** - Turn monitoring research (new monitors, grading strategies, backtesting results) into production-ready product features. We are "our own customer": You will use the product you build every day. ### **Infrastructure and scale (~35%)** - Design and operate backend systems that process large volumes of agent logs in real time. - Own reliability: robust error handling, graceful degradation, and observability (logging, metrics, tracing, alerting). We should catch issues before customers do. - Architect data models and storage that handle both high-throughput writes and complex analytical queries over historical trajectories. Retention policies have to fit highly sensitive customer data. - Make sure Watcher is safe to deploy: Work with our AI Security & Control Engineer on tenant isolation, encryption, and access controls. Watcher touches some of the most sensitive data our customers have. ### **Customer and integration engineering (~20%)** - Build and maintain integrations with the enterprise security stack, e.g. streaming monitoring alerts into SIEM systems and existing security operations workflows. - Support flexible deployment models: cloud-hosted, on-prem, and local backends, so sensitive agent logs never have to leave customer control. - Talk to customers, debug issues in their environments, and feed what you learn back into the roadmap. ### REPRESENTATIVE PROJECTS - **Secure a new frontier coding agent:** Take an agent we don't yet support (e.g. Cursor) from zero to fully monitored. Build the hook/integration layer, normalize its trajectory format into our data model, validate monitor accuracy on real traffic, and ship it to customers. - **Build the real-time blocking path:** Design the low-latency pipeline that lets monitors block dangerous actions (e.g. git push --force, secret exfiltration) before they execute, while keeping p95 latency low enough that developers don't feel it. Handle traffic spikes and partial failures gracefully. - **Ship an Analyzer capability end to end:** Design and build an organization-wide view (e.g. failure trends over time, cross-session pattern detection) from data model through API to UI, based on what security teams actually need. - **Deploy Watcher inside locked-down enterprises:** Build and document the self-hosted/on-prem deployment story so that enterprises with strict data requirements can run Watcher inside their own infrastructure. ### JOB REQUIREMENTS ### **Must-haves** - **4+ years building production software.** You have shipped and operated real systems with real users, and you know what it takes to keep them reliable. - **Full-stack capability.** You are strong on the backend (APIs, data pipelines, databases, cloud infrastructure) and productive on the frontend (modern web UI). You don't need to be world-class at both, but you must be able to own a feature across the whole stack. Our stack is primarily Python and TypeScript. - **High agency and ownership.** You are willing to own big chunks of the product, make decisions fast with incomplete information, and be accountable for the outcome. You don't wait to be told what to do and have an accurate sense of the roadmap for the product. - **High output volume.** You have used Claude Code, Codex, Cursor, or similar tools heavily to accelerate or have built agents or agent tooling yourself. - **Startup pace.** You are excited about a fast-moving environment, comfortable with ambiguity and changing priorities, and willing to grind when it matters. - **Product sense.** You can talk to users, translate vague requirements into concrete designs, and make good calls about what to build and what to cut. ### **Strong nice-to-haves** - **Previous work on developer tools, monitoring/observability systems, or security products.** Watcher sits at the intersection of all three. - **Real-time or large-scale data processing experience** (streaming pipelines, message queues, high-throughput log systems). - **Enterprise or on-prem deployment experience.** Shipping software into environments you don't control is its own skill. - **Experience building LLM-powered applications,** e.g. LLM-as-judge setups, evaluation pipelines, or familiarity with frameworks like Inspect. - **Early-stage startup experience.** You have been employee 1-15 somewhere and know what it feels like to build a product with no playbook. ### **Explicitly not required** - **Formal AI safety background.** We need excellent product engineers who can learn the AI safety context, not AI safety researchers who need to learn engineering. - **Management experience.** This is an IC role, at least initially. - **Deep experience in every technology we use.** We care about demonstrated ability to learn and ship, not checkbox familiarity with our exact stack. We want to emphasize that people who feel they don’t fulfill all of these characteristics but think they would be a good fit for the position, nonetheless, are strongly encouraged to apply. ### BENEFITS - **This role offers market competitive salary, equity, and competitive benefits.** - Salary: **San Francisco:** $222,000 – $290,000 **London:** £149,000 – £195,000. We will be looking to meaningfully raise salaries soon. - Our engineers effectively have an unlimited token budget. If a better result costs more compute, use it. - Flexible work hours and schedule - Unlimited vacation - Unlimited sick leave - Up to 6 months of paid parental leave - Comprehensive health, dental and vision insurance - Retirement savings with competitive employer matching (e.g. 401(k) for US employees) - Lunch, dinner, and snacks are provided for all employees on workdays - Paid work trips, including staff retreats, business trips, and relevant conferences - A yearly $1,000 (USD) professional development budget - Relocation support and visa fees (if applicable) ### LOGISTICS - **Time Allocation:** Full-time - **Location:** This is an in-person role working out of our London or San Francisco office. We offer flexible working hours and some wfh arrangements. - **Visa sponsorship:** We sponsor visas in both the UK and US. Sponsorship isn't guaranteed for every role or candidate, but if we make you an offer, we'll work with you to find the right visa route.