Applied AI/ML Engineer

Job description

# **Your Role** As a founding AI Engineer at Overtone, you will build and own the AI systems that power our core product experience, from conversational intelligence to matchmaking insight generation. You'll design prompting, evaluation, and model infrastructure that makes our AI reliable, trustworthy, and continuously improving. This is a hands-on, early-stage role: you'll run experiments, ship infrastructure, and build the tools that give the team real visibility into model behavior and quality. **What You’ll Do** - **LLM systems & prompt architecture:** Design and maintain the prompts, structured outputs, and orchestration systems that power Overtone's AI-driven features. These include both text-based LLM output, as well as transcripts generated from our code voice AI expereince. - **Evaluation frameworks:** Build evaluation infrastructure to measure AI quality at scale using tools such as Braintrust, Langfuse, TensorZero, and Promptfoo. You'll work with our team to make sure LLM judges are aligned. Evals should cover both generated text and voice transcripts, including transcription fidelity and response latency. - **Model experimentation:** Run structured experiments across models, prompts, and configurations to optimize quality, cost, and latency. - **AI observability & tooling:** Build internal tools (dashboards, admin interfaces, debugging workflows) that give the team visibility into system behavior and model performance. - **Matching & ML systems:** Apply data science and lightweight modeling to improve match quality, identifying when traditional ML models or ranking systems outperform prompt-based approaches. - **Model serving & production inference:** Own the model serving layer: deploying models, managing inference infrastructure (API providers and self-hosted), model versioning, and cost/latency optimization in production. Work with the backend lead to define clean service contracts between the AI layer and the rest of the system. - **Model development & fine-tuning:** Identify opportunities to fine-tune open-source models using techniques such as SFT, DPO, and RLHF for cost and latency improvements. - **Memory & personalization systems:** Design and maintain vector databases and memory architectures that enable personalization and context-aware experiences. - **Responsible AI systems:** Ensure systems are fair, unbiased, and auditable, building evaluation pipelines and human-in-the-loop processes where appropriate. - **Cross-functional collaboration:** Work closely with product, engineering, and research to bring AI capabilities into real user-facing features with scalability and reliability in mind. **Who You Are** You likely have: - **5+ years of engineering experience**, with meaningful time spent building **LLM-powered products in production** - Strong **prompt engineering and evaluation instincts**. You think in terms of **evals, not vibes** - Experience building **evaluation pipelines for LLM systems** - Familiarity with **vector databases and retrieval-augmented architectures** - Experience with **fine-tuning techniques** such as SFT, DPO, or RLHF - Strong understanding of **when ML models outperform prompt-based systems** - Experience working in **early-stage environments** with high ownership - Experience with Python, SQLAlchemy, FastAPI, PostgreSQL Nice to have: - Experience with **recommender systems or ranking systems** - Experience with **MLOps and model lifecycle infrastructure** - Experience building **AI evaluation frameworks or model observability systems** - Experience building **internal developer tooling** (React/TypeScript + Python) - Experience with **production voice pipelines**: streaming ASR (Deepgram, AssemblyAI, Whisper), TTS (ElevenLabs, Cartesia, PlayHT), or realtime voice agent frameworks - Experience with **latency-sensitive or streaming AI surfaces**, including WebRTC/LiveKit and evaluating voice quality (WER, naturalness, perceived responsiveness) # **Compensation & Benefits** Salary of $200k to $300k. This position will receive meaningful stock options with considerable upside potential. We provide standard benefits such as health insurance, dental insurance, etc. # **Our Core Qualities** *We look for these qualities in everyone we hire at Overtone.* **Curiosity:** You seek to understand before you decide. You ask thoughtful questions, test assumptions, and stay open to changing your mind as new information emerges. **Craft:** You care deeply about quality and coherence. You simplify complexity, exercise good judgment, and take pride in work that feels intentional, thoughtful, and well-made. **Collaboration:** You engage with others honestly and generatively. You challenge ideas with respect, listen deeply, and help synthesize perspectives into clearer, better outcomes. **Care:** You take responsibility for people and outcomes. You act with integrity, follow through on commitments, and protect trust—with users, teammates, and the work itself. # **About Overtone** Overtone is a modern matchmaking company, from the creator of Hinge. We’re building a voice- and audio-forward service, enabled by AI, grounded in relationship science. There’s no profiles or swiping. We get to know people in their own voice, then make thoughtful introductions, one at a time, to people we think are worth meeting. The service is in beta-testing and not yet open to the public. We’re building a small team of people passionate about relationships and human connection, to work together in person in our NYC office (5 days a week in Flatiron).