Senior Machine Learning Engineer, Jockey Core

Job description

WHO WE ARE Video is 90% of the world's data. Most of it is invisible to machines. TwelveLabs builds the intelligence layer to change that. Our multimodal AI models understand video the way humans do — across sight, sound, and motion — and power production-scale AI workloads across media, entertainment, sports, security, and government. We have raised more than $210 million from NEA, Radical Ventures, Amazon, NVIDIA, Snowflake, Databricks, Index Ventures, NAVER Ventures, Korea Investment Partners, Quadrille Capital, Red Bull Ventures, and AI pioneers including Fei-Fei Li, Silvio Savarese, and Alexandr Wang. We are a global company, headquartered in San Francisco with offices in Seoul, New York, and London, and employees around the world. We believe the differences in our cultural, educational, and life experiences make our products stronger. Building technology that understands the world in all its complexity requires people who see it from every angle. We are looking for individuals who are driven by hard problems and want their work to matter. Come build it with us! ABOUT JOCKEY Jockey is TwelveLabs' unified agentic system that reasons across your videos and images. It combines a reasoning model with a memory layer that builds a knowledge store from your corpus. No context window holds a video archive. We work at a million hours of video. A single model forward pass can tell you about one file; it can't reason across a corpus, and no context window closes that gap. Jockey decomposes a query, retrieves, segments, and reasons across thousands of videos and images. Point it at an archive, ask for a highlight reel or the best viral moments, and it returns timestamped cuts you can use. Corpus-level understanding you can act on is the whole product. Built for agents, not just people. As AI agents increasingly become the primary consumers of video, we're building production-grade infrastructure that scales to millions of hours while delivering reliable, high-quality results for both human users and autonomous agents. We build on models we own. Marengo, our embedding model, resolves a query like "the moment we almost missed the flight" into real retrieval. Pegasus, our video-language model, returns structured, timestamped moments on a schema you define. We ship and improve both continuously, so Jockey's quality compounds with every release — no re-integration for customers. Few teams get to build an agent on a stack they control end to end. Deep expertise, one system, open culture. Foundation models, knowledge construction, search, and the agent harness all live in one org. Each team owns its domain and is expected to have deep expertise in it — but like a Formula 1 team, we optimize for the global system, not local parts. A model gain that doesn't expand what the agent can do isn't a gain. We trace a single algorithm change through to end-system behavior, and share work in progress weekly, not just finished results. Anyone can pull the context they need from any team. ABOUT THE TEAM The Cognition Models team owns the models that turn video into structured understanding and reasoning: Pegasus, our video-language model, and Jockey Core, the reasoning LLM behind Jockey. In the model stack we sit between Perception Models (embeddings and retrieval) and the agent system — taking what's retrieved and producing structured understanding and the reasoning to act on it. We focus on multimodal systems with high instruction-following capability and complex, hierarchically structured outputs. Our work spans training infrastructure from pre-training to RL, temporal segmentation and structured metadata extraction, large-scale inference and serving systems, data-curation and evaluation pipelines, and building Jockey Core. We ship products with real-world value rather than doing research in isolation, working as a goal-oriented, cross-functional team of ML researchers and engineers — using the most advanced compute in the world, including NVIDIA B300s, to accelerate the research-to-production cycle. ABOUT JOCKEY CORE Jockey Core is the reasoning LLM at the center of Jockey — the model that decomposes a query, decides what to retrieve and segment, and reasons over the results into an answer you can act on. It sits in the critical path of every agent step, so its quality, latency, and cost directly shape what Jockey can do. Jockey Core is a model we own and serve end to end, and we improve it continuously so Jockey's quality compounds with every release. IN THIS ROLE, YOU WILL This role leads serving engineering for Jockey Core — from engine selection through production scale-out on Blackwell. - Build benchmarks and load tests that replay the agent's real traffic (short prompts, frequent round-trips, long tool-output contexts), measuring TTFT and inter-token latency separately. - Apply inference optimization — quantization, batching/scheduling, disaggregated prefill/decode, speculative decoding — to hit cost and latency targets. - Build cost models and drive production hardening — autoscaling, capacity planning, observability, failover — through rollback-safe rollouts. - Collaborate so model-efficiency gains land as real serving wins, and set the serving technical bar through design review. - Explore and adopt AI-assisted development tools (Claude, Gemini, GPT) to improve productivity. YOU MAY BE A GOOD FIT IF YOU HAVE - Significant experience serving and optimizing large-scale LLM inference in production (vLLM, TensorRT-LLM, SGLang, or similar), across techniques like batching/scheduling, quantization, disaggregated prefill/decode, and speculative decoding. - Experience designing and operating large-scale distributed systems in high-performance GPU environments. - A track record of driving ambiguous technical decisions with measured latency/throughput/cost data. - Experience building observability, SLOs, and failure-response for production services, with strong communication skills. PREFERRED QUALIFICATIONS - Experience contributing to or customizing the internals of an LLM inference server (vLLM, TensorRT-LLM, SGLang, or similar). - Understanding of the serving characteristics of compressed (pruned/quantized) models or reasoning/agentic LLMs. - Experience designing multi-region/multi-cluster serving infrastructure or large-scale GPU capacity planning. - A Master's/PhD in Machine Learning, Computer Science, or a related technical field. READ MORE ABOUT THE TEAM - 영상에 진심인 곳은 전 세계에 몇 군데 없어요 https://www.twelvelabs.io/ko/blog/kiankim-pegasus-research-interview - 아무리 뛰어난 모델도, 안 쓰이면 ‘신기하다’에서 끝이에요 https://www.twelvelabs.io/ko/blog/sjkim-mle-interview - Pegasus 1.5를 만든 사람들 https://www.twelvelabs.io/ko/blog/pegasus-1-5-seoul-builders - 비디오를 구조화된 자산으로: Time-Based Metadata(TBM) 파이프라인 구축기 https://www.twelvelabs.io/ko/blog/%EB%B9%84%EB%94%94%EC%98%A4%EB%A5%BC-%EA%B5%AC%EC%A1%B0%ED%99%94%EB%90%9C-%EC%9E%90%EC%82%B0%EC%9C%BC%EB%A1%9C-time-based-metadata(tbm)-%ED%8C%8C%EC%9D%B4%ED%94%84%EB%9D%BC%EC%9D%B8-%EA%B5%AC%EC%B6%95%EA%B8%B0 - Cutting Edge Isn’t Plug-and-Play: B300에서 FlashAttention-4 커스터마이징하기 https://www.twelvelabs.io/ko/blog/cutting-edge-isn%E2%80%99t-plug-and-play BENEFITS AND PERKS Growth & Tools - 글로벌 B2B 고객과 함께 성장하는 Global Team - 자율성과 협업을 모두 갖춘 하이브리드 근무 - 최신 맥북 및 70만 원 상당 재택근무 장비 지원, 3년 주기로 최신 장비 교체 - Tokens never sleep - Tech 직군 LLM 토큰 무제한 지원 - 강의, 컨퍼런스, 멤버십 등에 사용 가능한 연 140만원 상당 자기개발비 지원 - 영어 교육 프로그램 및 글로벌 버디 프로그램 운영 - 야간 및 주말 출퇴근 택시비 지원 Meal & Snack - 식비·교통비 등 자유롭게 사용할 수 있는 연 720만원 상당 법인카드 제공 - 사무실 내 스낵바 운영 (간식, 커피, 제철 과일 등) - 사무실 근무 시, 오후 7시 이후 저녁 식대 제공 Wellness & Family - 연 1회 본인 및 가족 1인의 건강검진 제공 - 단체보험 가입 (상해보험/치아보험/가족 상해보험 중 택 1) - 독감 예방접종비 지원 - 연말 2주간 유급 Holiday Break 운영