Staff ML Research Engineer, Multimodal Structure & Marengo

Job description

## **Who we are** Video is 90% of the world's data. Most of it is invisible to machines. TwelveLabs builds the intelligence layer to change that. Our multimodal AI models understand video the way humans do — across sight, sound, and motion — and power production-scale AI workloads across media, entertainment, sports, security, and government. We have raised more than $210 million from NEA, Radical Ventures, Amazon, NVIDIA, Snowflake, Databricks, Index Ventures, NAVER Ventures, Korea Investment Partners, Quadrille Capital, Red Bull Ventures, and AI pioneers including Fei-Fei Li, Silvio Savarese, and Alexandr Wang. We are a global company, headquartered in San Francisco with offices in Seoul, New York, and London, and employees around the world. We believe the differences in our cultural, educational, and life experiences make our products stronger. Building technology that understands the world in all its complexity requires people who see it from every angle. We are looking for individuals who are driven by hard problems and want their work to matter. Come build it with us! ## **About Jockey** Jockey is TwelveLabs' unified agentic system that reasons across your videos and images. It combines a reasoning model with a memory layer that builds a knowledge store from your corpus. **No context window holds a video archive. We work at a million hours of video.** A single model forward pass can tell you about one file; it can't reason across a corpus, and no context window closes that gap. Jockey decomposes a query, retrieves, segments, and reasons across thousands of videos and images. Point it at an archive, ask for a highlight reel or the best viral moments, and it returns timestamped cuts you can use. Corpus-level understanding you can act on is the whole product. **Built for agents, not just people.** As AI agents increasingly become the primary consumers of video, we're building production-grade infrastructure that scales to millions of hours while delivering reliable, high-quality results for both human users and autonomous agents. **We build on models we own.** Marengo, our embedding model, resolves a query like "the moment we almost missed the flight" into real retrieval. Pegasus, our video-language model, returns structured, timestamped moments on a schema you define. We ship and improve both continuously, so Jockey's quality compounds with every release — no re-integration for customers. Few teams get to build an agent on a stack they control end to end. **Deep expertise, one system, open culture.** Foundation models, knowledge construction, search, and the agent harness all live in one org. Each team owns its domain and is expected to have deep expertise in it — but like a Formula 1 team, we optimize for the global system, not local parts. A model gain that doesn't expand what the agent can do isn't a gain. We trace a single algorithm change through to end-system behavior, and share work in progress weekly, not just finished results. Anyone can pull the context they need from any team. ## About the Team This team develops two core capabilities for multimodal understanding: structure and semantics. Structure organizes video, audio, text, and documents into addressable units and models the relationships among them. Marengo, TwelveLabs' multimodal embedding model, represents what those units mean in a shared embedding space for understanding and retrieval. **End-to-end model development:** We work across a broad range of research areas, including video segmentation and tracking, temporal and hierarchical modeling, contrastive learning, and multimodal representation learning. The team owns the entire model development lifecycle, from building large-scale training datasets and designing model architectures to optimizing distributed training and developing robust evaluation frameworks. **Research at scale:** With access to world-class compute infrastructure, including NVIDIA B300 GPUs, we rapidly iterate on large-scale experiments, enabling fast progress on ambitious research problems. **Research with real-world impact:** The path from research to production is exceptionally short. We work closely with the Agent, Search, Product, and Infrastructure teams to continuously improve the models that power multimodal search and understanding for thousands of customers worldwide. ## **About the Role** As a Staff ML Research Engineer working across multimodal structure and embeddings, you will set the technical direction for TwelveLabs' next-generation models and own the end-to-end development process, from research strategy and data architecture to training systems, production model APIs, and evaluation frameworks. This is a high-autonomy role at the intersection of video understanding, multimodal representation learning, large-scale systems design, and cross-team technical leadership. We're looking for someone who thrives in ambiguity: someone who can identify the highest-impact research problems, define the technical approach, and drive cross-team execution to deliver models that serve customers worldwide. ## **In this role, you will** - Set the technical direction for multimodal structure, including how assets are organized into reusable, addressable units and how those units relate. - Define the architecture, training, and data strategy for next-generation multimodal embedding models. - Own end-to-end model development from research planning through large-scale distributed training to production evaluation. - Architect and optimize large-scale training infrastructure, including distributed training pipelines, data processing systems, experiment workflows, and GPU utilization. - Own production model APIs end to end, from model packaging and API design to inference optimization and reliable operation at scale. - Drive data strategy by building large-scale curation, filtering, and quality systems for both structure and embeddings. - Define evaluation methods and quality standards for structure, embeddings, and end-to-end multimodal understanding and retrieval. - Define the interface between structure and semantics, ensuring that structured units remain reusable and addressable while improving end-to-end understanding and retrieval. - Drive cross-functional alignment with Agent, Search, Product, and Infrastructure teams on model integration and performance requirements. - Raise the research engineering bar through design review, experiment review, and technical mentorship. Even if you don't check every box, we encourage you to apply. If you're a zero-to-one achiever, a ferocious learner, and a kind team player who motivates others, you'll find a home at TwelveLabs. ## **You may be a good fit if you have** - 7+ years of industry experience in computer vision, video understanding, or multimodal learning. - Demonstrated ability to take ambiguous, loosely-defined research problems and drive them to concrete, impactful solutions, from problem identification through delivery. - Strong judgment under changing constraints: you adapt as requirements evolve, make principled tradeoffs, and deliver the strongest result within the available time and resources. - Deep expertise in large-scale distributed model training (kernel optimization, FSDP, or similar). - Experience building and operating production model APIs for large-scale ML systems. - Deep expertise in video understanding, multimodal representation learning, or foundation model development. - Experience building end-to-end systems that connect multimodal structure with embeddings and retrieval. - Proven end-to-end ownership: not just running experiments, but defining what to build, building it, deploying it, and iterating on it in production. - Strong proficiency in Python and PyTorch. - Evidence of both research depth and engineering impact: publications paired with shipped products, not one or the other. We evaluate based on relevant technical skills and sustained industry impact. This role is typically a strong fit for engineers with an MS and deep industry experience who have evolved from individual contributor to technical leader in production ML environments. ## **Preferred Qualifications** - Experience training models at billion-parameter scale. - Experience with training operations: pipeline reliability, monitoring, fault tolerance, and cost optimization. - Experience with large-scale data curation and data quality systems. - Experience modeling temporal, spatial, or hierarchical structure across video and other multimodal content. - Experience with temporal video understanding or multimodal video modeling. - Deep experience optimizing training and inference systems for throughput, latency, GPU efficiency, and scale. - Track record of technical leadership: driving architectural decisions that shaped team or product direction. ## **What makes this role unique** The gap between research and production is remarkably short here. Models you build will be used by thousands of companies worldwide within months. In this role, you will shape how multimodal content is organized into reusable units and how those units are represented for understanding and retrieval. Rather than optimizing structure, embeddings, and retrieval in isolation, you will connect them into one end-to-end system. Our research philosophy balances rigorous experimentation with real-world application: we aim to build multimodal systems that are powerful, trustworthy, and genuinely useful. ## **Read more about the team** - [의미의 경계를 찾아서: 영상을 이해하는 임베딩을 만드는 사람](https://www.twelvelabs.io/ko/blog/dankim-marengo-research-interview) - [벤치마크에 없는 문제를 풀고 있습니다](https://www.twelvelabs.io/ko/blog/we-re-solving-problems-that-aren-t-in-any-benchmark_kr) ## **Benefits and Perks** - Growth & Tools - 글로벌 B2B 고객과 함께 성장하는 Global Team - 자율성과 협업을 모두 갖춘 하이브리드 근무 - 최신 맥북 및 70만 원 상당 재택근무 장비 지원, 3년 주기로 최신 장비 교체 - Tech 직군이 AI를 효율적이고 책임 있게 활용할 수 있도록 LLM 토큰을 무제한 지원 - 강의, 컨퍼런스, 멤버십 등에 사용 가능한 연 140만원 상당 자기개발비 지원 - 영어 교육 프로그램 및 글로벌 버디 프로그램 운영 - 야간 및 주말 출퇴근 택시비 지원 - Meal & Snack - 식비·교통비 등 자유롭게 사용할 수 있는 연 720만원 상당 법인카드 제공 - 사무실 내 스낵바 운영 (간식, 커피, 제철 과일 등) - 사무실 근무 시, 오후 7시 이후 저녁 식대 제공 - Wellness & Family - 연 1회 본인 및 가족 1인의 건강검진 제공 - 단체보험 가입 (상해보험/치아보험/가족 상해보험 중 택 1) - 독감 예방접종비 지원 - 연말 2주간 유급 Holiday Break 운영 ## **Additional Notes** - 한국 전 직군 공통 자격요건: 해외 여행에 결격 사유가 없는 분 - 모든 입사자에게 3개월의 수습기간이 적용됩니다. 해당 기간 동안 급여는 100% 지급됩니다. - 채용 과정에서 제출한 서류에 허위 사실이 있거나 부정 행위가 확인될 경우, 합격이 취소되거나 향후 채용이 제한될 수 있습니다.