ML Annotation QA Engineer

Job description

**This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a ML Annotation QA Engineer based in India.** As an ML Annotation QA Engineer, you will own the quality and reliability of annotated data supporting computer vision and machine learning systems. You will focus on the judgment-intensive analysis that requires strong technical reasoning and cannot be reliably outsourced. The role combines data analysis, quality assurance, root cause investigation, process design, and cross-functional collaboration with ML and engineering teams. You will build performance tracking systems that reveal patterns across facilities, equipment, data formats, and time. Your findings will help distinguish annotation errors, model regressions, tooling issues, and genuine degradation in real-world environments. Starting with warehouse forklift vision and barcode localization, you will help establish repeatable quality practices that can scale across new AI and robotics programs. ### Accountabilities: - Own the quality analysis of annotated ML and computer vision data that requires in-house judgment, managing daily review queues and producing clear verdicts and root-cause assessments within agreed quality and time thresholds. - Define, maintain, and continuously refine verdict taxonomies, decision rules, quality guidelines, and annotation standards for the data categories under your ownership. - Build and maintain performance trackers that measure error rates across facilities, sites, equipment, and data formats over time against agreed baselines and quality thresholds. - Detect anomalies against established baselines and escalate emerging issues quickly, ensuring problems are investigated while the underlying behavior remains observable. - Conduct root cause analysis on flagged anomalies, distinguishing between annotation errors, model or system failures, tooling issues, and genuine degradation in field performance. - Communicate findings to ML, QA, and engineering stakeholders using reproducible evidence, clearly documented reasoning, and stated confidence levels. - Identify systematic failure patterns rather than isolated errors and maintain a documented pattern library covering known failure modes and recommended responses. - Use Python and SQL, or equivalent tools, to query and analyze annotation data independently, allowing you to investigate hypotheses without relying on manually prepared data extracts. - Translate quality findings into concrete improvements to annotation SOPs, instructions, decision rules, and vendor guidance when labeling quality is identified as the root cause. - Identify opportunities to improve annotation tools and specify functionality that would reduce manual analysis effort, while validating fixes after implementation. - Establish quality analysis, reporting, and documentation processes for new annotation programs as additional computer vision and machine learning capabilities are introduced. - Track work through Jira and contribute to pre-release validation for the behaviors, systems, and workflows within your area of responsibility. - During the first 90 days, take ownership of barcode and location root-cause analysis, become fluent in the annotation pipeline and warehouse data domain, establish a facility performance tracker with agreed baselines and thresholds, and assume primary ownership of the relevant verdict taxonomy and failure-pattern documentation. ### Requirements - Hold a BS degree in Computer Science, Engineering, Electrical Engineering, or a related discipline, or demonstrate equivalent professional experience. - Bring 2–5 years of experience in ML QA, annotation quality, data quality, analytics, or a closely related function within an AI/ML environment. - Have hands-on experience working with annotated machine learning or computer vision datasets, including assessing label quality and identifying inconsistencies or anomalies. - Demonstrate strong statistical and analytical capabilities, with the ability to distinguish meaningful patterns from noise and communicate the limitations of available data. - Have proven root cause analysis skills, including the ability to develop competing hypotheses and identify the evidence required to determine which explanation is best supported. - Be comfortable using Python and SQL, or equivalent query languages, to investigate unfamiliar datasets and independently reach defensible conclusions without relying on pre-prepared extracts. - Have experience creating quality guidelines, decision rules, labeling taxonomies, SOPs, or similar structured documentation used by technical or operational teams. - Demonstrate excellent written communication skills, as the role requires producing clear reports and evidence that technical stakeholders can act upon. - Bring strong collaboration and stakeholder management skills, with the ability to work effectively with ML engineers, QA, engineering, and external annotation partners. - Have a strong understanding of data privacy and confidentiality requirements when handling customer operational data. - Experience with enterprise ticketing systems such as Jira, computer vision annotation and auditing, gold-set validation, inter-annotator agreement, sampling methodologies, or recurring BI/dashboard reporting is preferred. - Experience with video event labeling, bounding boxes, polygon segmentation, counting, classification, OCR, or other computer vision annotation workflows is an advantage. - Experience in warehouse automation, robotics, computer vision applications, or other physical-world AI environments is highly valued. - Strong spatial and geometric reasoning skills are beneficial, particularly for data describing physical locations, objects, racks, bins, levels, and equipment. - Be analytical, detail-oriented, persistent, resourceful, comfortable with ambiguity, and willing to present competing hypotheses rather than forcing a conclusion when the data is inconclusive. ### Benefits - **Fully remote work from India**, offering flexibility to work remotely while collaborating with a technically diverse engineering organization. - **Full-time employment** within a multidisciplinary environment spanning machine learning, computer vision, robotics, autonomy, embedded systems, cloud, and software engineering. - The opportunity to work directly with **real-world computer vision and robotics data**, with your analysis influencing systems deployed in operational environments. - Hands-on exposure to **ML annotation quality, data analytics, root cause analysis, computer vision, warehouse automation, and AI system performance**. - The opportunity to establish quality frameworks, reporting processes, decision taxonomies, and pattern libraries that can be reused across future AI programs. - Significant ownership and visibility, with individual contributions directly informing engineering decisions and improvements to production AI systems. - Close collaboration with ML engineers, QA professionals, software engineers, and other technical stakeholders in a fast-moving environment. - The opportunity to improve annotation tooling and processes while helping create measurable gains in data quality, operational efficiency, and model performance.