Senior AI Engineer (Evaluations - Canvas Agent)

Job description

At Instructure, we believe in the power of people to grow and succeed throughout their lives. Our goal is to amplify that power by creating intuitive products that simplify learning and personal development, facilitate meaningful relationships, and inspire people to go further in their education and careers. We do this by giving smart, creative, passionate people opportunities to create awesome. And that's where you come in: We are looking for an AI Engineer to join the team behind the Canvas Agent. As we build out agentic workflows that can "act" upon Canvas and assist educators and students, ensuring the quality, safety, and accuracy of these interactions is our top priority. In this role, you will design and implement robust LLM-as-a-judge evaluation pipelines that automatically assess the Canvas Agent’s multi-step reasoning, tool usage, and conversational helpfulness. What You’ll Do - Design the Evaluation Framework: Build and maintain scalable LLM-as-a-judge pipelines to automatically score the Canvas Agent’s actions, responses, and tool usage across a variety of complex educational workflows. - Develop Rubrics & Datasets: Create comprehensive grading rubrics and curate high-quality "golden" datasets (both real and synthetically generated) to baseline and test the agent's performance. - Optimize Judge Prompts: Engineer and iterate on prompts for the judges, ensuring automated scoring aligns with high quality evaluations. - Full-Stack Contribution: Step beyond evaluation pipelines to participate in full-stack product development as needed, collaborating with the team to build and refine the core AI features, UI components, and application architecture. - Accelerate Iteration: Integrate your automated evaluations directly into our CI/CD pipelines, creating a "paved path" that allows our AI product teams to ship updates with high velocity and total confidence. - Analyze & Report: Monitor evaluation metrics to identify failure modes, hallucination rates, and regressions. Translate these subjective quality signals into objective, actionable engineering tasks. What You’ll Need - LLM Evaluation Experience: A background that clearly demonstrates hands-on experience designing and writing automated tests for LLMs. You must have a proven track record of writing and implementing LLM-as-a-judge evaluations in real-world scenarios. - Technical Stack: A strong background in Python is highly preferred, or a demonstrated willingness and ability to learn it quickly. You should also have professional experience in full-stack or backend engineering to support both the evaluation infrastructure and general product development needs (experience with TypeScript/Node.js is a plus). - Prompt Engineering Expertise: Deep understanding of how to reliably prompt models for classification, extraction, and grading tasks without falling prey to common biases (e.g., position bias, verbosity bias). - Evaluation Tooling: Familiarity with modern LLM observability and evaluation frameworks (e.g., LangSmith, Braintrust, Ragas, promptfoo, or similar). - Agentic Architectures: Strong conceptual understanding of how AI agents plan and execute tool calls (e.g., ReAct, tool-use APIs) so you can effectively evaluate multi-step workflows. - Analytical Mindset: An ability to translate highly subjective concepts (like "helpfulness" or "tone") into rigorous, trackable metrics. - Collaboration Skills: Ability to work across product and engineering teams to understand the Canvas Agent's use cases and align evaluation criteria with user needs. Onsite Collaboration Requirement: This role requires working onsite on Tuesday and Wednesday, with Thursday strongly encouraged as part of our company’s in-person collaboration model. Why Join Us? In this role, you aren't just testing a feature; you are the guardian of quality for how AI interacts with the world of education. Your work will be the catalyst that allows Instructure to move faster than ever, safely turning static tools into active participants in the learning journey. We value speed, rigorous testing, and scalability. You will have the opportunity to work at the cutting edge of AI evaluations, setting the standards for how we measure the success of complex agentic behavior. If you enjoy building the "guardrails" that make cutting-edge AI reliable enough for real-world classrooms, this is the role for you. We invest in our employees’ growth and success through thoughtful mentorship, hack weeks, internal conferences, and a culture that values ownership, experimentation, and innovation. If you’re excited by building foundational AI infrastructure that empowers others and drives real-world impact, we'd love to meet you. Get in on all the awesome at Instructure! We offer competitive, meaningful benefits in every country where we operate. While they vary by location, here's a general idea of what you can expect: - Competitive compensation, plus all full-time employees participate in our ownership program - because everyone should have a stake in our success. - Flexible work culture. Our remote, hybrid and in-office collaboration spaces vary by role, team and location. - Generous time off, including local holidays and our annual “Dim the Lights” period in late December, when teams are encouraged to step back and recharge based on departmental needs. - Comprehensive wellness programs and mental health support - Learning and development resources, including professional development tools and tuition reimbursement, to support your growth - The technology and tools you need to do your best work - Motivosity employee recognition program - A culture rooted in inclusivity, support, and meaningful connection We believe in hiring great people and treating them right. The more diverse we are, the better our ideas and outcomes. Instructure is an Equal Opportunity Employer. We comply with applicable employment and anti-discrimination laws in every country where we operate. All employees must pass a background check as part of the hiring process. To help protect our teams and systems, we’ve implemented identity verification measures. Candidates may be asked to verify their legal name, current physical location, and provide a valid contact number and residential address, in accordance with local data privacy laws. Any attempt to misrepresent personal or professional information will result in disqualification.