Technical Product Manager, Applied Science – AI Products

Job description

Caseware is one of Canada's original Fintech companies, having led the global audit and accounting software industry for over 30 years, with more than 500,000 users across 130 countries and available in 16 different languages. While you might not have heard of us (yet) over 36,000 accounting and audit professionals list Caseware as a skill on their LinkedIn profiles! We're looking for a Technical Product Manager to own how Caseware builds, evaluates, and raises the quality of AI across the platform. You are the product owner of our core AI surfaces: the agents and agent builder practitioners use to do real work, the eval builder and eval roadmap that let us measure quality, and the agentic memory that lets the platform improve as it is used. You decide what we build, in what order, and whether it clears the bar before it reaches someone's audit file. You are also the closest thing the product has to a practitioner in the room. Acting as an SME proxy, you carry the voice of the auditor into every decision: shadowing real workflows, validating outputs against what a qualified practitioner would actually sign off on, and turning "this doesn't feel right" into specific, actionable failure modes. You partner closely with applied scientists, engineering, domain experts, and external partner firms. You own the product outcome and the quality bar; the applied science team owns the science underneath it. This role demands high agency. You'll operate with real autonomy, make decisions without waiting for permission, and own the outcome — success measured by results, not activity. You're intellectually curious, collaborate easily across teams, and take real pride in getting it right. This work sits in the financial audit and assurance domain. Independent audit is one of the quiet foundations of the global economy — when investors, lenders, and regulators can trust that financial statements are accurate, capital flows and markets function. That trust rests entirely on the quality of the work, which is why the bar for AI here is so high. You don't need an accounting background to start, though one — or any real connection to the profession — adds high value. What you do need is genuine curiosity about the complexity of auditing and a deep appreciation for why quality is non-negotiable. You'll build that fluency on the job alongside experienced domain experts, and it will sharpen your judgment on what "good" really means. ❗This a full-time, permanent position ❗This a new vacancy ❗This role is hybrid. You will be required to work from our Toronto office 3-days a week, located at 351 King Street East, Toronto, ON ### What you will be doing: - Own agents, the agent builder, the eval builder, and agentic memory as products: their roadmap, sequencing, and the business outcomes they drive, using results and practitioner feedback to shape what ships. - Own the applied-science delivery roadmap and cross-team orchestration for the Caseware AI platform — sequence the evaluation, agentic-memory, and platform workstreams into a single plan, and keep delivery moving. - Own the product acceptance bar for every AI feature — decide when an output is good enough to put in front of practitioners, and set the eval baseline so every agent and skill is measured against a stated bar. - Embed with domain experts and partner accounting and audit firms to validate AI outputs against professional standards — not just user preferences, but what a qualified practitioner would actually sign off on. - Be the voice of the practitioner as an SME proxy — shadow real audit workflows, test prototypes in context, and translate "this doesn't feel right" into specific, actionable failure modes the team can act on. - Work daily with engineering to turn precise problem statements into well-scoped requirements, with acceptance criteria agreed before development begins. - Stay ahead of the AI landscape — monitor model developments, new architectures, emerging agent patterns, and competitor moves, and bring a clear product point of view on what matters for Caseware. - Define and own product health metrics for AI features: task completion, correction rates, override frequency, trust signals, and time-to-completion — and use them to drive product decisions. ### What you will bring: - 6+ years of experience in technology-focused product management or product development, including 3–4 years working deeply with AI or machine learning–driven products. Software engineers transitioning into PM are welcome. - Proven ability to drive delivery across multiple technical teams coordinating dependencies, shared interfaces, and timelines across applied-science, and platform-engineering pod. - Direct experience building and running eval frameworks for production AI products: defining acceptance bars, running human-in-the-loop, practitioner validation, regression tracking across releases, and using results to decide what ships. - Technical fluency in LLMs, RAG, prompt engineering, embeddings, and agentic patterns, enough to drive substantive conversations with engineers and know when a product problem is a model problem. - High agency: you identify what needs to happen, move without waiting to be told, and take ownership of outcomes rather than just activities. - Intellectual curiosity about AI — you follow model releases, read research, understand what frontier labs are shipping, and form your own views on what matters. - Experience being the voice of the customer or practitioner: working with domain experts or subject matter specialists to define quality bars in high-stakes, regulated, or professional-grade contexts. - Comfortable with ambiguity and energized by it. Many of the problems this role works on do not have clear playbooks, when there is no established benchmark for the work, you can go and find out how it should be measured.