AI Benchmark Engineer | Native Language Specialist - French

Job description

**This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a AI Benchmark Engineer | Native Language Specialist - French based in Canada.** This opportunity is designed for an experienced software engineer with native-level French expertise who wants to contribute to the evaluation of advanced Artificial Intelligence systems. You will design rigorous Terminal-Bench tasks that test how effectively large language models handle multilingual software challenges. Your work will focus on French-language prompts, datasets, files, and locale-specific edge cases without relying on English translation. You will build realistic environments, develop reference implementations, and create reliable verification scripts to assess model performance. The role combines software engineering, language expertise, benchmark design, and quality assurance in a highly technical environment. As an independent contractor, you will have flexibility while contributing to projects that advance multilingual AI capabilities. ### Accountabilities - Design and engineer high-quality benchmark tasks that evaluate coding agents and large language models in multilingual software environments. - Build realistic task environments using datasets, files, prompts, and other assets written in French, ensuring that the target language remains central to the evaluation. - Identify failure points and challenging scenarios related to French-language processing, prompting, translation, and multilingual software workflows. - Develop robust reference implementations and highly reliable, deterministic verifier scripts, using rubric-based judging only when strictly necessary. - Analyze execution logs and calibrate task difficulty from Easy to Very Hard using standardized Terminal-Bench configurations across different model tiers. - Evaluate model behavior across multilingual software challenges, including language-specific, encoding, locale, and terminal workflow issues. - Participate in a four-layer human quality-control process covering task creation, human review, calibration review, and audit. - Work alongside automated Large Language Model (LLM)-based checks to ensure benchmark fairness, grammatical accuracy, technical reliability, and overall integrity. - Collaborate effectively while meeting agreed quality standards, deadlines, and task requirements. ### **Requirements** - At least 5 years of professional industry experience in software engineering. - Proven experience at leading technology companies and/or graduation from a highly regarded engineering university. - Native or near-native French fluency, with deep knowledge of French grammar, register, phrasing, and language conventions. - High proficiency in English for technical communication and collaboration. - Strong proficiency in Python, standard shell scripting, and data processing. - Extensive experience working with Terminal or Command-Line Interface (CLI)-based development workflows. - Working familiarity with coding agents and modern AI-assisted software development workflows. - Strong understanding of multilingual text-processing challenges, including encoding and decoding, Unicode normalization, locale-dependent casing and collation, and non-Gregorian date conventions. - Knowledge of text input/output, toolchain interoperability, safe string handling, and language-specific software behavior. - Strong analytical, problem-solving, quality-assurance, and attention-to-detail skills. - Ability to work independently, manage assigned tasks reliably, and deliver high-quality work on time. - Reliable availability and commitment are required once tasks are accepted; many assignments require at least 2 hours per day or 10 hours per week. - Must be able to work as an independent contractor and take responsibility for applicable tax obligations. - Geographic eligibility may be subject to applicable international embargoes or sanctions. ### **Benefits** - Independent contractor arrangement with flexible working schedules and no fixed daily hours. - Ability to choose when and how much to work based on available projects and personal availability. - Competitive contractor rates and prompt payments. - Opportunity to contribute to cutting-edge Artificial Intelligence and language technology projects. - Exposure to multilingual AI evaluation, benchmark engineering, coding agents, and advanced language-processing challenges. - Access to a global community of linguists, technical specialists, subject-matter experts, and language professionals. - Opportunities to work on diverse projects across industries and technical domains. - Ability to expand your professional portfolio and strengthen your technical and language expertise. - Remote work from Canada. - Training and assessment opportunities as part of the onboarding process for Applied AI projects. - Please note that project availability may fluctuate, hours are not guaranteed, and this opportunity is not intended to serve as a full-time primary income source. - Independent contractors are not provided with employee benefits such as health insurance, paid time off, or retirement contributions.