AI Evaluation Engineer

Job description

**Join the Market Leader in Electric Power Data and Analytics Solutions** The electrical grid is the largest and most complicated machine ever built. Yes Energy’s industry-leading electric power trading analytics software provides real-time visibility into the massive amount of data generated by the North American electrical grid daily. Our unique and innovative view of the data informs real-time trading decisions and mid-to-long-term investment decisions that keep utility prices low, support the energy transition, and keep the grid running. It’s both challenging work and work with a purpose. **Be a part of our successful, growing business during international transformation.** **Position Summary** In the Engineering group, we are accelerating the product vision by building solutions in a fun, collaborative, efficient, and adaptive team. As an AI Evaluation Engineer, you will own the quality bar for Yes AI — defining what "correct" looks like when an AI answers a complex power market question, and building the frameworks to measure it at scale. This is a foundational role at the intersection of LLM engineering, domain expertise, and rigorous testing. You will also work directly on projects to integrate systems and serve energy data to customers. We are curious people who take pride in our work, value high-quality communication, and strive to improve continuously. **Position Details** - Salary range: Net 16.000 - 19.000 RON/month - Location: Bucharest, Romania - Full-time - Hybrid - Will be required to work in the office 2-3 days a week - Reporting to: Engineering Manager **Primary Responsibilities** - Own the evaluation strategy for Yes AI end-to-end — design, build, and continuously improve frameworks that measure the quality, accuracy, and reliability of LLM-powered outputs across natural language answers, data retrieval, and semantic context responses - Develop benchmarks and automated test suites to detect hallucination, factual drift, and retrieval failures; set the quality bar for what "correct" means in power market AI - Partner directly with domain experts and senior engineers to translate deep power market knowledge into rigorous, repeatable evaluation criteria - Build and maintain scalable eval pipelines that run continuously against prompt changes, model updates, and Snowflake data changes — surfacing regressions before they reach customers - Lead prompt engineering and RAG experimentation efforts, using evaluation results to drive measurable improvements in answer quality - Influence engineering and product decisions by making AI quality visible and quantifiable — advocate for reliability as a first-class product requirement - Help define the discipline of AI evaluation at Yes Energy; this is a foundational role with room to build process and tooling from scratch **Minimum Qualifications** - Bachelor's degree in Computer Science, Data Science, Mathematics, or related field - 5+ years of software engineering experience, with significant focus on testing, evaluation, quality systems, or AI/ML tooling - Strong Python skills; experience building production-grade evaluation or testing pipelines - Solid understanding of LLM evaluation concepts: benchmarking, hallucination detection, retrieval-augmented generation quality, and semantic similarity - Experience designing and owning quality frameworks in ambiguous, fast-moving environments - Ability to work closely with non-engineer subject matter experts to define and codify what "good" looks like - Excellent judgment — knows when an eval result signals a real problem vs. noise - Strong communicator; able to present quality findings clearly to technical and non-technical audiences ## **Key Competencies / Preferred Qualifications** - **Technical Execution & Engineering Tech**: Hands-on experience with eval frameworks (RAGAS, LangSmith, PromptFlow) and proficiency in building production data pipelines using RAG or vector databases - **Problem Solving**: Applies structured thinking and sound judgment to determine when an eval result signals a real problem versus noise in complex data sets - **Curious**: Demonstrates an innate curiosity and a strong desire to learn the energy industry or apply experience from equities/commodities trading environments - **Stakeholder Focus**: Rooted in listening to technical and non-technical business partners to define and translate deep operational requirements into codified test criteria - **Data Management & Quality**: Background in data-intensive or highly regulated environments where answer accuracy is high-stakes and strict data governance is required - **Effective Communication**: Strong written and verbal communication skills, adjusting delivery style seamlessly based on the technical depth of the audience **At Yes Energy, we value connecting directly with candidates. We kindly ask that third-party recruiters and agencies not submit resumes, as we are not open to external recruiting partnerships.** **ABOUT YES ENERGY** **Overview** Yes Energy delivers real-time market data and electric power trading decision solutions. Over 1,000 market participants use Yes Energy solutions daily. The business is a leader in all aspects of information content collection and management, developing and delivering data and market analytics solutions. Since its inception in 2008, Yes Energy has become a trusted and respected supplier of innovative and reliable solutions focused on the needs of power market analysts, traders, and trade managers. Yes Energy has a team of 400 amazing professionals in Boulder, CO (HQ); Boston, MA; Chicago, IL; Glendora, CA; Richmond, VA; London, United Kingdom; Auckland, New Zealand; and Bucharest, Romania. **Culture** Yes Energy has been named one of the Best Places to Work in Colorado, and we have the culture to prove it. At Yes Energy, we care about saying “Yes” to customers. We like to listen, learn, and develop our solutions in line with their needs. We think about customers as business partners, and when we help them be more successful … we are more successful, too. Around the office, our culture is driven by some pretty fundamental values that we’re proud of: - We love innovation and solving tough challenges; - We are “high standards people” who combine passion and pride with hard work and rewards of all kinds-- in an ethic that is consistent across the company. - We’re team-focused with a flat hierarchy-- we work in small teams on well-defined projects that directly impact the success of the business; - We play to the strengths and experience of each person while each of us also works along a continuum of roles adjacent to our focus area. This presents the challenge of maintaining a broad set of skills as well as an opportunity to learn and contribute in many ways. - We are constantly growing. Professional development happens every day and every year. **Compensation and Benefits** We offer highly competitive salaries and real bonuses that are achievable and that you can impact. Our benefits package includes private medical insurance, wellness/gym benefits, flexible vacation, and flexible work schedules. Yes Energy encourages and funds investment in both formal and informal professional development. At Yes Energy, we are dedicated to building a diverse, inclusive, and authentic workplace. If you’re excited about this role but your experience doesn’t perfectly align with every qualification in the job description, we encourage you to apply anyway. You may be just the right candidate for this or other roles. *Yes Energy provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics. In addition to federal law requirements, Yes Energy complies with applicable state and local laws governing nondiscrimination in employment in every location where the company has facilities. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.*