From Retrieval to Reasoning: Evaluating Agentic RAG for Engineering Reports

Job description

From Retrieval to Reasoning: Evaluating Agentic RAG for Engineering Reports Background Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) offer promising ways to help engineers navigate and synthesise large collections of technical documentation. Generating a reliable engineering report, however, is considerably more challenging than answering individual questions: relevant information may be distributed across many documents, different sources may need to be combined, and the resulting text must be accurate, complete, consistent and traceable to its evidence. At SCK CEN, an experimental AI knowledge assistant called HOMER has been developed to explore these challenges using technical documentation from large engineering programmes such as MYRRHA and LEANDREA. HOMER currently combines RAG with a multi-agent architecture , in which specialised AI agents collaborate to perform different tasks involved in technical report generation. This architecture provides a starting point for further research into how agentic systems can support complex knowledge synthesis. Thesis Objective The main objective of this thesis is to design and implement alternative multi-agent architectures for AI-assisted engineering report generation, and experimentally investigate their effectiveness . The student will continue the architectural development of HOMER by exploring different ways of decomposing, coordinating and orchestrating the tasks involved in producing a technical report. Rather than assuming that the current architecture is optimal, the thesis will investigate alternative approaches and determine which architectural patterns are most suitable for reliable and traceable technical knowledge synthesis. Approximately 60% of the thesis will focus on system design and implementation , while approximately 40% will focus on evaluation and experimental analysis . System Design and Development The core of the thesis will be the exploration of alternative agentic RAG architectures . A technical report may involve several distinct activities, such as: identifying the information required to answer a question; retrieving relevant evidence from multiple documents; analysing and synthesising retrieved information; identifying gaps or missing evidence; checking consistency between different sources; constructing and refining the report; verifying claims and their supporting evidence. The student will investigate how these activities can best be distributed among interacting AI agents. Possible approaches include: specialised agents with clearly defined roles; hierarchical or sequential agent workflows; iterative retrieval-and-reasoning loops; planner/executor architectures; critic or verification agents; different mechanisms for agent coordination and information sharing; alternative strategies for decomposing complex report-generation tasks. The existing HOMER architecture, including its current multi-agent coordination approach, will provide a baseline for comparison. The student will be encouraged to design and experiment with alternative architectures , rather than simply implementing a predefined solution. Experimental Evaluation The second major component of the thesis will establish a systematic methodology for evaluating the resulting systems. The evaluation will consider dimensions relevant to engineering knowledge synthesis, including: factual correctness; relevance and coverage of retrieved information; completeness of generated reports; consistency; traceability of claims to source material; robustness across different document types and tasks; efficiency and computational cost. Where appropriate, automated evaluation will be complemented by structured assessment of representative outputs. The evaluation framework will allow the student to compare the alternative architectures and identify their respective strengths, weaknesses and failure modes. Particular attention will be given to understanding when additional agents and more sophisticated orchestration actually provide a benefit , and when a simpler architecture may be preferable. Research Questions The precise research questions will be refined with the student, but may include: Architecture: How can complex technical report-generation tasks be effectively decomposed into interacting AI agents? Orchestration: Which multi-agent coordination strategies are most effective for combining retrieval, reasoning, synthesis and verification? Knowledge synthesis: Can agentic architectures improve the completeness, consistency and traceability of generated engineering reports? Architectural trade-offs: What are the trade-offs between architectural complexity, report quality, reliability and computational cost? Evaluation: How can the quality and reliability of AI-generated technical reports be measured in a reproducible way? Expected Contribution The thesis is expected to produce: one or more alternative multi-agent RAG architectures integrated into the HOMER research platform; an experimental implementation demonstrating the proposed approaches; a reproducible evaluation methodology for technical knowledge synthesis; experimental results comparing alternative architectures; an analysis of their strengths, limitations and failure modes; recommendations for the further evolution of HOMER. The contribution is therefore both architectural and experimental : the student will actively design and build new AI system architectures, while also developing the evidence needed to understand their effectiveness. Skills and Technologies The project provides hands-on experience with current Generative AI technologies, potentially including: Large Language Models and Generative AI; Retrieval-Augmented Generation; agentic AI and multi-agent systems; LLM-based planning and reasoning; embeddings and vector databases; LLM evaluation and benchmarking; prompt and context engineering; Python; LangChain / LangGraph or comparable frameworks. The project is particularly suitable for a student interested in Generative AI, agentic AI, RAG, NLP, AI system architecture, or the reliability of LLM-based systems . Research Environment The thesis will be conducted in collaboration with SCK CEN , using the HOMER research platform and representative technical documentation from the MYRRHA and LEANDREA programmes. The project combines a genuine engineering application with an open-ended AI research problem: how can multiple AI agents collaborate effectively to transform large collections of technical evidence into reliable, traceable and useful engineering reports? The student will have the opportunity to design and experiment with modern agentic AI architectures in a real-world setting, while maintaining a clearly defined research scope appropriate for a Master's thesis.