EDB-IPP Project: Advancing Domain-Specific Multimodal Large Language Models and AI Governance for Vertical Domain Applications

Job description

Job Description: Rakuten Asia, in partnership with the Economic Development Board ( EDB ) through the Industrial Postgraduate Programme ( IPP ), is seeking new PhD students. We are looking for individuals with a robust understanding of deep learning, machine learning, and natural language processing to contribute to our innovative research projects. Essential requirements include proven hands-on expertise and strong engineering skillsets, specifically in the development and training of PyTorch models. IPP Programme Benefits Candidates successfully selected for this programme will receive full sponsorship for their postgraduate studies and will be hired by Rakuten Asia upon successful completion. Collaboration Model The collaboration will include joint PhD student supervision. The outcomes will strengthen Singapore’s position as a leader in responsible AI innovation, enhance the competitiveness of local industries, and contribute to the global research community through publications and open-source releases. Project Outline This research proposal aims to develop and optimize domain-specific multimodal LLMs (MLLMs) tailored for verticals such as e-commerce, travel marketplaces, customer service, and the fintech industry. The collaboration between AIDD and academic partners under the EDB IPP will focus on three core areas: (1) building MLLMs that integrate domain knowledge for rapid business deployment, (2) advancing AI governance through intelligent, explainable review systems, (3) optimizing model architectures for multilingual, multi-task, and multimodal performance in real-world e-commerce scenarios. The project will leverage cutting-edge AI and machine learning techniques, with an emphasis on practical deployment and governance, to drive innovation and responsible AI adoption in Singapore’s digital economy. Requirements of IPP candidates: Strong academic records in machine learning, deep learning and multimodal AI. They should have expertise in one of the following Areas: Computer Vision (CV) and multimodal: In-depth knowledge in fields such as image search, classification, segmentation, detection, OCR, graph neural networks, etc. Natural Language Processing (NLP): Solid knowledge in pretraining, NLU, multilingual and cross-lingual learning, NLG, semi-supervised learning, etc. Audio signal processing: Expertise in audio signal processing, including speech recognition, speaker identification, audio event detection, and audio feature extraction. Experience in LLM-related projects and applying them to unify real-world problems is a plus. Strong practical abilities, with achievements in competitions and publication records in top-tier conferences (e.g., ACL, EMNLP, ICML, CVPR, ICCV, AAAI, ICASSP) are highly valued. Excellent hands-on expertise and engineering skillsets: Experience with training and deploying PyTorch models. Knowledge of training acceleration methods such as mixed precision training and distributed training is a plus. Introduction/Background The rapid evolution of large language models (LLMs) and multimodal AI systems has transformed industries by enabling advanced natural language understanding, content generation, and intelligent automation. However, most state-of-the-art models are trained on general-purpose data, limiting their effectiveness in specialized domains such as e-commerce, customer service, and fintech. Furthermore, the increasing complexity and societal impact of AI systems necessitate robust governance frameworks to ensure responsible deployment, particularly in high-stakes applications such as product moderation. Singapore’s digital economy, anchored by vibrant e-commerce and fintech sectors, stands to benefit significantly from domain-adapted AI solutions. By collaborating with leading universities through the IPP, AIDD seeks to bridge the gap between academic research and industrial application, fostering talent development and innovation in AI model development, deployment, and governance. Research Questions or Hypotheses How can domain-specific LLMs and MLLMs be effectively developed and fine-tuned to address the unique requirements of e-commerce, customer service, and fintech applications? What algorithmic and architectural innovations can enhance the explainability and effectiveness of AI governance systems for content moderation and IPR protection? How can model optimization techniques improve the multilingual, multi-task, and multimodal performance of large models in real-world e-commerce scenarios? Research Design: A multi-phase, collaborative research approach will be adopted: Data Collection and Curation: Partner with industry to collect and anonymize large-scale, domain-specific datasets (text, images, audio, video) from e-commerce, customer service, and fintech platforms. Annotate data for key tasks (e.g., product classification, content moderation, IPR detection). Model Development: Pretrain and fine-tune LLMs/MLLMs using state-of-the-art frameworks (e.g., PyTorch, TensorFlow). Integrate domain knowledge via supervised and self-supervised learning, leveraging techniques such as prompt engineering, knowledge distillation, and graph neural networks. AI Governance Optimization: Develop explainable review systems using XAI methods (e.g., attention visualization, counterfactual reasoning). Implement and evaluate algorithmic improvements for automated moderation. Model Optimization and Deployment: Apply training acceleration methods (mixed precision, distributed training) for scalable model development. Optimize models for multilingual, multi-task, and multimodal performance, benchmarking against public datasets and real-world scenarios. Expected Outcomes and Significance The project is expected to deliver: Domain-Specific LLMs/MLLMs: High-performance models tailored for e-commerce, customer service, and fintech, enabling rapid deployment in business scenarios. Advanced AI Governance Systems: Explainable, automated review systems that improve decision transparency and compliance in content moderation. Optimized Model Architectures: Scalable, efficient models with superior multilingual, multi-task, and multimodal capabilities, validated in real-world industrial settings. Rakuten is an equal opportunities employer and welcomes applications regardless of sex, marital status, ethnic origin, sexual orientation, religious belief or age.