DeepMMPlanner: Deep Multimodal Large Language Model Training Planner

Project Duration: September 2025 ~ August 2026

Total Grant Amount: KRW ~100,000,000 (approx. USD 100K)

Project Overview

• Optimization for Multimodal LLM Training: Proposes a unified system that dynamically selects CUDA kernels, allocates GPU resources per module (e.g., encoders, connectors, decoders), and schedules data batches based on sequence lengths, to eliminate performance bottlenecks in training large multimodal models
• Kernel and Resource Planning: Builds a kernel planner that profiles and selects optimal CUDA kernels at runtime based on input features and hardware specs, and a model planner that performs fine-grained GPU resource partitioning and asynchronous execution tailored to each modality (text, image, speech, video)
• Adaptive Scheduling: Designs a data planner that reduces padding and synchronization overhead by scheduling batches according to sequence length and predicted compute time, improving efficiency in distributed GPU training
• Open, Extensible System Architecture: Implements the DeepMMPlanner system with compatibility for mainstream training frameworks (e.g., PyTorch, DeepSpeed), and releases it as open-source to foster community adoption and industrial deployment
• Expected Impact: Reduces training cost and time significantly, enabling broader accessibility to multimodal LLMs in domains such as healthcare, autonomous driving, and smart interfaces. Promotes sustainable AI by reducing energy use and hardware demand