Efficient MLLM Inference via Approximate and Exact Computing

Numerical Optimization for Efficient MLLM Inference: Bridging Approximate and Exact Computing

Multimodal Large Language Models (MLLMs) deliver unprecedented capabilities but face severe deployment bottlenecks due to their massive memory footprints and computational demands. While standard efficiency techniques often rely on ad-hoc heuristics—such as naive magnitude pruning or nearest-integer rounding—these methods lack theoretical guarantees and frequently cause unpredictable degradation in complex reasoning tasks. This tutorial discusses the paradigm by reframing model compression, specifically sparsification and quantization, as rigorous, constrained numerical optimization problems. By mapping discrete, NP-hard combinatorial challenges into continuous spaces, we can systematically trade computational exactness for hardware efficiency. The session will cover convex relaxations for structured sparsity (e.g., group $\ell_1$-norm penalties), continuous optimization formulations for local Post-Training Quantization (PTQ) reconstruction, and the mechanics of handling non-smooth step functions in Quantization-Aware Training (QAT) via surrogate gradients and ADMM. Furthermore, the tutorial will explore how embedding exact optimization problems directly into the network architecture via implicit layers allows highly compressed, low-precision models to maintain rigorous mathematical bounds. Attendees will leave equipped with a principled mathematical toolkit to optimize MLLM inference without sacrificing representational capacity or reliability.

Building LLaVA-OneVision-2: From Codec-Aligned Visual Encoding to Long-Video Curriculum Learning

TBD

Serving Omni-Modal Models at Scale: Systems Design and Lessons from SGLang Omni

A request to a modern omni-modal model no longer runs through a single autoregressive loop: it traverses heterogeneous stages — modality encoders, thinkers, talkers, autoregressive or diffusion generators, codec decoders, and streaming vocoders. Existing systems fix how these stages execute: monolithic engines force stages with different bottlenecks under one scheduler, while stage-graph systems hard-wire full disaggregation, paying orchestration costs on exactly the edges that need per-step feedback. This talk presents SGLang Omni, an open-source serving framework that decouples execution structure from stage topology and derives it instead from three measurable characteristics: a stage's bottleneck resource selects its execution engine, the synchronization granularity of each inter-stage dependency decides whether the boundary is fused or split, and per-replica device saturation sets replica counts and same-device colocation. A model declares its topology once and is served behind OpenAI-compatible chat and speech APIs; every structural decision remains a deployment-time configuration rather than an architectural change. Across representative TTS models, SGLang Omni improves request throughput by 2.2×–3.05× over dedicated tuned baseline at high concurrency with comparable or better intelligibility, sustaining real-time generation where baselines fall behind.