Efficient MLLM Inference via Approximate and Exact Computing

Serving Omni-Modal Models at Scale: Systems Design and Lessons from SGLang Omni

A request to a modern omni-modal model no longer runs through a single autoregressive loop: it traverses heterogeneous stages — modality encoders, thinkers, talkers, autoregressive or diffusion generators, codec decoders, and streaming vocoders. Existing systems fix how these stages execute: monolithic engines force stages with different bottlenecks under one scheduler, while stage-graph systems hard-wire full disaggregation, paying orchestration costs on exactly the edges that need per-step feedback. This talk presents SGLang Omni, an open-source serving framework that decouples execution structure from stage topology and derives it instead from three measurable characteristics: a stage's bottleneck resource selects its execution engine, the synchronization granularity of each inter-stage dependency decides whether the boundary is fused or split, and per-replica device saturation sets replica counts and same-device colocation. A model declares its topology once and is served behind OpenAI-compatible chat and speech APIs; every structural decision remains a deployment-time configuration rather than an architectural change. Across representative TTS models, SGLang Omni improves request throughput by 2.2×–3.05× over dedicated tuned baseline at high concurrency with comparable or better intelligibility, sustaining real-time generation where baselines fall behind.

LLaVA-OneVision-2: Understanding the Visual World Across Space and Time

How should vision-language models understand a world that changes across space and time? Our work begins with OneVision-Encoder, a shared representation for images and video that uses codec signals to preserve visual changes within a limited token budget. LLaVA-OneVision-2 extends this idea through codec-stream tokenization, adapting visual token allocation as events unfold. OV-2 is progressively trained from OV-1.5's image-pretrained foundation, extending its image understanding capabilities to long-video understanding and spatial grounding. This progression connects efficient visual representation with a training curriculum that builds temporal and spatial capabilities on an established image-language foundation.

The Trinity of MLLM Compression — Advanced PTQ, Pruning, and Context Compression

Deploying massive Multimodal Large Language Models (MLLMs) is fundamentally restricted by memory bandwidth and compute bottlenecks during inference. In this tutorial, we will discuss these constraints by detailing a synergistic framework of approximate model compression across three distinct axes: (1) The Precision Axis (Quantization): Alleviates memory bandwidth limits by compressing weights and KV caches into lower bit-widths using methods like AWQ, GPTQ, and TurboQuant. (2) The Structural Axis (Pruning): Lowers the required arithmetic workload (FLOPs) per token by permanently removing redundant parameters via Taylor-based Vision Transformer pruning and rank constrained pruning. (3) The Data Axis (Token Merging): Addresses on the $\mathcal{O}(N^2)$ compute bottleneck of long visual contexts by dynamically fusing redundant tokens to shrink input sequence length without retraining. Combining these techniques, we can execute 70B+ parameter MLLMs efficiently while maintaining visual reasoning capabilities as desired.