LingBot‑VA 2.0: How Ant Group’s Causal Video‑Action Model Is Redefining Embodied AI

Why a native video‑action foundation matters

Most video‑action models in robotics are cobbled together from components designed for digital media. A reconstruction‑oriented VAE preserves pixel fidelity but discards the physical structure needed for manipulation, while a bidirectional diffusion backbone is too slow for closed‑loop control. Ant Group’s embodied‑AI unit, Robbyant, argues that these mismatches create three concrete bottlenecks: limited physical semantics, prohibitive inference latency, and a training objective that never teaches a robot how its actions reshape the world.

LingBot‑VA 2.0 is positioned as the first “embodied‑native” foundation model that tackles all three. Instead of fine‑tuning a pre‑trained video generator, the team pre‑trains the entire stack from scratch for embodiment, yielding a model that can predict future states, re‑ground on fresh observations, and drive a robot at a staggering 225 Hz.

From VAE to a semantic visual‑action tokenizer

The original pipeline relied on a compression‑only VAE. LingBot‑VA 2.0 replaces it with a tokenizer inspired by RepWAM, adding two extra objectives to ordinary reconstruction. First, a semantic alignment loss pulls visual latents toward a frozen perception encoder, ensuring that the latent space encodes meaningful object‑level information. Second, a latent‑action objective extracts compact transition variables between consecutive latents, while an inverse dynamics model predicts each latent action and a forward dynamics model decodes it into a transport map plus residual.

Because world states and actions now share a single latent space, even unlabeled web videos provide action‑relevant supervision. The impact is measurable: swapping the older WAN2.2 VAE for the new semantic tokenizer lifts a 1.3 B‑parameter model’s performance from 78.0 to 86.6 (the exact metric is not named but the jump is reported directly in the paper).

Causal DiT and the sparse‑MoE video stream

Version 1.0 of LingBot‑VA finetuned a bidirectional diffusion model into a causal one. Version 2.0 goes further by pretraining a causal DiT (Diffusion Transformer) natively. The architecture retains the Mixture‑of‑Transformers layout, but now the video expert replaces its dense feed‑forward network (FFN) with a sparse Mixture‑of‑Experts (MoE) layer. This layer holds 128 routed SwiGLU experts, top‑8 routing, and one shared expert, using the “Loss‑Free Balancing” strategy to keep the load even without auxiliary losses.

The action expert stays dense, with a hidden dimension of 768. In total, the video backbone contains roughly 13.0 B parameters, about 1.9 B active at any forward pass. Adding the action expert and Multi‑Chunk Prediction (MCP) heads brings the training footprint to about 15.3 B parameters, with roughly 2.5 B active per token at inference. These numbers illustrate how the model balances massive capacity with sparsity to stay tractable for real‑time control.

Training tricks that speed up learning

Two innovations accelerate convergence dramatically. The first is Multi‑Chunk Prediction (MCP), which attaches three lightweight modules that predict the next three video chunks instead of the usual single‑step teacher forcing. In ablation studies, MCP matched the baseline’s 45 k‑step accuracy in just 20 k steps, a 2.3× training speedup.

The second is co‑training five objectives simultaneously: text‑to‑image (T2I), text‑to‑video (T2V), text‑image‑to‑video‑action (TI2VA), in‑context learning (ICL), and human‑robot co‑training. By keeping all objectives alive throughout training, the model avoids catastrophic forgetting and retains a richer set of priors.

Optimization uses a rectified‑flow objective combined with a hybrid Muon plus AdamW optimizer, while distillation reduces the video sampler from five steps to two and the action sampler from ten steps to two, shaving latency further.

Real‑world deployment: asynchronous foresight reasoning

Even with a sparse backbone, a serial bottleneck can stall a robot if it waits for the model’s output. LingBot‑VA 2.0 solves this with “Foresight Reasoning”: the video expert runs ahead, imagining the outcome of the current chunk a_t, while the robot executes it. The action expert then decodes the next action a_{t+1} from that imagined future. When the real observation arrives, it overwrites the imagined latent z_{t+1}, and a forward‑dynamics grounding loss trains the video expert to stay aligned.

This asynchronous pipeline enables control at 225 Hz, while a higher‑level Vision‑Language Model (VLM) planner—LoRA‑finetuned on a frozen vision tower—runs at about 2 Hz behind a shared buffer. The planner emits structured JSON (e.g., {"instruction":..., "local_scene_description":...}) that the policy reads at each chunk boundary, ensuring planner latency never blocks execution.

Evaluation spans both simulation (RoboTwin 2.0) and real hardware across 50 tasks, using 2,500 clean and 25,000 randomized demonstrations. Distilled FP8 TensorRT engines, a paged/ragged KV cache with FlashInfer attention, and host‑side overhead removal complete the deployment stack.

Open challenges and next steps

LingBot‑VA 2.0 demonstrates that a causal, sparsely‑expert video backbone can power high‑frequency robot control, but several questions remain. The current system still relies on a chunk‑level policy; extending this to truly long‑horizon planning will require tighter integration between the VLM planner and the low‑level foresight module. Moreover, the drift‑mitigation strategy—re‑grounding on the true latent after each observation—works well for short horizons, but its robustness over dozens of seconds of continuous interaction is untested.

Future work could explore hierarchical latent spaces that separate short‑term dynamics from long‑term task semantics, or adaptive routing in the MoE layer that prioritizes experts based on the physical context. Addressing these gaps will be crucial for moving from laboratory benchmarks to everyday embodied AI assistants.