Qwen-RobotWorld：具身智能体的无界世界

Paper Embodied intelligence requires agents to perceive, reason about, and act within physical environments. World models offer a scalable path forward — but current approaches face a fundamental tension. General video generation models learn rich visual priors but lack the ability to model embodied physics. Domain-specific embodied models are tailored to individual scenarios and cannot generalize across embodiments.

Qwen-RobotWorld bridges this gap by treating natural language as a universal action interface. A single instruction like "pick up the red cup and place it on the shelf" implicitly encodes the complete action sequence, goal state, and physical constraints — no robot-specific control interface needed. This allows manipulation, autonomous driving, and indoor navigation to be trained jointly, with each domain's physical knowledge reinforcing the others.

Language unifies the action space: world knowledge and embodied knowledge reinforce each other within a single model, enabling cross-scenario, cross-task physical generalization.

Key Highlights#

Top-Tier Across 4 Benchmarks

20+Robot Embodiments Unified

8.6M Cross-Scenario Training Pairs

1300+Manipulation Skills

Language-Driven Unified Action Interface — natural language standardizes 20+ robot embodiments and 500+ action categories into one interface, enabling joint cross-scenario training Dual-Stream Diffusion World Model — MMDiT with Qwen2.5-VL as action encoder, combining deep language understanding with internalized physical world knowledge Cross-Scenario Physical Generalization — manipulation, driving, navigation, and human-to-robot transfer jointly trained under 8.6M video-text pairs Multi-View Geometrically Consistent Generation — synchronized 2–4 camera streams with 3D-consistent object identity and motion trajectories

Model Architecture#

Dual-Stream Diffusion World Model#

Qwen-RobotWorld adopts a dual-stream Multimodal Diffusion Transformer (MMDiT):

Understanding stream processes semantic features from a frozen Qwen2.5-VL encoder, representing the language action $a{t}$a t​. Generation stream processes visual latents from a video-compatible VAE, representing the visual state $s{t}$s t​.

The two streams interact via joint attention at every layer, enabling bidirectional cross-modal fusion throughout the denoising process.

Using an MLLM as the action encoder — rather than lightweight encoders like T5 or CLIP — provides two key advantages: (1) deep language understanding accurately parses complex, compositional instructions into precise condition signals; (2) internalized world knowledge (e.g., that robot arms are rigid bodies with fixed joint constraints) implicitly constrains physically plausible transitions, preventing common failure modes like object deformation across frames.

Scene2Robot: Human-to-Robot Transfer#

Scene2Robot enables cross-embodiment video editing: human demonstrations are retargeted to 14 robot morphologies via a multi-segment conditioning mechanism, where joint attention allows the generation to simultaneously attend to scene appearance and robot motion trajectory. This capability both serves as a data scaling engine during training and enables human-to-robot transfer at inference time.

Multi-View Geometrically Consistent Generation#

Single-camera observation inevitably occludes critical contact and spatial details. Qwen-RobotWorld generates 2–4 synchronized camera streams — main view, wrist-mounted views, and third-person views — with geometrically consistent object identity and motion across all viewpoints. During training, synchronized frames from multiple cameras are spatially concatenated into a single input; the model generates all views simultaneously, with asymmetric 3D RoPE providing spatial encoding and attention layers naturally establishing cross-view correspondence — without any architectural modification. This cross-view consistency further acts as a geometric regularizer, teaching the model object shape, depth, and spatial layout.

Data: Embodied World Knowledge#

EWK Dataset#

The Embodied World Knowledge (EWK) dataset is organized along four complementary axes, each targeting a distinct source of physical variation:

Multi-Embodiment — human hands, 7 robot arm configurations, ego vehicles, mobile agents, spanning 20+ distinct robot models Multi-Task — atomic manipulation skills, long-horizon compositions, locomotion, dynamic/deformable interactions across 500+ action categories Multi-Scenario — real-world first, sim-augmented: kitchens, workshops, outdoor settings, plus photorealistic simulation for downstream VLA evaluation Multi-View — main, wrist, and synchronized multi-view streams (~1.6M of 6M embodied samples include 2–4 view concatenations)

View Detailed Dataset Inventory

Action-Language Mapping#

The central challenge in building a universal world model is representational heterogeneity: manipulation uses joint angles, driving uses steering commands, navigation uses heading vectors — each requiring a separate model. Our action-language mapping framework resolves this by projecting all action signals onto a shared natural language space, so that videos from a Franka gripper, an autonomous vehicle, and a navigation agent all become instances of the same language-conditioned video generation task.

A hierarchical five-layer annotation pipeline ensures caption quality and precision:

Task GoalHigh-level intent — what should change between states

Action DetailSpatio-temporal trajectories with explicit viewpoint declaration

Physical FeedbackObservable consequences on the environment

Comprehensive CaptionFull description for precise prediction

Concise CaptionEssential elements for brief task-level commands

During training, comprehensive and concise descriptions are sampled with equal probability, so the model handles both detailed trajectory specifications and brief task-level commands.

Training#

Training follows a general-to-expert progressive curriculum:

Pretraining on general data and human interaction videos (Ego4D, EPIC-Kitchen) builds broad visual priors — the T2I task specifically anchors object geometry that transfers to video generation through the shared backbone. SFT then progressively deepens embodied expertise across four phases while keeping general data in every batch, ensuring both capabilities advance together rather than trade off.

Demos#

Fine-Grained Language Grounding#

Given identical initial frames, the model produces qualitatively distinct videos when a single keyword differs. It also handles complex, multi-step instructions requiring long-horizon reasoning.

Contrastive Instruction Following:

L: Pick up the red strawberry

R: Pick up the yellow potato

L: Place pen on the wooden tray

R: Place pen on the white paper

L: Hand glue to the person

R: Place glue into the penholder

Complex Multi-Step Instructions:

Sequentially pick up the red and yellow bell peppers, place them on the table from left to right

Grab the yellow-and-blue stacked block, position it above the green-and-blue stacked block

Cross-Domain Generalization#

(A) Cross-Embodiment:

(B) Cross-Task × Cross-Environment:

(C) Multi-View Consistency:

(D) Zero-Shot Robustness:

Ours

LVP

Cosmos2.5-14B

Human-to-Robot Transfer#

The Scene2Robot mechanism preserves task intent from a human demonstration (left) while adapting motion to embodiment-specific kinematic constraints (right).

Beyond Manipulation: Driving and Navigation#

The learned world model generalizes beyond robot manipulation to broader mobility scenarios.

Autonomous Driving. Generated driving episodes from Bench2Drive, NVIDIA PhysicalAI-AD, Sekai, and Waymo demonstrate coherent scene dynamics and vehicle behaviors.

Indoor Navigation. Egocentric navigation episodes from VLNVerse show the model’s ability to simulate first-person movement through complex indoor environments.

From tabletop manipulation to autonomous driving and indoor navigation — Qwen-RobotWorld demonstrates that a unified world model can generalize beyond a single morphology or scenario family.

We evaluate against general video generation models (Sora2, Veo3, Wan2.6, Kling, LTX-2) and embodied world models (Cosmos, LVP, GigaWorld, Vidar, Wow) across four benchmarks.

EWMBench

4.60

Embodied motion fidelity

DreamGen

4.952

Instruction following & physics alignment

WorldModelBench

8.99

Physical reasoning & instruction following

PBench

0.804

Physical behavior evaluation

EWMBench

Strong motion fidelity (HSD 0.566), high scene consistency (0.914), and perfect logic constraint satisfaction.

DreamGen Bench

Strong object-level compositional generalization (GR1-Object IF: 0.878), with consistent physics alignment across all subsets.

WorldModelBench

Perfect physics adherence (1.00) across Newton's laws, mass conservation, fluid dynamics, and gravity, with strong instruction following (2.33/3.0).

PBench

Strong domain understanding (0.857) and motion smoothness (0.990), reflecting consistent temporal coherence across physical scenarios.

If you find our work helpful, feel free to give us a cite.

@article{qwenrobot-world, title={Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation}, author={Qwen Team}, year={2026}}