蚂蚁 inclusionAI 开源 Realtime-Venus 全双工交互系统

A full-duplex interaction system with asynchronous delegation

Venus Team · Ant Group · Tsinghua University

Overview · Quick start · Results · Code map · Citation

Overview

See, listen, and respond in real time—with background tasks running alongside the conversation. Realtime-Venus combines full-duplex conversational models with an asynchronous execution framework. You can ask the assistant to work on a task and continue talking while it runs. The result returns to the same conversation for spoken delivery.

The project brings together three components:

What Realtime-Venus enables

• Continuous audiovisual interaction. Omni observes streaming camera and microphone input and can initiate a response when the evolving scene calls for it.
• Full-duplex conversation. The models process incoming speech while speaking, learning to distinguish acknowledgments and background speech from interruptions that call for a revised response.
• Asynchronous delegation. A model can hand a natural-language task to Harness for reasoning or tool execution while live interaction continues.
• Results in the same conversation. Harness prepares a reply; the conversational model handles its timing and native speech output. Work tracking separates task completion from spoken delivery.

The paper describes the model family, dual-loop runtime, training data, and evaluation. This source release includes the Omni model integration, the reusable Harness package, and a browser demo using a Codex task backend. The demo's microphone-only mode also uses Omni.

Quick start

1. Install dependencies

Run the commands from the repository root. For standalone inference, use Python 3.10 with CUDA and FFmpeg, and install the dependencies for your model:

# Realtime-Venus-Omni
python -m pip install -r /path/to/Realtime-Venus-Omni/requirements.txt

# Realtime-Venus-Audio
python -m pip install -r requirements.txt

For the browser demo, run the installer on Linux. It creates an isolated Python 3.11/3.12 environment and installs the model, web service, and Harness dependencies:

bash install.sh

Then choose standalone inference or the online experience below.

2. Offline inference · Frontend Usage

Download the model checkpoint from the links above and replace the example paths with your local directories.

Realtime-Venus-Omni

export REALTIME_VENUS_MODEL_PATH=/path/to/Realtime-Venus-Omni

python frontend/Realtime-Venus-Omni/offline_chat.py
# With long-video Memory
python frontend/Realtime-Venus-Omni/offline_memory_chat.py

These examples read videos from the checkpoint's assets/ directory and output text and speech. To run full-duplex inference on recorded inputs instead:

python frontend/Realtime-Venus-Omni/duplex_chat.py
python frontend/Realtime-Venus-Omni/duplex_speech_in_chat.py
python frontend/Realtime-Venus-Omni/duplex_memory_chat.py

Duplex examples save subtitled videos. See the Omni guide for inputs and output paths.

Realtime-Venus-Audio

python frontend/Realtime-Venus-Audio/audio_offline_chat.py \
  --model-path /path/to/Realtime-Venus-Audio \
  --audio frontend/Realtime-Venus-Audio/case/case_offline.wav

Offline inference returns text. To run full-duplex inference on recorded audio and save a 24 kHz WAV:

python frontend/Realtime-Venus-Audio/audio_duplex_chat.py \
  --model-path /path/to/Realtime-Venus-Audio \
  --audio frontend/Realtime-Venus-Audio/case/case_duplex.wav \
  --output output/duplex_response.wav

Both scripts accept --system-prompt and --prompt; see the Audio guide for decoding options.

3. Online Duplex experience

The online Duplex demo requires at least one NVIDIA A100 GPU. Place the complete Omni checkpoint in model_weight/, then start the demo:

bash start.sh

For an existing checkpoint elsewhere:

bash start.sh --model-path /path/to/Realtime-Venus-Omni

Complete Codex login if prompted. The launcher starts the model API on 8031 and the browser service on 8032.

If running on a remote server, keep this tunnel open on your local computer, replacing user@server with your SSH login:

ssh -N -L 8032:127.0.0.1:8032 user@server

Open http://localhost:8032, select voice, camera + microphone, or video upload, and start a conversation. Checkpoint layout, configuration, and service management are covered in the Demo guide.

Results

_{Figure 1. Video and audio understanding results from the paper.}

_{Figure 2. Full-duplex interaction results from the paper.}

Code map

Realtime-Venus/
├── harness/                # Context, routing, agents, work state, and delivery
│   ├── README.md
│   └── requirements.txt    # Harness media dependencies
├── frontend/
│   ├── Realtime-Venus-Omni/ # Audiovisual inference examples
│   └── Realtime-Venus-Audio/ # Audio inference examples and input samples
├── demos/
│   ├── model/              # Checkpoint adapter and model HTTP API
│   ├── server/             # Web sessions, media, settings, and artifacts
│   ├── static/             # Browser UI, capture, playback, and logos
│   ├── launcher/           # Configuration and process supervision
│   ├── settings.py         # Saved task and model preferences
│   ├── install.py          # Isolated environment installation
│   └── requirements.txt    # Model and application dependencies
├── assets/                 # Paper figures, report, and Demo screenshot
├── pyproject.toml          # Standalone Harness package
├── requirements.txt        # Shared dependency entry for inference examples
├── config.example.json     # Server configuration template
└── install.sh / start.sh    # Install, start, inspect, and stop the Demo

Citation

@article{zhao2026realtime,
  title={{Realtime-Venus}: A full-duplex interaction system with asynchronous delegation},
  author={{Venus Team,Ant Group;Tsinghua University}},
  journal={arXiv preprint arXiv:2609.13814},
  year={2026}
}

License

The source code in this repository is licensed under the Apache License 2.0, except for components with separate license notices. Third-party fonts and paper figures retain their respective licenses; see the license files in demos/static/fonts/ and assets/README.md.