JoyAI-Video-Edit

Real-Time Open-Ended Video Editing with Autoregressive Diffusion

🐶 JoyAI-Video-Edit

JoyAI-Video-Edit is a real-time, instruction-guided video editing framework designed for open-ended video streams.

Unlike conventional offline video editing models that require a complete video before inference, JoyAI-Video-Edit processes frames causally as they arrive. It can continuously edit a live camera stream or an uploaded video according to natural-language instructions without accessing future frames or requiring a predefined video length.

The framework combines:

• an MLLM-based condition encoder for understanding editing instructions,
• a causal video VAE for streaming visual representation,
• and a 16B multimodal diffusion transformer for high-quality video editing.

Through autoregressive diffusion modeling, aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling, JoyAI-Video-Edit achieves stable long-range editing while substantially reducing inference latency.

Our deployment system reaches 30.19 FPS end-to-end at 720 × 1280 resolution, enabling real-time streaming video editing.

---

🔥 News

• 2026.08.06 — 🎉 We released the live demo of JoyAI-Video-Edit.
  Try it here: Streaming V2V Demo

• 2026.08.05 — 🎉 We released the model checkpoints, deployment code, online demo, and technical report.

---

💎 Highlights

⚡ Real-Time Streaming Editing

JoyAI-Video-Edit edits frames continuously as they arrive instead of waiting for the entire video.

It supports open-ended streams and does not require a predefined video duration.

🎯 Instruction-Guided Video Editing

The model supports a broad range of editing instructions, including:

• Subject modification
• Subject addition
• Subject removal
• Subject replacement
• Local object editing
• Background replacement
• Style transfer
• Appearance modification
• Motion editing
• Reference-guided video editing

🔄 Autoregressive Diffusion

JoyAI-Video-Edit introduces an autoregressive diffusion framework consisting of:

• MLLM condition encoder
• Causal video VAE
• 16B multimodal diffusion transformer
• Chunk-wise autoregressive generation
• Long-horizon temporal optimization
• Bounded KV-state inference

🚀 Real-Time 720p Deployment

The full deployment pipeline achieves:

30.19 FPS @ 720 × 1280

with end-to-end streaming inference.

---

🎬 Demo

Try our online real-time video editing demo:

👉 https://joyai-labs.jd.com/v2v/

Project repository:

👉 https://github.com/jd-opensource/JoyAI-Video-Edit

Technical report:

👉 https://arxiv.org/abs/2608.03974

---

📦 Model Download

The released JoyAI-Video-Edit checkpoints are hosted on Hugging Face.

Hugging Face CLI

First install the Hugging Face CLI:

pip install -U huggingface_hub

Then download the complete model repository:

hf download jdopensource/JoyAI-Video-Edit \
    --local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit

Alternatively, you can use:

huggingface-cli download jdopensource/JoyAI-Video-Edit \
    --local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit

The downloaded checkpoint directory should look like:

deploy/deps/checkpoints/JoyAI-Video-Edit/
├── config.json
├── dit/
│   └── joyai_video_edit_dit_0804.pth
└── vae/
    ├── config.json
    └── diffusion_pytorch_model.safetensors

Note

JoyAI-Video-Edit currently uses its own deployment and inference pipeline. The released checkpoint should not be loaded directly with diffusers.DiffusionPipeline.from_pretrained().

MiMo-VL and the ONNX detector files are additional runtime dependencies. Please refer to the deployment documentation in the GitHub repository for details.

---

🚀 Quick Start

1. Clone the Repository

git clone https://github.com/jd-opensource/JoyAI-Video-Edit.git
cd JoyAI-Video-Edit

2. Create the Environment

conda create -n joyai-video-edit python=3.10 -y
conda activate joyai-video-edit

python -m pip install -r requirements.txt

3. Download JoyAI-Video-Edit

hf download jdopensource/JoyAI-Video-Edit \
    --local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit

4. Prepare Runtime Dependencies

MiMo-VL and the required ONNX detector files are external runtime dependencies.

Please follow:

DEPLOYMENT.md

for detailed checkpoint preparation and deployment instructions.

5. Launch

cd deploy
bash run_server.sh

Then open:

http://localhost:8080

For deployment on a remote server, bind the service to 0.0.0.0 and expose the selected port or use SSH port forwarding.

---

🛠️ Custom Deployment

For custom deployment settings, edit:

deploy/run_server.sh

You can configure:

• JoyAI-Video-Edit checkpoint paths
• CUDA device placement
• Server host
• Server port
• Runtime cache directories

The default deployment script uses persistent TorchInductor, Triton, and CUDA cache directories so that compiled artifacts can be reused across launches.

TORCHINDUCTOR_AUTOGRAD_CACHE is not required for inference-only serving.

---

🚧 Roadmap

• Consumer GPU support
  Optimize deployment for consumer GPUs such as NVIDIA GeForce RTX 5090.

• Stronger JoyAI-Video-Edit model
  Improve overall editing quality with a particular focus on reference-image-guided video editing (RV2V).

• Training framework
  Release the complete training pipeline.

• Data pipeline
  Release the data generation and processing framework.

---

📚 Citation

If JoyAI-Video-Edit is useful for your research or project, please cite:

@article{xiao2026joyai,
  title={JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion},
  author={Xiao, Yicheng and Dai, Wenxun and Qin, Xinran and Song, Lin and Zhang, Maoquan and Xu, Hang and Chen, Yukang and Li, Yitong and Zhang, Guohui and Zhang, Yuan and Zhang, Xuying and Zhang, Tommy and Yuan, Jianlong and Li, Peihao and Lu, Shuai and Fu, Siming and Zhao, Chuyang and Han, Xin and Huang, Jie and Li, Wenbo and Ma, Guoqing and Huang, Wei and Qi, Xiaojuan and Huang, Haoyang and Duan, Nan},
  journal={arXiv preprint arXiv:2608.03974},
  year={2026}
}

---

📄 License

JoyAI-Video-Edit is released under the Apache License 2.0.

Please refer to the project repository for the complete license:

https://github.com/jd-opensource/JoyAI-Video-Edit/blob/main/LICENSE