Building Pretraining Data for World Models: An Unreal Engine-Based Pipeline for Action-Conditioned Video Generation

A new Unreal Engine-based production pipeline is being used to generate large-scale action-conditioned video for world-model pretraining. The goal is to create training data where visual changes are aligned with the actions that caused them, something real-world video usually cannot provide because the underlying control signals are missing.

The system splits work into two stages. First, it runs real physics in PIE and records character states, control inputs, and camera states into an intermediate trajectory format. Then it replays those trajectories in a separate engine process and renders them offline with Movie Render Queue, which lets the team combine simulation fidelity with higher-quality final output.

The production setup is substantial: 25 servers, each with eight NVIDIA RTX 5090 GPUs. From 2,384 asset packs, the pipeline kept 429 levels and a pool of 40 humanoid characters. It has already produced 2,691 hours of 1080p video and 6,076 hours of 720p video, with extra tooling for cache-aware scheduling, automated scene screening, recovery from partial failures, and cluster health monitoring.

For developers, the interesting part is less the AI headline than the production engineering behind it. This is a reminder that Unreal can serve as more than a runtime or content-creation tool; it can also be the backbone for synthetic-data factories. The authors also call out a practical limitation: perceptual quality filters are useful, but they are not a perfect proxy for whether data is actually good for world-model training.