Hy-Nomas: Towards Free-form Interactive World Generation

Reading guide

Problem and motivation

Interactive video world models act as simulated virtual environments that generate future visual observations in response to ongoing user actions. Unlike standard text-to-video (T2V) systems that render pre-specified clips, an interactive world model must:

1. Accept dynamically arriving commands during generation.
2. Handle both semantic interventions (free-form text) and spatial navigation (continuous camera poses).
3. Prevent earlier actions from being mistakenly repeated in subsequent segments while retaining enduring scene context and visual consequences.
4. Avoid severe compounding drift during autoregressive rollout without suffering catastrophic memory explosion.

Hy-Nomas addresses these challenges by adapting an 8B bidirectional diffusion transformer (HunyuanVideo 1.5) into a unified, segment-wise autoregressive video world model.

Key ideas

• Action-Aware Causal Attention: Distinguishes between persistent descriptive scene context ("head") and temporally localized action instructions ("tails"). Descriptive context remains accessible to future segments, while action tokens are restricted to their originating segment and temporal bounds, preventing action re-execution.
• Continuous Camera Conditioning: Uses frame-to-frame relative transformations $\Delta P_t=P_{t-1}^{-1}{P}_t$ normalized by video-level motion scales and injected via dedicated per-block cross-attention.
• Prompt Enhancer (PE): A fine-tuned multimodal vision-language model (based on Qwen3.8-27B) that translates multi-turn user intent and historical observations into structured JSON conditions specifying scene changes, camera tags, and contained actions.
• Segment Forcing with Streaming Backward: Trains the autoregressive generator on its own one-step denoised predictions to match the inference-time error distribution while keeping memory bounded.
• Distillation to 4 NFEs: Distills the 100-step Euler sampler down to 4 network function evaluations per segment using Phased Consistency Model (PCM) initialization, Decoupled Distribution Matching Distillation (DDMD), and causal adversarial refinement.

Method (with the key equations)

Latent Space Video Representation

RGB video $X\in {\mathbb{R}}^{B\times 3\times F\times H\times W}$ is compressed via a causal DCAE-V encoder $\mathcal{E}$ into latents $Z$: $$Z=\mathcal{E}(X)\in {\mathbb{R}}^{B\times 32\times F_z\times H_z\times W_z},F_z=\lfloor \frac{F-1}{4}\rfloor +1,H_z=\lfloor \frac{H}{32}\rfloor ,W_z=\lfloor \frac{W}{32}\rfloor$$ Each latent position is projected to the 2,048-dimensional DiT hidden space.

Training Objective

The model is trained via linear flow matching on clean latents $Z$ and noise $ϵ\sim \mathcal{N}(0,I)$: $$Z_t=(1-t)Z+tϵ,V_t=ϵ-Z$$ The network predicts velocity $V_t$, supervised with mean squared error.

Action-Aware Causal Attention & Camera Conditioning

For segment $j$, video tokens attend causally to past segments and bidirectionally within segment $j$. Within each segment, query tokens inside an action window only attend to that action's text tokens and the environment head. In cross-attention, visual tokens $V$ attend to 64-D relative camera tokens $C$: $$\tilde{V}=V+\mathrm{CrossAttn}(V,C)$$

Segment Forcing

To prepare the model for inference-time error accumulation, segment $s$ computes its flow loss and updates the KV cache for segment $s+1$ using a single-step denoised latent: $${\tilde{\mathbf{x}}}_{s,t_{\mathrm{ctx}}}={\mathbf{x}}_{s,t}+(t_{\mathrm{ctx}}-t){\hat{\mathbf{v}}}_s,t_{\mathrm{ctx}}\sim \mathcal{U}(0.1,0.3)$$ Streaming backward releases intermediate activations segment-by-segment, keeping only one segment's computation graph in GPU memory.

Distributed Packing FLOP Estimation

Videos are packed into batches up to 360K tokens, balanced by estimated computational FLOPs: $$F_i=\alpha n_i+\beta A_i$$ where $n_i$ is the number of valid tokens, and $A_i={\sum }_{p\in i}|\{q:M_{pq}^{(i)}\ne -\infty \}|$ is the number of unmasked query-key pairs.

Results (with the most important numbers)

Evaluated on Hy-Nomas-Bench (280 test scenarios across 4 benchmarks):

• Bench-1 (Multi-Turn Text Interaction):
  • Prompt following: Hy-Nomas achieves 65.0% win vs. 10.0% loss against HappyOyster-Directing, and 75.0% win vs. 8.3% loss against LingBot-World v2.
  • Physical plausibility: Hy-Nomas achieves 45.0% win vs. 11.7% loss against HappyOyster-Directing, and 68.3% win vs. 5.0% loss against LingBot-World v2.
• Bench-2 (Camera Navigation): Hy-Nomas achieves higher net win rates across 100 trajectories: 32.0% win vs. 14.0% loss against Genie 3, and 36.0% win vs. 10.0% loss against HappyOyster-Wandering.
• Bench-3 (Interleaved Text & Camera): Achieves 75.0% win (vs. 8.3% loss) in prompt following, and 73.3% win (vs. 1.7% loss) in physical plausibility against LingBot-World v2.
• Bench-4 (Spatial Memory): Hy-Nomas outperforms LingBot-World v2 (58.3% win vs. 1.7% loss) and HappyOyster (30.0% win vs. 23.3% loss), but trails Genie 3 (13.3% win vs. 30.0% loss, with 56.7% ties), which benefits from Street View grounding.
• Single-GPU Deployment: INT8 quantization, sliding-window attention, and CPU offloading enable 100-second 720p rollouts on a single NVIDIA RTX 4090 with 20.53 GiB peak GPU memory.

Limitations and open questions

• Long-term Spatial Memory Gap: Genie 3 remains superior in spatial consistency when revisiting locations after complex camera excursions; training without looped trajectory data limits anchor retention.
• LoRA Camera Regression: Fine-tuning on domain-specific datasets (e.g., cinematic styles) improved prompt adherence and visual aesthetics but degraded camera control performance (19% win vs. 37% loss) due to poor camera trajectory diversity in downstream data.
• Physical Accuracy in Robotics: Although out-of-distribution dual-arm actions can be generated via LoRA adaptation, video diffusion does not natively guarantee execution-level robot physics or force-torque compliance.
• Off-Screen State Changes: Changes occurring outside the field of view (e.g., an unobserved clock ticking) cannot be captured purely by visual KV retrieval without explicit external symbolic agent state tracking.