Streaming Video Generation: Past and Future

Zihan Su, Junhao Zhuang, Weiyang Jin, Zeyue Xue

From Video Generation to Real-Time Interaction

In real-time interactive video generation, what appears next is determined progressively by user input. When the user changes the camera direction, the model needs to generate a new view; when the user changes the prompt, subsequent frames need to respond with new content. As long as the interaction continues, generation needs to continue. This requires the model to generate the next step promptly from the existing video and new input.

CausVid 10 is an important starting point for this direction. Bidirectional video diffusion jointly denoises an entire video, allowing the current frame to draw on future information. Causal generation unfolds computation over time, using the past to generate the present. CausVid distills the capabilities of a bidirectional teacher into a few-step causal student and reuses historical computation through a KV cache, bringing high-quality video models toward streaming output.

Once video generation is unfolded over time, the model repeatedly predicts the next step from what it has already produced. This resembles autoregressive generation in language models, but there are two important differences: the space in which content is generated, and the computation performed at each inference step.

The first difference is between discrete and continuous spaces. A language model selects the next token from a finite vocabulary, constraining its outputs to a discrete set of candidates. Video diffusion generates visual content in a continuous space, without such a vocabulary constraint. Both face error accumulation from self-generated history. In video, subtle deviations in appearance, structure, or motion can enter the conditions for the next step and accumulate through repeated prediction, gradually causing the imagery to drift.

The second difference is in inference computation. At each step, a language model constructs the current token's KV and outputs the distribution over the next token in a single forward pass. Autoregressive video diffusion alternates between two types of forward pass: denoising noisy inputs at \(t>0\), and writing the generated clean frames into historical KV at \(t=0\). Denoising may require multiple forward passes, while history writing prepares the conditions for the next frame or video chunk.

A video model's output needs to be a good frame, and it also becomes the history on which subsequent generation depends. This closely links the quality of the current prediction to the generation that follows.

These characteristics raise two questions about training and efficiency: how to organize the history and current content so that the model learns to continue from the past, and how to reduce the denoising computation at each step so that generation can keep up with interaction.

Teacher Forcing trains subsequent predictions from real history, while Scheduled Sampling 7 explored mixing real and generated history during training. Diffusion Forcing 8 further allows different positions to have different noise levels, bringing sequence prediction and diffusion denoising into a common framework. Few-step distillation addresses another practical issue: even if a model can produce outputs sequentially, each step must still be fast enough to keep up with user interaction.

The history used during training is particularly important. Teacher Forcing provides history from real videos, whereas at inference the model faces its own previous outputs. The ability to predict from real history does not directly substitute for the ability to continue generating from its own outputs.

Learning to predict from real history therefore does not imply that a model can keep generating from its own outputs. Self Forcing 2 introduces the model's own generated history into training: through a rollout with KV caching, the model continues predicting from previously generated content and receives video-level distribution-matching supervision. Both TF and SF learn to predict the current content from history; they differ in whose history is used during training:

Here, \(i\) denotes the current frame or video chunk, \(z_i^t\) is the noisy input, and \(\hat{x}_i\) is the predicted clean content. The key difference between the two lines lies in the conditioning term: real history \(x_{<i}^{\mathrm{data}}\) is replaced by the model's own generated history \(\hat{x}_{<i}\).

This change can also be understood through generation trajectories. Combining generation and history writing gives a stochastic discrete dynamical system, \(h_i=F_\theta(h_{i-1},\varepsilon_i,u_i)\): \(h_i\) includes historical KV together with cache and positional state, \(\varepsilon_i\) represents sampling randomness, and \(u_i\) represents prompts or interactive input. \(F_\theta\) includes the complete sampling process for the current chunk and the cache update. Each prediction changes the conditions for the next step, so local deviations may gradually accumulate through autoregression. Self Forcing unfolds training along the model's own generated history, allowing supervision to cover the deviations encountered during actual generation.

The two curves in the figure illustrate generation trajectories before and after a model update. The model first completes an autoregressive rollout, then updates its parameters through video-level distribution matching, and subsequently generates again with the updated model. The orange arrows represent the effect of this training update: it changes how the model continues from its own history, pushing the generated video distribution toward the target distribution. The curves and background are a conceptual projection of this process.

Taking DMD 13 17 as an example, this effect can be understood as a pull induced by the joint video distributions. On a noisy video \(X_\tau\), the ideal direction is \(f_\tau=\nabla_{X_\tau}\log p_\tau-\nabla_{X_\tau}\log q_{\theta,\tau}\), where \(p_\tau\) and \(q_{\theta,\tau}\) are the noisy video distributions of the target and student, respectively. The target score points toward more plausible video regions, while the student score corrects for where the current distribution is concentrated. This evaluates the relationships among subjects, scenes, and motion across the whole video, and many plausible trajectories can exist under the same prompt.

The history a model encounters depends on its own previous outputs. Self Forcing incorporates this feedback into training, allowing the model to learn subsequent predictions under the conditions it actually produces. The “pull” in the figure therefore takes place through parameter learning; at inference, the model continues to generate step by step on its own.

Toward Better and Longer Generation

After Self Forcing brings self-generated history into training, several questions remain for autoregressive video generation: what model should the student start from, how can supervision over short windows cover a long process, and how can limited context support longer inference?

What initialization does the student need? This is the question addressed by Causal Forcing 11. During ODE initialization of a few-step model, a bidirectional teacher can use future information, while a causal student can only see the past and its current input. If a regression target depends on content unavailable to the student, the same visible input may correspond to different targets. Causal Forcing therefore uses an autoregressive teacher for ODE initialization, aligning the regression target with the information accessible to the student, before proceeding to Self Forcing distribution-matching training.

Causal Forcing++ 12 further advances framewise generation in 1–2 steps. Through causal consistency distillation, it learns a few-step initialization using online teacher supervision between adjacent timesteps, reducing the need to precompute and store complete ODE trajectories. Fewer sampling steps and finer output granularity together reduce the wait during interaction.

Initialization improves the model's starting point, but generating longer videos also requires addressing the gap between training and inference duration. Training windows are limited because directly extending them increases the number of spatiotemporal tokens and intermediate states, raising memory use and computational cost. The reliable supervision range of existing short-video teachers is also limited by the temporal span of their training windows. Actual interactions, however, may last minutes or even hours, far beyond the time span that training can directly cover.

How can short-window supervision train a longer generation process? Through streaming long tuning, LongLive 4 lets the model continue segment by segment along its own generated history, supervising each segment with a short-video teacher while using short-window attention and a frame sink to control cost and retain historical anchors. Self-Forcing++ 5 samples clips from long videos generated by the model for distillation, allowing supervision from a short-video teacher to cover longer generation trajectories. Both methods break a long process into local segments that the teacher can evaluate, applying supervision over a limited range to longer self-generated trajectories. However, longer autoregressive unrolling during training also adds training cost, and segment-wise supervision must handle complex boundary issues in temporal VAE encoding and decoding.

How can limited context support longer video inference? As inference duration increases, the model cannot indefinitely retain and read its entire history. It must select useful information within a limited cache budget. Deep Forcing 6 retains early history as an anchor and selects historical KV according to attention from recent predictions, reducing redundant content so that a limited cache can better support subsequent generation. Infinity-RoPE 14 instead addresses temporal positional encoding in an already trained model. Without additional training, it organizes history and current content through a local coordinate system that advances with generation, preventing temporal coordinates from growing beyond the model's original range. These methods improve the use of context during long-horizon inference through history selection and positional encoding, respectively.

These directions act on different parts of the process: initialization determines the student's starting point, long-horizon training extends the generation trajectories the model has experienced, history selection determines what information is retained at each step, and positional encoding organizes the temporal relationship between that information and the current prediction.

Letting Future Losses Supervise Historical Representation Learning

The preceding work improves autoregressive video generation through initialization, long-horizon training, history selection, and positional encoding. Yet noticeable error accumulation, including visual drift and collapse, still appears when inference extends beyond the training window. SGF locates one contributing issue in the learning of historical representations: under SF's KV gradient truncation, the model learns how to use history, while history writing lacks direct supervision from future losses.

Historical frames need to be encoded into KV representations before subsequent generation can read them. But the fact that history participates in prediction does not mean that constructing history is also supervised. If historical KV is treated as a fixed condition, later losses can guide the model in how to use it, but cannot follow that path to guide how it is constructed.

To make this distinction clear, we can divide the model's computation into two parts: reading historical KV to denoise the current content, and writing generated frames into new KV for subsequent prediction.

Denoting the existing historical cache as \(\mathrm{KV}_{<i}\), the two computations can be written as:

The two lines use the same parameters \(\theta\) but serve different roles: the first uses history to denoise the current content at \(t>0\), while the second writes clean frames into historical KV at \(t_{\mathrm{ctx}}=0\). Historical KV connects the two computations, and whether gradients pass through the second line determines whether future losses can train history writing.

In Self Forcing, gradients through this writing path are truncated, and historical KV is treated as a fixed condition for subsequent prediction. Stopping gradients through history does not mean that the writer remains unchanged. Taking a single SGD update as an example, for the same historical input, the output of writing at \(t=0\) will generally change with the shared parameters:

The writing capability inherited from initialization may therefore drift or deteriorate, while the truncated KV path prevents future losses from directly correcting this change.

Self Gradient Forcing (SGF) 3 restores this path, allowing future losses to supervise the construction of historical representations. It both helps the writer adapt to self-generated history and supplies feedback for its continually changing writing capability. The difference from SF can be expressed directly through the placement of stop-gradient:

\(\operatorname{sg}\) denotes stop-gradient. SF places the entire KV construction process within its scope; SGF fixes only the already generated frames, allowing the computation that constructs KV to keep learning. The same history can thus be encoded into more useful representations according to the needs of subsequent prediction.

To implement this gradient path, SGF divides training into two passes. The first performs an autoregressive rollout without gradients, recording generated frames and the inputs at selected exit steps. The second reconstructs historical KV and current predictions in parallel through a causal mask, matching the rollout's attention and positional relationships. The generated frames and sampling trajectory remain detached, while KV construction participates in backpropagation. This preserves the history the model actually encounters while avoiding the need to retain a backward graph for the complete sequential sampling process.

Within a short window, SGF learns to construct historical representations that help future prediction, then repeatedly uses this writing mechanism during longer inference. The paper compares SF and SGF with the same initialization, five-second training window, and cache protocol: short-video performance is broadly similar, while most quality and consistency metrics improve at 60 and 240 seconds. These controlled experiments show that more fully training the construction of historical representations can significantly improve long-horizon extrapolation, without extending the training window, adding training data, introducing an additional KV cache design, or adding a new loss function.

Restoring gradients through history writing improves the consistency of long video generation with SGF, but another observation emerges: in a small number of examples, SGF's first frame is slightly blurrier than SF's.

The first frame has no prior video KV to read, so its quality change provides a clue: could the gradients that learn historical representations interfere with those for denoising the current frame?

The two computations in Eq. (2) share parameters. Once gradients through history writing are restored, parameter updates therefore receive contributions from both context writing and current denoising. Denote the contributions of the same generation loss along these two paths as \(g_{\mathrm{context}}\) and \(g_{\mathrm{denoise}}\). Together they form the gradient of the shared parameters, and their directional relationship can be measured by cosine similarity:

When both gradients are nonzero and \(s<0\), their directions conflict: an update favorable to one role may be unfavorable to the other. When added together, their opposing components partially cancel.

Motivated by this observation, Self Gradient Forcing Plus (SGF+) 1 measures the gradient contributions of the two roles separately in Wan2.1 15 and computes their paired cosine similarities. Across 128 prompts and four exit steps, the examined Attention and FFN weights each yield 512 gradient pairs, all with negative cosine similarity. This shows a conflict between history writing and current denoising in their updates to shared parameters. The two contributions partially cancel when added, making it difficult to optimize both roles fully at the same time.

We further conducted supplementary measurements on LTX-2.5 with 22B parameters 16. The heatmap below shows that the mean paired gradient cosine remains negative for most combinations of layer and noise level, with conflict strength varying across layers and noise levels. This shows that gradient conflict between writing and denoising persists in a larger model.

There is a deeper point here: the same objective does not require both roles to change in the same way. In a local analysis with a fixed replay, if \(g_{\mathrm{context}}=-g_{\mathrm{denoise}}\ne0\), the shared gradient cancels, yet each role still has a descent direction of its own. Shared weights constrain this possibility of adapting separately.

Based on this observation, SGF+ separates the shared model in Eq. (2) into a context-writing model \(\mathcal C_{\theta_c}\) and a denoising model \(\mathcal D_{\theta_d}\). They are responsible for constructing historical representations and generating the current frames, respectively:

Compared with Eq. (2), the two computations no longer share parameters. As in SGF in Eq. (3), generated frames remain detached, while the construction of historical KV remains learnable.

Parameter separation does not disconnect the two roles. The loss directly updates the denoising model and, through its dependence on historical KV, continues to supervise the context-writing model. The two are still trained jointly with the same generation objective, but can optimize their respective parameters separately.

In both framewise and chunkwise settings, SGF+ further improves most visual quality and long-horizon consistency metrics. Trained with a five-second window, it also demonstrates four 24-hour continuous generation examples, without long-video fine-tuning. Systematic quantitative evaluation covers 60 and 240 seconds, while the 24-hour examples demonstrate a longer span of continuous operation. Parameter separation increases weight storage, but the two types of model call already exist; under the paper's 81-frame workload, inference takes 4.962 seconds for SGF and 4.969 seconds for SGF+.

From SGF to SGF+, supervision of historical representations and the division of roles across parameters are progressively incorporated into training. SGF allows future losses to train history writing; SGF+ further addresses the conflict between this learning and current denoising. Both designs center on the same question: how can a model leave behind a past that is more useful for future generation?

This line of research concerns two fundamental choices in training: which computations a loss can train, and whether different roles must share the same parameters. SGF and SGF+ retain self-generated history, the video generation objective, and the KV read/write interface, while changing the gradient paths and parameter-sharing constraints, respectively, so that existing task supervision can act more fully on the generation process. Longer training lets the model experience more states; these two changes help it learn more useful historical representations from the same experience. They can also work together with initialization, long-horizon training, and cache design.

Some Reflections

History writing and current denoising are closely connected through KV, but this cooperation does not require them to use exactly the same weights. Separating the roles raises three questions worth exploring further.

Connections to GPT and LLMs. Both are forms of autoregressive generation, so what computational differences make it worth reconsidering parameter sharing in video models? Language understanding and generation have been widely modeled within a unified token-prediction framework, while many questions remain open about unifying visual understanding and generation. Computationally, GPT constructs KV and predicts the next token in one forward pass, whereas autoregressive video diffusion alternates between context writing at t=0 and denoising at t>0. Context writing needs to extract historical information for future prediction, while denoising needs to recover the current frame from a noisy input. These differences in input states and computational roles may place different, even conflicting, optimization demands on shared parameters.

Parameter separation can also begin earlier. Differences and negative alignment between context-writing and denoising gradients can already be observed in the TF-initialized model, indicating that the conflict does not first appear during SGF training. It is therefore worth testing whether moving role separation into the TF stage helps, so that the model learns separately from initialization onward how to construct historical representations and how to use history for prediction. This idea can also extend to World Action Models. World Action Models trained with Teacher Forcing provide an application setting for this direction. Separating history writing from visual and action prediction may help the model learn historical representations that serve subsequent prediction more effectively.

Lightweight writers, independent memory, and KV compression. Role separation allows model capacity to be allocated as needed, with networks of different sizes for context writing and denoising. We can further explore independent memory designs around the writer: how to organize, write, and update history, for example by retaining recent details and long-term information at different levels, or by selectively writing key events. This memory would then participate in subsequent generation through a KV interface compatible with the denoiser.

One direction worth exploring is a lightweight writer that compresses redundant video history into KV representations for subsequent prediction within a limited cache budget. During training, the denoiser's parameters could be fixed, preserving how it reads historical KV, while future generation losses optimize the learnable writing and compression modules. The aim is to reduce parameter storage, KV cache use, and attention cost while preserving generation quality and long-horizon consistency.

Toward Infinite Video Extrapolation

Users do not end an interaction just because a training clip would have ended. Ideally, streaming video generation should let users watch and adjust the video as it unfolds, with generation continuing alongside the interaction.

SGF and SGF+ explore a path from short-window training to long-horizon extrapolation by learning historical representations. Two questions remain worth pursuing: How far can generation extend beyond a finite training window? And what should a finite memory retain for the future?

“Det er ganske sandt, hvad Philosophien siger, at Livet maa forstaaes baglænds. Men derover glemmer man den anden Sætning, at det maa leves forlænds.”

EN

Life must be understood backwards, yet lived forwards.

References

Cite

@article{su2026sgf+,
  title={SGF+: Decoupling Gradient Flows for Autoregressive Video Generation},
  author={Su, Zihan and Zhuang, Junhao and Li, Yaowei and Lu, Siwen and Li, Haoran and Li, Lingen and Wu, Haoyu and Jin, Weiyang and Zhang, Songchun and Huang, Haoyang and others},
  journal={arXiv preprint arXiv:2610.10429},
  year={2026}
}

@article{zhuang2026self,
  title   = {Self Gradient Forcing: Native Long Video Extrapolation},
  author  = {Zhuang, Junhao and Zhang, Shiyi and Bian, Yuxuan
             and Li, Yaowei and Luo, Yawen and Liu, Yijun
             and Jin, Weiyang and Zhang, Songchun and He, Xianglong
             and Zhang, Xuying and others},
  journal = {arXiv preprint arXiv:2607.20368},
  year    = {2026}
}