Rajat Modi, Sebastian Noel, Xin Liang, Yogesh Rawat

@article{modi2026ffn,
  title={Forget, Anticipate and Adapt: Test Time Training for Long Videos},
  author={Modi, Rajat and Noel, Sebastian and Liang, Xin and Rawat, Yogesh S.},
  journal={arXiv preprint arXiv:2606.26515},
  year={2026}
}

@inproceedings{modi2026ffneccv,
  title={Forget, Anticipate and Adapt: Test Time Training for Long Videos},
  author={Modi, Rajat and Noel, Sebastian and Liang, Xin and Rawat, Yogesh S.},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}

Hi folks. This one is a bit special because it’s a paper i actually got to write, along with my labmates Sebastian Noel, Xin Liang, and my advisor Yogesh Rawat, at CRCV, UCF. It’s called `Forget, Anticipate and Adapt: Test Time Training for Long Videos’, or FFN for short (yeah I know, unfortunate acronym overlap with feed-forward-network, we couldn’t resist). Paper is here, code is here. Let’s get into it.

What even is Test Time Training

Usually, when we train a neural net, we freeze the weights once training is done. At test time, the model just does inference, no more learning happens. Test Time Training (TTT) says: why stop there? Why not let the model keep updating itself even while it’s being tested, using some self-supervised task that doesn’t need ground truth labels?

The classic example is: the model reconstructs the input (or predicts some transformed version of it), computes a loss against the real thing, and takes a gradient step. No labels needed, since the `label’ here is just the input itself.

This is neat for images. But we got curious: what happens when you try to do this for videos', and specifically, long’ videos, like hours long. Turns out, it breaks in a pretty fundamental way.

The problem with sliding windows

Existing methods that do TTT on video rely on a `sliding window’. You keep the last $k$ frames in a buffer, and every time a new frame comes in, you redo TTT on the whole window. So window $W_t$ has frames $[x_{t-k},…,x_t]$, and the next window $W_{t+1}$ has $[x_{t-k+1},…,x_{t+1}]$.

Look closely at those two windows. They share almost all' their frames. Only one frame leaves ($x_{t-k}$), and one frame enters ($x_{t+1}$). But the existing approach doesn't care, it just backprops through the entire window, every single timestep’. For a 2 hour video (~7200 frames), this literally takes 8 hours to process. That is absurd for anything that wants to run in real time, say on a drone, or a phone.

There’s a second, sillier issue: the model does this expensive TTT step even when two consecutive frames are basically identical (imagine a static CCTV feed). All that compute, for zero new information.

Our idea: just forget, anticipate, adapt

Here’s the key realization we had: since only one frame enters and one frame exits the window at each step, why not build a mechanism that `only’ touches those frames, instead of redoing the whole window? Just 3 frames: the one leaving, the current one, and the one we anticipate coming next.

We called this the Frame Forgetting Network (FFN). It has two components.

Memory Restoration Mechanism (MRM): when a frame $x_{t-k}$ exits the window, we want the model to forget' whatever adaptation it did on that frame, essentially restore the features to what they were before’ TTT touched them. We do this with a small temporal module (just a 3-layer MLP, conditioned on a positional encoding of time, similar to what transformers use for position). It predicts what the pre-adaptation features $f_{t-k-1}(x_{t-k})$ looked like, and we take a single gradient step to pull the model back toward them:

Adaptive Window Algorithm (AWA): this is the anticipate and adapt' part. Given the current frame $x_t$, the model predicts what the next frame $x'_{t+1}$ will look like. Then we compare that prediction against the actual $x_{t+1}$. If they're really different, the model has been surprised’, and it decides to adapt. If not, it just does plain inference and moves on.

To make this rigorous, we defined a `surprise metric’. First, a pixel-space visual difference term:

But raw pixel difference alone is a bit dumb: if the camera just shakes slightly, or the object rotates a tiny bit, pixel difference shoots up even though nothing meaningful changed. So we also compute a latent-space agreement term $A$, which is the cosine similarity between the current frame’s features and the features of the most recently `adapted’ frame. Higher layers of the network tend to stay invariant to small nuisance changes, so this term helps filter out the noise. We combine both into surprise:

Now, instead of hardcoding some threshold, we let it float dynamically, one standard deviation above the running mean of recent surprise values:

If $S_t \geq \tau_t$, adapt. Otherwise, just infer and move on. Simple as that.

Principle of locality

There’s a nice intuition underlying all this: frames far apart in time are often just irrelevant to each other. In the figure above, the first few frames are outdoors, the last few are indoors. There’s no reason to force the model to adapt using both, they belong to entirely different `contexts’. Our surprise-gated adaptation naturally figures this out on its own, without us telling it explicitly where the boundary is.

EpicTours: our new dataset

One annoying thing we ran into while doing this project: most existing video benchmarks for this kind of thing are tiny', like 5 minutes long, max. That's nowhere close to testing whether something works for hours’ long video.

So we built our own dataset, EpicTours: walking city-tour videos, up to 3 hours long, spanning different cities across the globe. We densely annotated them (semantic + instance level, 30 COCO-subset classes) using SAM 3 as an initial pass, followed by manual expert refinement.

These videos are real, messy, low-resolution, often phone-captured footage, with lots of people moving around. Felt like a good stress-test for something meant to eventually run on, say, a drone doing disaster-relief work, or a delivery robot.

Does it actually work?

We ran FFN across 11 datasets: dense segmentation (COCO-Videos, KITTI-STEP), depth estimation (6 datasets, following the Video-Depth-Anything setup), action classification (UCF101, Something-Something v2), and our own EpicTours.

A few numbers, roughly:

• COCO-Videos instance segmentation: 45.1 AP, about +7.5 over the TTT-Online baseline.
• COCO-Videos panoptic segmentation: 29.6 PQ, about +7.9.
• KITTI-STEP: +1.9 on val, +5.2 on test.
• UCF101 / Something-Something v2: +0.8 / +0.7 top-1.
• EpicTours: +6.5% semantic, +5.3% instance segmentation.
• Per-frame latency: 0.7 seconds for FFN, vs 4.1 seconds for TTT-Online. That’s roughly 6x cheaper per step.

Interestingly, our streaming FFN even edges out the offline TTT-MAE oracle, which gets to see the `entire’ video ahead of time. We take this as some support for the locality argument: seeing more of the past isn’t automatically better, if most of that past isn’t relevant.

The plots above (from our ablations) show a few nice things: (i) buffer size helps up to about 50 frames, then performance drops, so keeping everything' around is actually bad, the model's capacity is finite. (ii) FFN barely needs more than 1 TTT iteration, whereas TTT-Online needs many more to reach its peak. (iii) predicting the next’ frame works better than reconstructing the `current’ one, anticipation seems to give a genuinely useful inductive bias. (iv) most tellingly, on the rightmost plot, FFN stays roughly stable across 3 hour long videos, while TTT-Online degrades sharply past the 50 minute mark. This last one is really the whole point of the paper.

What’s next

We’re pretty aware this isn’t the end of the story. TTT still relies on backprop, which means there’s a `layer-lock’ problem, early layers wait around for gradients from later ones, wasting compute cycles. There might be nicer local-learning alternatives (forward-forward, target propagation, no-prop) that avoid this, though none of them beat plain backprop yet at scale.

Also, our anticipative head just predicts one step ahead. There’s probably a more general story here about meta-learning the self-supervised task itself, rather than hand-picking `predict the next frame’. Some very recent work (TTT-MLP) has started poking at this.

If you’re into TTT, long videos, or streaming perception in general, would love to hear your thoughts. And obviously, go read the actual paper, i’ve butchered a fair bit of nuance trying to keep this post short-ish.

And unfortunately, i could not travel to eccv this year, but i will remember you in my dreams. If you still want to see me, do shoot me an email at rajatmodi62@gmail.com. I promish to reply :-).

Till we meet next,
love,
rajat p.s. thku thku xin for all beautiful figures.