Papers
arxiv:2504.05298

One-Minute Video Generation with Test-Time Training

Published on Apr 7
· Submitted by zhaoyue-zephyrus on Apr 8
#2 Paper of the day
Authors:
,
,
,
,
,

Abstract

Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with complex multi-scene stories because their hidden states are less expressive. We experiment with Test-Time Training (TTT) layers, whose hidden states themselves can be neural networks, therefore more expressive. Adding TTT layers into a pre-trained Transformer enables it to generate one-minute videos from text storyboards. For proof of concept, we curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers generate much more coherent videos that tell complex stories, leading by 34 Elo points in a human evaluation of 100 videos per method. Although promising, results still contain artifacts, likely due to the limited capability of the pre-trained 5B model. The efficiency of our implementation can also be improved. We have only experimented with one-minute videos due to resource constraints, but the approach can be extended to longer videos and more complex stories. Sample videos, code and annotations are available at: https://test-time-training.github.io/video-dit

Community

Nice one for Test Time!

interesting!

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

A dark, moonlit ocean sparkles under a cloudy sky. A large pirate ship with tattered black sails gently rocks on the waves. The ship’s wooden deck creaks softly. Lanterns swing with the breeze, casting warm flickering light on the weathered planks. Captain Jack, a daring raccoon in a worn tricorn hat and long navy coat, grips the wheel confidently. His gold-toothed grin shines under the moonlight as he squints toward the horizon. Below deck, wooden barrels and crates line the walls. A dim lantern sways from the ceiling, casting long shadows. The crew, a mix of animal pirates — a tall parrot with an eyepatch, a burly bear with striped pants, and a sneaky ferret with a dagger — gather around a large treasure map spread on a wooden table. The parrot squawks excitedly, tapping an island marked with an X. On the main deck, the ocean becomes choppy. A rival pirate ship appears in the distance, its black flag bearing a silver skull. Thunder rumbles as clouds gather. Captain Jack lifts a brass spyglass, narrowing his eyes. “They’ve found us,” he mutters. The crew scrambles into action. Cannons are loaded, sails are tightened, and swords are drawn. As the two ships draw closer, cannonballs fly through the air. Explosions rock both ships. The raccoon captain swings on a rope across to the enemy ship, his coat fluttering like a cape. He lands with flair, drawing his curved saber. The enemy captain, a fierce hawk with a feathered coat and a golden hook, squawks a battle cry as they clash swords dramatically. Back on Jack’s ship, Jerry the brown mouse — dressed in a tiny pirate outfit — sneaks quietly past a distracted guard raccoon, dragging a small chest filled with stolen gold. He grins mischievously, disappearing into a mouse-sized hatch near the captain’s quarters. On the enemy ship, Captain Jack knocks away the hawk’s sword and ties him up with a rope in one slick move. Jack looks back toward his ship and gives a signal. The mouse pokes his head out of the hatch, gives a thumbs-up, and the crew erupts in cheers. The storm begins to clear. As the sun rises, Jack’s ship sails away from the burning wreck of the enemy vessel. Seagulls cry above. Jack stands on the bow, wind blowing through his fur. Jerry climbs up to his shoulder and hands him a gold coin. They both laugh victoriously as the camera pans up to the sky and the scene fades to black.
A dark, moonlit ocean sparkles under a cloudy sky. A large pirate ship with tattered black sails gently rocks on the waves. The ship’s wooden deck creaks softly. Lanterns swing with the breeze, casting warm flickering light on the weathered planks. Captain Jack, a daring raccoon in a worn tricorn hat and long navy coat, grips the wheel confidently. His gold-toothed grin shines under the moonlight as he squints toward the horizon. Below deck, wooden barrels and crates line the walls. A dim lantern sways from the ceiling, casting long shadows. The crew, a mix of animal pirates — a tall parrot with an eyepatch, a burly bear with striped pants, and a sneaky ferret with a dagger — gather around a large treasure map spread on a wooden table. The parrot squawks excitedly, tapping an island marked with an X. On the main deck, the ocean becomes choppy. A rival pirate ship appears in the distance, its black flag bearing a silver skull. Thunder rumbles as clouds gather. Captain Jack lifts a brass spyglass, narrowing his eyes. “They’ve found us,” he mutters. The crew scrambles into action. Cannons are loaded, sails are tightened, and swords are drawn. As the two ships draw closer, cannonballs fly through the air. Explosions rock both ships. The raccoon captain swings on a rope across to the enemy ship, his coat fluttering like a cape. He lands with flair, drawing his curved saber. The enemy captain, a fierce hawk with a feathered coat and a golden hook, squawks a battle cry as they clash swords dramatically. Back on Jack’s ship, Jerry the brown mouse — dressed in a tiny pirate outfit — sneaks quietly past a distracted guard raccoon, dragging a small chest filled with stolen gold. He grins mischievously, disappearing into a mouse-sized hatch near the captain’s quarters. On the enemy ship, Captain Jack knocks away the hawk’s sword and ties him up with a rope in one slick move. Jack looks back toward his ship and gives a signal. The mouse pokes his head out of the hatch, gives a thumbs-up, and the crew erupts in cheers. The storm begins to clear. As the sun rises, Jack’s ship sails away from the burning wreck of the enemy vessel. Seagulls cry above. Jack stands on the bow, wind blowing through his fur. Jerry climbs up to his shoulder and hands him a gold coin. They both laugh victoriously as the camera pans up to the sky and the scene fades to black.

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2504.05298 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2504.05298 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2504.05298 in a Space README.md to link it from this page.

Collections including this paper 10