From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuning
Paper
•
2504.16080
•
Published
•
14
ControlNet 1.1 Preview
AI & ML interests
None defined yet.
Recent Activity
ControlNet-1-1-preview's activity
sayakpaul
authored
a
paper
5 days ago
pcuenq
authored
a
paper
20 days ago
sayakpaul
authored
a
paper
about 2 months ago
Post
3645
Inference-time scaling meets Flux.1-Dev (and others) 🔥
Presenting a simple re-implementation of "Inference-time scaling diffusion models beyond denoising steps" by Ma et al.
I did the simplest random search strategy, but results can potentially be improved with better-guided search methods.
Supports Gemini 2 Flash & Qwen2.5 as verifiers for "LLMGrading" 🤗
The steps are simple:
For each round:
1> Starting by sampling 2 starting noises with different seeds.
2> Score the generations w.r.t a metric.
3> Obtain the best generation from the current round.
If you have more compute budget, go to the next search round. Scale the noise pool (
This constitutes the random search method as done in the paper by Google DeepMind.
Code, more results, and a bunch of other stuff are in the repository. Check it out here: https://github.com/sayakpaul/tt-scale-flux/ 🤗
Presenting a simple re-implementation of "Inference-time scaling diffusion models beyond denoising steps" by Ma et al.
I did the simplest random search strategy, but results can potentially be improved with better-guided search methods.
Supports Gemini 2 Flash & Qwen2.5 as verifiers for "LLMGrading" 🤗
The steps are simple:
For each round:
1> Starting by sampling 2 starting noises with different seeds.
2> Score the generations w.r.t a metric.
3> Obtain the best generation from the current round.
If you have more compute budget, go to the next search round. Scale the noise pool (
2 ** search_round) and repeat 1 - 3.This constitutes the random search method as done in the paper by Google DeepMind.
Code, more results, and a bunch of other stuff are in the repository. Check it out here: https://github.com/sayakpaul/tt-scale-flux/ 🤗
Post
2076
We have been cooking a couple of fine-tuning runs on CogVideoX with finetrainers, smol datasets, and LoRA to generate cool video effects like crushing, dissolving, etc.
We are also releasing a LoRA extraction utility from a fully fine-tuned checkpoint. I know that kind of stuff has existed since eternity, but the quality on video models was nothing short of spectacular. Below are some links:
* Models and datasets:
finetrainers
* finetrainers: https://github.com/a-r-r-o-w/finetrainers
* LoRA extraction: https://github.com/huggingface/diffusers/blob/main/scripts/extract_lora_from_model.py
We are also releasing a LoRA extraction utility from a fully fine-tuned checkpoint. I know that kind of stuff has existed since eternity, but the quality on video models was nothing short of spectacular. Below are some links:
* Models and datasets:
finetrainers
* finetrainers: https://github.com/a-r-r-o-w/finetrainers
* LoRA extraction: https://github.com/huggingface/diffusers/blob/main/scripts/extract_lora_from_model.py
- 1 reply
Post
2026
We have authored a post to go over the state of video generation in the Diffusers ecosystem 🧨
We cover the models supported, the knobs of optims our users can fire, fine-tuning, and more 🔥
5-6GBs for HunyuanVideo, sky is the limit 🌌 🤗
https://huggingface.co/blog/video_gen
We cover the models supported, the knobs of optims our users can fire, fine-tuning, and more 🔥
5-6GBs for HunyuanVideo, sky is the limit 🌌 🤗
https://huggingface.co/blog/video_gen
Post
4429
Commits speak louder than words 🤪
* 4 new video models
* Multiple image models, including SANA & Flux Control
* New quantizers -> GGUF & TorchAO
* New training scripts
Enjoy this holiday-special Diffusers release 🤗
Notes: https://github.com/huggingface/diffusers/releases/tag/v0.32.0
* 4 new video models
* Multiple image models, including SANA & Flux Control
* New quantizers -> GGUF & TorchAO
* New training scripts
Enjoy this holiday-special Diffusers release 🤗
Notes: https://github.com/huggingface/diffusers/releases/tag/v0.32.0
Post
17625
Google drops Gemini 2.0 Flash Thinking
a new experimental model that unlocks stronger reasoning capabilities and shows its thoughts. The model plans (with thoughts visible), can solve complex problems with Flash speeds, and more
now available in anychat, try it out: https://huggingface.co/spaces/akhaliq/anychat
a new experimental model that unlocks stronger reasoning capabilities and shows its thoughts. The model plans (with thoughts visible), can solve complex problems with Flash speeds, and more
now available in anychat, try it out: https://huggingface.co/spaces/akhaliq/anychat
- 4 replies
Post
2248
In the past seven days, the Diffusers team has shipped:
1. Two new video models
2. One new image model
3. Two new quantization backends
4. Three new fine-tuning scripts
5. Multiple fixes and library QoL improvements
Coffee on me if someone can guess 1 - 4 correctly.
1. Two new video models
2. One new image model
3. Two new quantization backends
4. Three new fine-tuning scripts
5. Multiple fixes and library QoL improvements
Coffee on me if someone can guess 1 - 4 correctly.
- 1 reply
Post
2179
Introducing a high-quality open-preference dataset to further this line of research for image generation.
Despite being such an inseparable component for modern image generation, open preference datasets are a rarity!
So, we decided to work on one with the community!
Check it out here:
https://huggingface.co/blog/image-preferences
Despite being such an inseparable component for modern image generation, open preference datasets are a rarity!
So, we decided to work on one with the community!
Check it out here:
https://huggingface.co/blog/image-preferences
- 7 replies
Post
2205
The Control family of Flux from
@black-forest-labs
should be discussed more!
It enables structural controls like ControlNets while being significantly less expensive to run!
So, we're working on a Control LoRA training script 🤗
It's still WIP, so go easy:
https://github.com/huggingface/diffusers/pull/10130
It enables structural controls like ControlNets while being significantly less expensive to run!
So, we're working on a Control LoRA training script 🤗
It's still WIP, so go easy:
https://github.com/huggingface/diffusers/pull/10130
sayakpaul
authored
a
paper
5 months ago
Post
1563
Let 2024 be the year of video model fine-tunes!
Check it out here:
https://github.com/a-r-r-o-w/cogvideox-factory/tree/main/training/mochi-1
Check it out here:
https://github.com/a-r-r-o-w/cogvideox-factory/tree/main/training/mochi-1
Post
17644
QwQ-32B-Preview is now available in anychat
A reasoning model that is competitive with OpenAI o1-mini and o1-preview
try it out: https://huggingface.co/spaces/akhaliq/anychat
A reasoning model that is competitive with OpenAI o1-mini and o1-preview
try it out: https://huggingface.co/spaces/akhaliq/anychat
- 1 reply
Post
4459
New model drop in anychat
allenai/Llama-3.1-Tulu-3-8B is now available
try it here: https://huggingface.co/spaces/akhaliq/anychat
allenai/Llama-3.1-Tulu-3-8B is now available
try it here: https://huggingface.co/spaces/akhaliq/anychat
Post
3333
anychat
supports chatgpt, gemini, perplexity, claude, meta llama, grok all in one app
try it out there: https://huggingface.co/spaces/akhaliq/anychat
supports chatgpt, gemini, perplexity, claude, meta llama, grok all in one app
try it out there: https://huggingface.co/spaces/akhaliq/anychat
Post
2720
It's been a while we shipped native quantization support in
We currently support
This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
diffusers 🧨We currently support
bistandbytes as the official backend but using others like torchao is already very simple. This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
enable_model_cpu_offload()5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
- 1 reply
Post
2804
Did some little experimentation to resize pre-trained LoRAs on Flux. I explored two themes:
* Decrease the rank of a LoRA
* Increase the rank of a LoRA
The first one is helpful in reducing memory requirements if the LoRA is of a high rank, while the second one is merely an experiment. Another implication of this study is in the unification of LoRA ranks when you would like to
Check it out here:
sayakpaul/flux-lora-resizing
* Decrease the rank of a LoRA
* Increase the rank of a LoRA
The first one is helpful in reducing memory requirements if the LoRA is of a high rank, while the second one is merely an experiment. Another implication of this study is in the unification of LoRA ranks when you would like to
torch.compile() them. Check it out here:
sayakpaul/flux-lora-resizing
- 1 reply
sayakpaul
authored
a
paper
8 months ago
Post
2984
Here is a hackable and minimal implementation showing how to perform distributed text-to-image generation with Diffusers and Accelerate.
Full snippet is here: https://gist.github.com/sayakpaul/cfaebd221820d7b43fae638b4dfa01ba
With @JW17
Full snippet is here: https://gist.github.com/sayakpaul/cfaebd221820d7b43fae638b4dfa01ba
With @JW17