Lightricks

/

LTX-2.5-Diffusers

like 33

Follow

LTX.io 4.68k

This PR lets the LTX-2.5 modular pipeline automatically load the prompt_enhancer and processor components from the google/gemma-4-E2B-it repo without needing to include its weights in the Lightricks/LTX-2.5-Diffusers repo, by setting those components' pretrained_model_name_or_path config in modular_model_index.jsonto point to the Gemma repo.

You can test the changes as follows:

import torch
from diffusers import ComponentsManager, ModularPipeline
from diffusers.models.autoencoders.ltx2_diffusion_decoder import LTX2VideoVaeNeighborhoodNattenProcessor
from diffusers.pipelines.ltx2.utils import DEFAULT_NEGATIVE_PROMPT
from diffusers.utils import encode_video

revision = "refs/pr/14"

cm = ComponentsManager()
pipe = ModularPipeline.from_pretrained("Lightricks/LTX-2.5-Diffusers", revision=revision, components_manager=cm)
pipe.load_components(
    revision={"default": revision, "prompt_enhancer": None, "processor": None},
    dtype=torch.bfloat16,
)
cm.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="20GB")
pipe.diffusion_decoder.set_attn_processor(LTX2VideoVaeNeighborhoodNattenProcessor())
pipe.diffusion_decoder.enable_tiling()

frame_rate = 24.0
prompt = (
    "A cinematic shot of a red fox walking through a snowy forest at dawn, golden light filtering through pine trees."
)

output_state = pipe(
    prompt=prompt,
    negative_prompt=DEFAULT_NEGATIVE_PROMPT,
    width=768,
    height=512,
    num_frames=None,  # Set to an int (e.g. 121) to specify a fixed video length
    frame_rate=frame_rate,
    num_inference_steps=30,
    use_cross_timestep=True,
    enable_prompt_enhancement=True,
    generator=torch.Generator("cuda").manual_seed(42),
    output_type="np",
)
video = output_state.get("videos")
audio = output_state.get("audio")

encode_video(
    video[0],
    fps=frame_rate,
    audio=audio[0].float().cpu(),
    audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
    output_path="ltx2_5_modular_t2v.mp4",
)

The changes in this PR also fill in the library and class names for components besides the prompt_enhancer and processor, which shouldn't affect loading or inference.