Package {diffuseR}

Title:	Functional Interface to Diffusion Models in R
Version:	0.2.2
Description:	A native R implementation of diffusion models providing a functional interface to state-of-the-art generative AI. Inspired by the 'Python' library 'diffusers' from 'Hugging Face' https://huggingface.co/, 'diffuseR' generates and manipulates images from text prompts using models such as 'Stable Diffusion', with no 'Python' dependency. Supports multiple diffusion schedulers and device acceleration.
License:	Apache License (≥ 2)
URL:	https://github.com/cornball-ai/diffuseR
BugReports:	https://github.com/cornball-ai/diffuseR/issues
Encoding:	UTF-8
Imports:	torch, jsonlite, grid, png, jpeg
Suggests:	av, hfhub, safetensors, simplermarkdown, tinytest
VignetteBuilder:	simplermarkdown
RoxygenNote:	7.3.3
NeedsCompilation:	no
Packaged:	2026-08-03 06:24:31 UTC; troy
Author:	Troy Hernandez [aut, cre], cornball.ai [cph], The HuggingFace Team [cph] (portions ported from the diffusers library (Apache-2.0); see inst/COPYRIGHTS), Lightricks Ltd. [cph] (LTX checkpoint layout and pipeline constants; see inst/COPYRIGHTS)
Maintainer:	Troy Hernandez <troy@cornball.ai>
Repository:	CRAN
Date/Publication:	2026-08-08 12:40:02 UTC

Build fallback device configuration

Description

Build fallback device configuration

Usage

.build_fallback_devices(model, strategy)

Arguments

Value

Named list of device assignments.

Detect Available VRAM

Description

Asks nvidia-smi.

Usage

.detect_vram(use_free = FALSE)

Arguments

Value

Numeric. VRAM in GB, or 0 if no GPU detected.

Check if FFmpeg is Available

Description

Check if FFmpeg is Available

Usage

.ffmpeg_available()

Value

Logical. TRUE if ffmpeg is in PATH.

Pack a transformer block's weights for the JIT stack

Description

Returns the block's tensors in the fixed 114-slot layout consumed by the compiled stack_nf4/block_nf4 TorchScript functions. Tensor handles are borrowed, not copied.

Usage

.ltx23_jit_pack_block(block)

Arguments

Value

List of 114 tensors.

Run the block stack through the compiled TorchScript path

Description

One R-to-libtorch crossing for all blocks: no per-op dispatch, no R tensor garbage, fused SDPA. Masks must already be additive [B, 1, 1, S] (or NULL); rope tensors are the [.., r] cos/sin pairs used by the eager path.

Usage

.ltx23_jit_run_stack(blocks, hidden_states, audio_hidden_states,
                     encoder_hidden_states, audio_encoder_hidden_states, temb,
                     temb_audio, temb_ca_scale_shift,
                     temb_ca_audio_scale_shift, temb_ca_gate,
                     temb_ca_audio_gate, temb_prompt, temb_prompt_audio,
                     video_rotary_emb, audio_rotary_emb, ca_video_rotary_emb,
                     ca_audio_rotary_emb, encoder_attention_mask = NULL,
                     audio_encoder_attention_mask = NULL,
                     cond_token_index = NULL)

Value

list(hidden_states, audio_hidden_states)

Release the attention scratch buffers

Description

Release the attention scratch buffers

Usage

.ltx23_release_attn_buffers()

Value

Invisibly, NULL.

Release all cached decode traces

Description

Traces hold references to the weight tensors they captured; drop them when a component leaves the GPU so its memory actually frees.

Usage

.ltx23_release_vae_traces()

Value

Invisibly, NULL.

Run a module forward through a shape-specialized trace

Description

Run a module forward through a shape-specialized trace

Usage

.ltx23_traced_call(module, x, forward = NULL, tag = "")

Arguments

Value

The forward result.

Pin a component's tensors for fast phase transfer

Description

Pin a component's tensors for fast phase transfer

Usage

.pin_component(module, extra = NULL)

Arguments

Value

A list of list(live, pinned) tensor pairs, or NULL if pinning is unavailable (no CUDA, or page-locking failed).

TRUE when every staged tensor sits on the expected device type

Description

TRUE when every staged tensor sits on the expected device type

Usage

.resident_all_on(staging, type)

Arguments

Value

Logical.

Refuse a bulk activation that cannot fit

Description

Fails before the transfer rather than part-way through it. A partial onload that OOMs is recoverable (activation rolls back), but it wastes the transfer and reports a libtorch allocator error instead of the actual problem, which is that this model does not fit this card.

Usage

.resident_check_fits(res, free_gb = NULL)

Arguments

Value

Invisibly TRUE, or an error naming both figures.

Every nn_module field of a pipeline, by name

Description

Discovery beats a hard-coded list: the families disagree on which components exist (FLUX.1 has two text encoders, LTX adds a video VAE, an audio VAE and a vocoder), and a field added later is picked up without touching this file.

Usage

.resident_components(pipeline)

Arguments

Value

A named list of the pipeline's nn_module fields, possibly empty.

Refuse operations that the current state cannot serve

Description

Refuse operations that the current state cannot serve

Usage

.resident_guard(res, verb)

Arguments

Value

Invisibly TRUE, or an error.

How many components actually have their tensors on the GPU

Description

Ground truth, as opposed to the handle's declared state. The two can disagree: a pipeline built with phase_offload = TRUE swaps each component back to pinned host memory as its phase finishes, so after a render the handle is still "active" while the card holds nothing. A broker deciding who to evict needs the measurement, not the claim.

Usage

.resident_on_gpu_count(staging)

Arguments

Value

Integer. Number of components whose live tensors are on CUDA.

Pin every component of a pipeline for fast transfer

Description

Re-uses any staging the loader already built (the phase-offload path pins as part of loading), and pins the rest. Pinning a component that is currently on the GPU also evicts it, since .pin_component copies into page-locked host memory and re-points the live tensors at it, so this doubles as the initial offload.

Usage

.resident_pin(pipeline, verbose = TRUE)

Arguments

Value

A named list of staging sets, one per component that could be pinned. Components that fail to page-lock are absent, and fall back to the pageable $to() path.

Total pinned host bytes across a staging set

Description

Total pinned host bytes across a staging set

Usage

.resident_pinned_bytes(staging)

Arguments

Value

Numeric. Bytes of page-locked host memory held.

Probe whether the installed safetensors can READ a dtype

Description

Hand-builds a tiny safetensors file of the dtype (via .st_write_min, no safetensors writer involved) and tries to load it back. Cached per session; options(diffuseR.st_read_caps = list(bfloat16 = TRUE, ...)) overrides the probe for tests and for forcing a tier.

Usage

.st_can_read(dtype = c("bfloat16", "float8_e4m3fn"))

Arguments

Value

Logical.

Return a pinned component to the CPU

Description

Weights are immutable during inference, so the pinned host copies are still current: offload is a pointer swap, no transfer.

Usage

.staged_offload(staging)

Move a pinned component onto the compute device

Description

Non-blocking copies from pinned memory share the default stream, so later kernels are ordered after them; no explicit sync needed.

Usage

.staged_onload(staging, device)

Normalize a verbosity flag

Description

Normalize a verbosity flag

Usage

.verbosity(verbose)

Arguments

Value

One of "silent", "progress", "steps".

Basic Transformer Block

Description

Basic Transformer Block

Usage

BasicTransformerBlock(dim, n_heads, d_head, context_dim = NULL)

CLIP Attention Block

Description

Multi-head self-attention with separate Q/K/V projections (HuggingFace style)

Usage

CLIPAttention(embed_dim, num_heads)

Arguments

CLIP MLP Block

Description

Feed-forward network with configurable activation

Usage

CLIPMLP(in_dim, hidden_dim, gelu_type = "tanh")

Arguments

Tokenize a prompt

Description

Tokenize a prompt

Usage

CLIPTokenizer(prompt,
              merges = system.file("tokenizer/merges.txt", package = "diffuseR"),
              vocab_file = system.file("tokenizer/vocab.json", package = "diffuseR"),
              pad_token = 0L)

Arguments

Value

A 2D torch tensor of shape c(1, 77) containing the token IDs.

CLIP Transformer Block

Description

Pre-norm transformer block with attention and MLP (HuggingFace style)

Usage

CLIPTransformerBlock(embed_dim, num_heads, mlp_dim, gelu_type = "tanh")

Arguments

Downsample Block

Description

Downsample Block

Usage

Downsample2D(channels)

FeedForward Network

Description

FeedForward Network

Usage

FeedForward(dim, mult = 4L)

GEGLU Feedforward

Description

GEGLU Feedforward

Usage

GEGLU(dim_in, dim_out)

Spatial Transformer (Attention Block)

Description

Spatial Transformer (Attention Block)

Usage

SpatialTransformer(in_channels, n_heads, d_head, depth = 1L, context_dim = NULL)

Cross-Attention for UNet

Description

Cross-Attention for UNet

Usage

UNetCrossAttention(query_dim, context_dim = NULL, heads = 8L, dim_head = 64L)

ResNet Block for UNet

Description

ResNet Block for UNet

Usage

UNetResBlock(in_channels, out_channels, time_embed_dim)

Upsample Block

Description

Upsample Block

Usage

Upsample2D(channels)

VAE Attention Block

Description

Self-attention for VAE mid block

Usage

VAEAttentionBlock(channels, norm_groups = 32)

Arguments

VAE Mid Block

Description

VAE Mid Block

Usage

VAEMidBlock(channels, norm_groups = 32)

Arguments

VAE ResNet Block

Description

VAE ResNet Block

Usage

VAEResnetBlock(in_channels, out_channels, norm_groups = 32)

Arguments

VAE Up Block

Description

VAE Up Block

Usage

VAEUpBlock(in_channels, out_channels, num_resnets = 3, add_upsample = TRUE,
           norm_groups = 32)

Arguments

Apply BPE merge rules

Description

Apply BPE merge rules

Usage

apply_bpe_merges(tokens, merge_priority, vocab)

Apply rotary position embeddings

Description

Apply rotary position embeddings

Usage

apply_rotary_pos_emb(q, k, cos, sin)

Arguments

Audio Conditioning Frontend for LTX-2.3

Description

Turns user audio into the normalized, packed audio latents the joint denoiser conditions on (lip sync): decode to 16 kHz stereo PCM, log-mel via a causal STFT (filter 1024, hop 160, 64 slaney-normed mel bins to 8 kHz — the checkpoint's preprocessing spec), then the audio VAE encoder in argmax mode. The STFT and mel-filterbank constructors were verified against the checkpoint's stored vocoder bases (identical up to bf16 rounding), so the convention matches training.

LTX-2.3 Audio VAE

Description

Fresh R port of the LTX-2 audio autoencoder from the diffusers reference (Apache-2.0, autoencoder_kl_ltx2_audio.py), configured per the checkpoint: pixel norm, height-axis causality, base 128 channels with multipliers (1, 2, 4), 8 latent channels, 64 mel bins, no attention. The decoder produces mel for the vocoder; the encoder turns user audio into conditioning latents (lip sync).

Auto-Configure Device Assignment

Description

Automatically determines optimal device configuration for diffusion model components based on available VRAM (via nvidia-smi) and GPU architecture.

Usage

auto_devices(model = "sdxl", strategy = "auto")

Arguments

Details

Strategies:

"auto"

Detect free VRAM and choose the best strategy

"full_gpu"

All components on CUDA (10GB+ free VRAM for SDXL)

"unet_gpu"

Only unet on CUDA, rest on CPU (6GB+ for SDXL)

"cpu_only"

All components on CPU

On Blackwell GPUs (RTX 50xx), "unet_gpu" is forced due to TorchScript compatibility issues, regardless of available VRAM. The native modules ('use_native_unet' and friends) do not have this restriction.

Value

A named list of device assignments suitable for 'models2devices()'.

Examples

# Force a strategy: no GPU or nvidia-smi needed.
str(auto_devices("sdxl", strategy = "cpu_only"))

str(auto_devices("sd21", strategy = "unet_gpu"))

# Auto-detect free VRAM and pick a strategy for this machine.
str(auto_devices("sdxl"))

BPE Tokenizer

Description

Native R implementation of Byte-Pair Encoding tokenizer. Loads from HuggingFace tokenizer.json format.

Usage

bpe_tokenizer(tokenizer_path)

Arguments

Value

A bpe_tokenizer object.

FLUX Checkpoint Readers

Description

FLUX transformers ship in the diffusers layout: a directory with config.json, one or more diffusion_pytorch_model*.safetensors shards, and (when sharded) a diffusion_pytorch_model.safetensors.index.json weight map. These helpers open that layout behind the same checkpoint interface as ltx23_open_checkpoint, so the LTX group loaders and quantization machinery work unchanged. FLUX module names mirror the checkpoint keys 1:1 - no key mapping is needed.

LTX-2.3 Single-File Checkpoint Reader

Description

LTX 2.3 checkpoints ship as one safetensors file containing every component (transformer, connectors, video VAE, audio VAE, vocoder), with the model version and full component configuration embedded in the safetensors metadata. These helpers open the file, validate the version, split the key space by component, and stream tensors into R torch modules one at a time so the 46 GB file is never fully materialized in memory.

Clear VRAM Cache

Description

Forces garbage collection and clears CUDA memory cache.

Usage

clear_vram(verbose = FALSE)

Arguments

Value

Invisibly returns NULL.

Examples

if (torch::torch_is_installed()) {
  clear_vram()
}

Pooled CLIP output at the EOS position

Description

The HF CLIPTextModel pooler_output: the final-layer-norm hidden state at the EOS token position, located by argmax over the token ids (EOS is the highest id in the CLIP vocab, and causal attention makes any padding after it irrelevant). No text projection is applied - this is what FLUX uses as pooled_projections.

Usage

clip_pooled_output(hidden_states, input_ids)

Arguments

Value

Tensor [B, D].

LTX-2.3 Prefix Conditioning (image-to-video, video continuation)

Description

Fresh R port of the frame-conditioning mechanics from the diffusers reference (Apache-2.0, pipelines/ltx2/pipeline_ltx2_image2video.py and pipeline_ltx2_condition.py), restricted to prefix conditioning at latent index 0 with strength 1: a single start image (i2v) or the leading pixel frames of a previous clip (continuation). Conditioned latent tokens are initialized from the VAE-encoded pixels, see a per-token timestep of zero, and are frozen through the Euler loop.

LTX-2.3 Text Embedding Connectors

Description

Fresh R port of the LTX text connectors from the diffusers reference (Apache-2.0, src/diffusers/pipelines/ltx2/connectors.py). The connectors take raw stacked per-layer Gemma3 hidden states [batch, seq, caption_channels, num_layers + 1], normalize and project them per modality, replace padding with learnable registers, and run a small 1D transformer per modality to produce the DiT text embeddings.

Convert cornball SD 2.1 TorchScript weights to a diffusers artifact

Description

Rebuilds a diffusers-layout directory (unet/, vae/, text_encoder/) from the cornball-ai/sd21-R TorchScript component .pt files, so the native safetensors pipeline (download_sd21 / sd_pipeline_from_safetensors) can load SD 2.1 with no TorchScript.

Usage

convert_sd21_pt_to_diffusers(pt_dir = NULL, output_dir = NULL,
                             dtype = c("float16", "float32"), verbose = TRUE)

Arguments

Details

A TorchScript trace preserves the exact parameter tensors, so the result is bit-identical to the source at the chosen dtype. This is the provenance-clean way to build the hosted artifact: the upstream stabilityai/stable-diffusion-2-1 repo was deprecated, SD 2.1 is CreativeML OpenRAIL++-M (redistributable), and cornball already hosts these weights as .pt. At float16 the components are all sub-2 GB single files (unet ~1.7 GB, text_encoder ~0.65 GB, vae ~0.16 GB), so they load on stock CRAN safetensors.

Value

Invisibly, output_dir.

Create sliding window causal attention mask

Description

Create sliding window causal attention mask

Usage

create_sliding_window_mask(seq_len, window_size, device = "cpu")

Create a DDIM Scheduler

Description

Creates a Denoising Diffusion Implicit Models (DDIM) scheduler for use with diffusion models. DDIM schedulers provide a deterministic sampling process that offers faster inference compared to DDPM while maintaining high quality outputs.

Usage

ddim_scheduler_create(num_train_timesteps = 1000, num_inference_steps = 50,
                      eta = 0,
                      beta_schedule = c("linear", "scaled_linear", "cosine"),
                      beta_start = 0.00085, beta_end = 0.012,
                      rescale_betas_zero_snr = FALSE,
                      dtype = torch::torch_float32(),
                      device = torch::torch_device("cpu"))

Arguments

Details

DDIM (Denoising Diffusion Implicit Models) was introduced by Song et al. (2020) as an extension to DDPM (Denoising Diffusion Probabilistic Models). It offers a deterministic sampling process and allows for controlling the number of inference steps independently from the training process.

The scheduler contains the noise schedule and methods for computing alpha, beta, and other parameters used in the diffusion process.

Value

A DDIM scheduler object that can be used with diffusion models to generate samples.

References

Song, J., Meng, C., & Ermon, S. (2020). "Denoising Diffusion Implicit Models." https://arxiv.org/abs/2010.02502

Examples

if (torch::torch_is_installed()) {
  scheduler <- ddim_scheduler_create(
    num_train_timesteps = 1000,
    num_inference_steps = 5,
    eta = 0.5,
    beta_schedule = "scaled_linear"
  )
  scheduler$timesteps
}

Perform a DDIM scheduler step

Description

Performs a single denoising step using the DDIM (Denoising Diffusion Implicit Models) algorithm. This function takes the output from a diffusion model at a specific timestep and computes the previous (less noisy) sample in the diffusion process.

Usage

ddim_scheduler_step(model_output, timestep, sample, schedule, eta = 0,
                    use_clipped_model_output = FALSE, thresholding = FALSE,
                    generator = NULL, variance_noise = NULL,
                    clip_sample = FALSE, set_alpha_to_one = FALSE,
                    prediction_type = c("epsilon", "sample", "v_prediction"),
                    dtype = torch::torch_float32(), device = "cpu")

Arguments

Details

The DDIM step function implements the core sampling algorithm of DDIM described in Song et al. 2020. It computes the previous sample x_t-1 given the current sample x_t and the model output.

The algorithm differs from DDPM by using a non-Markovian diffusion process that allows for deterministic sampling and fewer inference steps without sacrificing quality.

When using 'prediction_type="epsilon"' (most common), the model predicts the noise that was added to create the current noisy sample. For 'prediction_type="sample"', the model predicts the clean sample directly. The 'v_prediction' option implements the v-parameterization from Salimans & Ho (2022).

Value

A list containing:

'prev_sample'

The less noisy sample at timestep t-1

'pred_original_sample'

The predicted denoised sample

References

Song, J., Meng, C., & Ermon, S. (2020). "Denoising Diffusion Implicit Models." https://arxiv.org/abs/2010.02502

Salimans, T., & Ho, J. (2022). "Progressive Distillation for Fast Sampling of Diffusion Models." https://arxiv.org/abs/2202.00512

Examples

if (torch::torch_is_installed()) {
  scheduler <- ddim_scheduler_create(num_inference_steps = 5)
  sample <- torch::torch_randn(c(1, 4, 8, 8))
  model_output <- torch::torch_randn(c(1, 4, 8, 8))
  result <- ddim_scheduler_step(
    model_output = model_output,
    timestep = scheduler$timesteps[1],
    sample = sample,
    schedule = scheduler,
    eta = 0, # Deterministic sampling
    prediction_type = "epsilon")
  result$shape
}

Decode token IDs to text

Description

Decode token IDs to text

Usage

decode_bpe(tokenizer, ids, skip_special_tokens = TRUE)

Arguments

Value

Character string or vector.

Detect text encoder architecture from TorchScript file

Description

Detect text encoder architecture from TorchScript file

Usage

detect_text_encoder_architecture(torchscript_path)

Arguments

Value

List with vocab_size, context_length, embed_dim, num_layers, num_heads, mlp_dim

Detect UNet architecture from TorchScript file

Description

Detect UNet architecture from TorchScript file

Usage

detect_unet_architecture(torchscript_path)

Arguments

Value

List with architecture parameters

Detect SDXL UNet architecture from TorchScript file

Description

Detect SDXL UNet architecture from TorchScript file

Usage

detect_unet_sdxl_architecture(torchscript_path)

Arguments

Value

List with architecture parameters

FLUX Transformer (MMDiT)

Description

Fresh R port of FluxTransformer2DModel from the diffusers reference implementation (Apache-2.0, src/diffusers/models/transformers/transformer_flux.py). The module tree mirrors the diffusers state-dict keys 1:1, so checkpoints load without remapping. FLUX.1-schnell has no guidance embedder (guidance_embeds = FALSE); the guidance-distilled dev variant is not implemented.

FLUX.2 Transformer (MMDiT)

Description

Fresh R port of Flux2Transformer2DModel from the diffusers reference implementation (Apache-2.0, src/diffusers/models/transformers/transformer_flux2.py). Defaults are the klein-4B configuration (5 double + 20 single blocks). Guidance embeddings (FLUX.2-dev) are not implemented; klein is step-distilled with guidance_embeds = false. Timestep conditioning has no pooled-text component, and the three modulation projections are shared across all blocks.

FLUX.2 Transformer Building Blocks

Description

Fresh R port of the FLUX.2 MMDiT blocks from the diffusers reference implementation (Apache-2.0, src/diffusers/models/transformers/transformer_flux2.py). Key differences from FLUX.1: modulation is computed ONCE at model level by shared flux2_modulation projections and passed into the blocks (block norms are parameterless), feed-forwards use SwiGLU with the gate fused into linear_in, the single-stream block is a ViT-22B-style parallel block with fully fused projections, and every linear is bias-free. Module field names mirror the diffusers state-dict keys 1:1. Reuses flux_attention (bias = FALSE), ltx23_rms_norm, .ltx23_sdpa, and flux_apply_rotary_emb.

FLUX Transformer Building Blocks

Description

Fresh R port of the FLUX MMDiT blocks from the diffusers reference implementation (Apache-2.0, src/diffusers/models/transformers/transformer_flux.py and src/diffusers/models/normalization.py). Module field names mirror the diffusers state-dict keys 1:1 so checkpoints load without remapping. Reuses the LTX primitives ltx23_rms_norm, .ltx23_sdpa and ltx23_feed_forward.

LTX-2.3 Audio-Video Diffusion Transformer

Description

Fresh R port of the LTX-2 transformer from the diffusers reference (Apache-2.0, src/diffusers/models/transformers/transformer_ltx2.py), configured for LTX 2.3: gated attention, cross-attention modulation, prompt AdaLN, split RoPE, and connector-projected text embeddings (no in-model caption projection).

LTX-2.3 Transformer Building Blocks

Description

Fresh R port of the LTX-2 transformer components from the diffusers reference (Apache-2.0, src/diffusers/models/transformers/ transformer_ltx2.py and shared modules). Field names mirror the diffusers module tree so checkpoint keys map 1:1.

Z-Image Transformer Block Modules

Description

Fresh R port of the Z-Image DiT building blocks from the diffusers reference (Apache-2.0, src/diffusers/models/transformers/transformer_z_image.py). Z-Image is a single-stream DiT: text and image tokens share one sequence and one set of block weights. Each block uses sandwich RMSNorms (a learned norm before AND after both the attention and the feed-forward) and a scale/gate-only modulation — four chunks (scale_msa, gate_msa, scale_mlp, gate_mlp), no shift, gates tanh-squashed, scales 1 + x. The attention is plain joint self-attention, so the FLUX attention module is reused with bias = FALSE and eps = 1e-5.

Download a single TorchScript model component

Description

Downloads a specific model component file (e.g., UNet, decoder, text encoder) using hfhub::hub_download() from the cornball-ai dataset repos.

Usage

download_component(model_name = "sd21", component, device = "cpu",
                   overwrite = FALSE, show_progress = TRUE)

Arguments

Value

The local file path to the downloaded component (character string).

Examples

## Not run: 
path <- download_component("sd21", "text_encoder", "cpu")

## End(Not run)

Download and Prepare FLUX.1-schnell Weights

Description

Downloads FLUX.1-schnell from HuggingFace (weights Apache-2.0, but the repo is gated behind a license click-through) and quantizes the 12B transformer to a local NF4 (~7 GB) or fp8 (~12 GB) artifact.

Download FLUX.1-schnell and build the quantized artifact

Description

Skips work already done: a valid quantized manifest short-circuits the transformer download; cached files are not re-fetched. Needs HF_TOKEN set for the gated repo (see the error message it raises without one). The bf16 transformer source (~24 GB in the HuggingFace cache) may be deleted after quantization.

Usage

download_flux1(quantize = TRUE, precision = c("nf4", "fp8"), output_dir = NULL,
               text_encoders = TRUE, verbose = TRUE)

Arguments

Value

Invisibly, a list with transformer_dir, artifact_dir, and support (named file paths).

Download and Prepare FLUX.2 Klein 4B Weights

Description

Downloads FLUX.2-klein-4B from HuggingFace (Apache-2.0, ungated) and quantizes the 4B transformer to a local fp8 (~4 GB) or NF4 (~2.3 GB) artifact.

Download FLUX.2-klein-4B and build the quantized artifact

Description

Skips work already done: a valid quantized manifest short-circuits the transformer download; cached files are not re-fetched. No token is needed (the repo is ungated). The bf16 transformer source (~7.8 GB in the HuggingFace cache) may be deleted after quantization.

Usage

download_flux2_klein(quantize = TRUE, precision = c("auto", "fp8", "nf4"),
                     output_dir = NULL, text_encoders = TRUE, verbose = TRUE)

Arguments

Value

Invisibly, a list with transformer_dir, artifact_dir, and support (named file paths).

Download the LTX-2.3 checkpoint and build a quantized artifact

Description

Skips work that is already done: a valid manifest short-circuits everything; a cached 46 GB source skips the download. The source file may be deleted after quantization (it is never removed automatically).

Usage

download_ltx2(quantize = TRUE, precision = c("nf4", "fp8"), output_dir = NULL,
              text_encoder = TRUE, verbose = TRUE)

Arguments

Details

Both quantized tiers are buildable here. recommend returns nf4 for LTX on any card with 14 GB or more (it prefers nf4 at 1280 px over fp8 at 1024 px, since video trades weight precision for resolution), so nf4 is the tier most users want. fp8 additionally needs a safetensors that can write float8; asking for it without one warns and builds nf4 instead rather than failing inside the quantizer.

Value

Invisibly, a list with checkpoint (source path or NULL), artifact_dir, precision, text_encoder_dir, and fp8_dir for back-compatibility – the artifact directory when precision is "fp8", NULL otherwise, since a field named fp8_dir pointing at an nf4 artifact would be a trap.

Download and Prepare LTX-2.3 Model Weights

Description

Downloads the LTX-2.3 distilled checkpoint (46 GB, LTX-2 Community License) and the Gemma3 text encoder from HuggingFace with an explicit consent prompt, then quantizes the transformer to the local fp8 artifact (~26 GB) used by the GPU-poor pipeline.

Download TorchScript model files for Stable Diffusion

Description

Downloads the required model files (e.g., UNet, decoder, text encoder) for a given Stable Diffusion model using hfhub::hub_download().

Usage

download_model(model_name = "sd21",
               devices = list(unet = "cpu", decoder = "cpu", text_encoder = "cpu"),
               unet_dtype_str = NULL, overwrite = FALSE, show_progress = TRUE,
               download_models = FALSE)

Arguments

Details

Files are cached by hfhub (typically ~/.cache/huggingface/hub/). Legacy files in the old R_user_dir() location are also recognized.

Value

A named list of full file paths, keyed by component name.

Examples

## Not run: 
paths <- download_model("sd21")

## End(Not run)

Download the Stable Diffusion 2.1 diffusers weights

Description

Fetches the UNet, VAE, and CLIP text encoder from the cornball-ai/sd21-R HuggingFace dataset (fp16 diffusers safetensors, converted from the original OpenRAIL weights; the upstream stabilityai repo was deprecated). About 2.5 GB, one-time. The native tokenizer and DDIM scheduler need no downloads.

Usage

download_sd21(verbose = TRUE)

Arguments

Value

Invisibly, the diffusers directory (the parent of unet/, vae/, text_encoder/).

Download the Stable Diffusion XL diffusers weights

Description

Fetches the UNet (re-sharded to sub-2 GB shards), VAE, and both CLIP text encoders from the cornball-ai/sdxl-R HuggingFace dataset (fp16 diffusers safetensors, converted from the original stabilityai/stable-diffusion-xl-base-1.0 OpenRAIL++ weights). About 7 GB, one-time. The native tokenizer and DDIM scheduler need no downloads.

Usage

download_sdxl(verbose = TRUE)

Arguments

Value

Invisibly, the diffusers directory (the parent of unet/, vae/, text_encoder/, text_encoder_2/).

Download and Prepare Z-Image-Turbo Weights

Description

Downloads Z-Image-Turbo from HuggingFace (Apache-2.0, ungated) and quantizes the 6B transformer to a local fp8 (~6.3 GB) or NF4 (~3.6 GB) artifact. The checkpoint ships the transformer in float32 (24.6 GB), so the one-time quantize saves a lot of disk and load time.

Download Z-Image-Turbo and build the quantized artifact

Description

Skips work already done: a valid quantized manifest short-circuits the transformer download; cached files are not re-fetched. No token is needed (the repo is ungated). The float32 transformer source (~24.6 GB in the HuggingFace cache) may be deleted after quantization.

Usage

download_zimage_turbo(quantize = TRUE, precision = c("auto", "fp8", "nf4"),
                      output_dir = NULL, text_encoders = TRUE, verbose = TRUE)

Arguments

Value

Invisibly, a list with transformer_dir, artifact_dir, and support (named file paths).

Encode text to token IDs

Description

Encode text to token IDs

Usage

encode_bpe(tokenizer, text, add_special_tokens = TRUE, max_length = NULL,
           padding = "none", truncation = FALSE, return_tensors = "list")

Arguments

Value

List with input_ids and attention_mask.

Encode prompts with the Qwen tokenizer

Description

With chat_template = TRUE each prompt is wrapped as a single user turn with the generation prompt, matching apply_chat_template(..., add_generation_prompt = TRUE). With enable_thinking = FALSE (the FLUX.2 klein pipeline behavior) the template closes with an empty thinking block; with enable_thinking = TRUE (the Z-Image pipeline behavior) it ends at the assistant turn. Right-pads with <|endoftext|>.

Usage

encode_qwen(tokenizer, texts, max_length = 512L, chat_template = TRUE,
            enable_thinking = FALSE)

Arguments

Value

List with input_ids and attention_mask integer matrices [length(texts), max_length] (ragged lists when max_length is NULL). Ids are 0-based.

Encode a single text string

Description

Encode a single text string

Usage

encode_single(tokenizer, text, add_special_tokens = TRUE)

Encode text with a Unigram tokenizer

Description

Normalizes (strip-right, multi-space collapse, control whitespace to space), applies the Metaspace pre-tokenizer, segments each pre-token by Viterbi over the Unigram scores, fuses consecutive unknowns, and appends EOS. T5 semantics: right padding with <pad> (id 0), truncation to max_length - 1 before the EOS.

Usage

encode_unigram(tokenizer, texts, max_length = 256L, add_eos = TRUE, pad = TRUE)

Arguments

Value

List with input_ids and attention_mask, each an integer matrix [length(texts), max_length] (or ragged lists when max_length is NULL). Ids are 0-based (HuggingFace convention); add 1 for R torch embedding lookups.

Encode text with Gemma3 for LTX-2

Description

Full pipeline for encoding text prompts using Gemma3 text encoder. Returns the raw stacked per-layer hidden states (embedding layer plus all transformer layers) for downstream connector modules, which handle normalization and projection themselves.

Usage

encode_with_gemma3(prompts, model = NULL, tokenizer = NULL,
                   max_sequence_length = 1024L, device = "cuda",
                   dtype = "float16", verbose = TRUE)

Arguments

Value

List with prompt_embeds (raw stacked hidden states, shape [batch, seq_len, hidden_size, num_layers + 1]) and prompt_attention_mask.

Encode prompts with the Qwen3 encoder for FLUX.2

Description

Tokenizes with the chat template, runs the encoder with the padding mask, and concatenates the requested mid-stack hidden states per token, matching Flux2KleinPipeline._get_qwen3_prompt_embeds.

Usage

encode_with_qwen3(prompts, model, tokenizer, max_sequence_length = 512L,
                  out_layers = c(9L, 18L, 27L), device = NULL)

Arguments

Value

Tensor [length(prompts), max_sequence_length, 3 * hidden_size].

Encode prompts with the T5 encoder

Description

Tokenizes with encode_unigram (right padding to max_sequence_length) and runs the encoder. Matching the FLUX reference pipeline, no attention mask is used.

Usage

encode_with_t5(prompts, model, tokenizer, max_sequence_length = 256L,
               device = NULL)

Arguments

Value

Tensor [length(prompts), max_sequence_length, d_model].

Generate a filename from a prompt

Description

This function generates a filename from a prompt by removing all non-alphanumeric characters and replacing them with underscores. The filename is limited to 50 characters. If 'datetime' is set to TRUE, the current date and time are prepended to the filename.

Usage

filename_from_prompt(prompt, datetime = TRUE)

Arguments

Value

A character string representing the generated filename.

Examples

filename_from_prompt("A beautiful sunset over the mountains")
filename_from_prompt("A beautiful sunset over the mountains", datetime = FALSE)

Calculate shift for dynamic shifting

Description

Computes the shift parameter (mu) based on sequence length for resolution-dependent timestep shifting.

Usage

flowmatch_calculate_shift(seq_len, base_seq_len = 256L, max_seq_len = 4096L,
                          base_shift = 0.5, max_shift = 1.15)

Arguments

Value

Numeric. The computed shift value (mu).

Scale noise for flow matching forward process

Description

Applies the forward process in flow-matching: interpolates between the clean sample and noise.

Usage

flowmatch_scale_noise(sample, timestep, noise, schedule)

Arguments

Value

torch tensor. The noisy sample at timestep t.

Create a FlowMatch Euler Discrete Scheduler

Description

Creates a FlowMatch scheduler for use with flow-matching diffusion models like LTX-2. FlowMatch schedulers use Euler integration for sampling, which is simpler and often faster than DDIM-style schedulers.

Usage

flowmatch_scheduler_create(num_train_timesteps = 1000L, shift = 1,
                           use_dynamic_shifting = FALSE, base_shift = 0.5,
                           max_shift = 1.15, base_seq_len = 256L,
                           max_seq_len = 4096L, invert_sigmas = FALSE,
                           shift_terminal = NULL,
                           time_shift_type = c("exponential", "linear"))

Arguments

Details

FlowMatch (Flow Matching) is a framework for training continuous normalizing flows by regressing onto target probability paths. The Euler discrete scheduler implements simple Euler integration for sampling from trained flow models.

The core update rule is: prev_sample = sample + dt * model_output where dt = sigma_next - sigma_current.

Value

A FlowMatch scheduler object (list) containing:

sigmas

The noise schedule

timesteps

The timestep schedule

num_train_timesteps

Training timesteps

config

All configuration parameters

References

Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., & Le, M. (2022). "Flow Matching for Generative Modeling." https://arxiv.org/abs/2210.02747

Examples

if (torch::torch_is_installed()) {
  scheduler <- flowmatch_scheduler_create(
    num_train_timesteps = 1000,
    shift = 1.0
  )

  # Set timesteps for inference
  scheduler <- flowmatch_set_timesteps(scheduler, num_inference_steps = 8)
  scheduler$timesteps
}

Perform a FlowMatch scheduler step

Description

Performs a single denoising step using Euler integration. This is the core sampling function for FlowMatch models.

Usage

flowmatch_scheduler_step(model_output, timestep, sample, schedule,
                         generator = NULL)

Arguments

Details

The FlowMatch Euler step is remarkably simple: prev_sample = sample + dt * model_output where dt = sigma_next - sigma_current.

This implements the Euler method for solving the probability flow ODE in continuous normalizing flows.

Value

A list containing:

prev_sample

The denoised sample at the previous timestep

schedule

The updated scheduler with incremented step_index

Set timesteps for inference

Description

Configures the scheduler timesteps for a specific number of inference steps. This must be called before using the scheduler for denoising.

Usage

flowmatch_set_timesteps(schedule, num_inference_steps = 50L, device = "cpu",
                        mu = NULL, sigmas = NULL, timesteps = NULL)

Arguments

Value

Updated scheduler with configured timesteps and sigmas.

Normalize patchified latents with the VAE BatchNorm statistics

Description

FLUX.2 has no scalar scaling/shift factor; latents are standardized per packed channel with the VAE's bn.running_mean / bn.running_var (eps 1e-4). Reference: encode/decode paths of Flux2KleinPipeline.

Usage

flux2_bn_normalize(latents, bn_mean, bn_var, eps = 1e-04, inverse = FALSE)

Arguments

Value

Tensor like latents.

FLUX.2 double-stream (MMDiT) block

Description

Image and text streams with externally supplied (shift, scale, gate) modulation triples, joint attention (txt first), and SwiGLU feed-forwards. Reference: Flux2TransformerBlock.

Usage

flux2_double_block(dim, num_attention_heads, attention_head_dim, mlp_ratio = 3,
                   eps = 1e-06, bias = FALSE)

Arguments

Value

Module whose forward(hidden_states, encoder_hidden_states, temb_mod_img, temb_mod_txt, image_rotary_emb) returns list(encoder_hidden_states, hidden_states).

Empirical timestep shift for FLUX.2

Description

BFL's piecewise-linear fit of the dynamic-shifting mu as a function of image sequence length and step count; replaces FLUX.1's calculate_shift. Reference: compute_empirical_mu (adapted from BFL sampling.py).

Usage

flux2_empirical_mu(image_seq_len, num_steps)

Arguments

Value

Numeric mu for flowmatch_set_timesteps.

FLUX.2 feed-forward (fused SwiGLU)

Description

linear_in projects to twice the inner dim; SwiGLU gates the first half with SiLU and multiplies by the second half; linear_out projects back. Reference: Flux2FeedForward + Flux2SwiGLU.

Usage

flux2_feed_forward(dim, dim_out = NULL, mult = 3, bias = FALSE)

Arguments

Value

Module whose forward(x) returns the SwiGLU-gated projection of x, a tensor with the last axis of width dim_out.

Test whether a FLUX.2 key is in the quantization cast set

Description

Test whether a FLUX.2 key is in the quantization cast set

Usage

flux2_is_quant_key(key)

Arguments

Value

Logical vector.

Load the FLUX.2 klein pipeline

Description

Loads the quantized transformer artifact plus the FLUX.2 VAE decoder, Qwen3 text encoder, and tokenizer from the HuggingFace cache populated by download_flux2_klein. With fp8 precision the ~4 GB transformer stays GPU-resident.

Usage

flux2_load_pipeline(model_dir = NULL, device = "cuda",
                    precision = c("auto", "fp8", "nf4", "bf16"),
                    text_device = NULL, attn_chunk = NULL,
                    phase_offload = TRUE, pin = NULL, verbose = TRUE)

Arguments

Value

A flux2_pipeline list.

FLUX.2 shared modulation projection

Description

linear(silu(temb)) producing mod_param_sets triples of (shift, scale, gate). Computed once per forward at model level and broadcast to every block. Reference: Flux2Modulation.

Usage

flux2_modulation(dim, mod_param_sets = 2L, bias = FALSE)

Arguments

Value

Module whose forward(temb) returns the modulation tensor linear(silu(temb)), holding mod_param_sets triples of (shift, scale, gate) along the last axis.

Pack patchified FLUX.2 latents into tokens

Description

[B, C, H, W] -> [B, H * W, C] (row-major spatial flatten, channels-last). Reference: Flux2KleinPipeline._pack_latents.

Usage

flux2_pack_latents(latents)

Arguments

Value

Tensor [B, H * W, C].

FLUX.2 parallel self-attention (single-stream)

Description

ViT-22B-style parallel block internals: one fused projection produces QKV and the SwiGLU MLP input; one fused projection consumes cat(attention output, MLP output). Reference: Flux2ParallelSelfAttention + Flux2ParallelSelfAttnProcessor.

Usage

flux2_parallel_self_attention(query_dim, heads, dim_head, mlp_ratio = 3,
                              eps = 1e-06, bias = FALSE)

Arguments

Value

Module whose forward(hidden_states, image_rotary_emb, chunk_size) returns the block output [B, S, query_dim]: attention and MLP branches computed in parallel from one fused projection, concatenated, and projected back by a second fused layer.

Patchify FLUX.2 latents (2x2 -> channels)

Description

[B, C, H, W] -> [B, 4C, H/2, W/2], channel order (C, ph, pw). Reference: Flux2KleinPipeline._patchify_latents.

Usage

flux2_patchify_latents(latents)

Arguments

Value

Tensor [B, C * 4, H / 2, W / 2].

Build FLUX.2 latent position ids

Description

Columns (T, H, W, L) with H and W carrying the packed-grid position (row-major: H varies slowest), T = L = 0. Reference: Flux2KleinPipeline._prepare_latent_ids.

Usage

flux2_prepare_latent_ids(height, width, device = "cpu")

Arguments

Value

Float tensor [height * width, 4].

Build FLUX.2 text position ids

Description

Columns (T, H, W, L) with only L varying: 0..len-1. Reference: Flux2KleinPipeline._prepare_text_ids.

Usage

flux2_prepare_text_ids(len, device = "cpu")

Arguments

Value

Float tensor [len, 4].

FLUX.2 single-stream block (parallel)

Description

Parameterless LayerNorm with external modulation, then the fused parallel attention+MLP. Operates on the pre-concatenated [text; image] sequence (the reference model concatenates once before the stack). Reference: Flux2SingleTransformerBlock.

Usage

flux2_single_block(dim, num_attention_heads, attention_head_dim, mlp_ratio = 3,
                   eps = 1e-06, bias = FALSE)

Arguments

Value

Module whose forward(hidden_states, temb_mod, image_rotary_emb) returns the joint hidden states.

FLUX.2 transformer model

Description

Shared modulation computed once per forward; double blocks over separate text/image streams, then single (parallel) blocks over the concatenated [text; image] sequence. Rotary embeddings are precomputed by the caller with flux_pos_embed (axes_dim = c(32, 32, 32, 32), theta = 2000) over the concatenated [text; image] 4-axis position ids.

Usage

flux2_transformer(in_channels = 128L, num_layers = 5L, num_single_layers = 20L,
                  attention_head_dim = 128L, num_attention_heads = 24L,
                  joint_attention_dim = 7680L, mlp_ratio = 3,
                  timestep_guidance_channels = 256L,
                  axes_dims_rope = c(32L, 32L, 32L, 32L), rope_theta = 2000,
                  eps = 1e-06, out_channels = NULL)

Arguments

Value

Module whose forward(hidden_states, encoder_hidden_states, timestep, image_rotary_emb) returns the predicted velocity for the image tokens [B, S_img, out_channels]. timestep is in sigma space (0-1); it is scaled by 1000 internally.

Unpack FLUX.2 tokens back to a latent grid via position ids

Description

Scatters tokens to (H, W) positions taken from the id columns (H = column 2, W = column 3, 0-based values). Reference: Flux2KleinPipeline._unpack_latents_with_ids.

Usage

flux2_unpack_latents_with_ids(x, ids, height, width)

Arguments

Value

Tensor [B, C, height, width].

Unpatchify FLUX.2 latents (channels -> 2x2)

Description

Inverse of flux2_patchify_latents. Reference: Flux2KleinPipeline._unpatchify_latents.

Usage

flux2_unpatchify_latents(latents)

Arguments

Value

Tensor [B, C, H * 2, W * 2].

FLUX.2 VAE decoder

Description

The AutoencoderKLFlux2 decode path: post_quant_conv (1x1, 32 channels) followed by the standard AutoencoderKL decoder body (reused from vae_decoder_native), plus the BatchNorm running statistics used for latent (de)normalization. Reference: src/diffusers/models/autoencoders/autoencoder_kl_flux2.py.

Usage

flux2_vae_decoder(latent_channels = 32L,
                  block_channels = c(512L, 512L, 256L, 128L), norm_groups = 32L)

Arguments

Value

Module whose forward(z) decodes [B, 32, H, W] latents to [B, 3, 8H, 8W] images; $bn$running_mean / $bn$running_var carry the normalization statistics.

FLUX continuous adaLN (final norm)

Description

Scale/shift conditioning of the final norm. Note the chunk order: scale first, then shift (the reverse of adaLN-Zero). Reference: diffusers AdaLayerNormContinuous as used by FLUX norm_out (elementwise_affine = FALSE, eps = 1e-6).

Usage

flux_ada_layer_norm_continuous(dim, cond_dim = dim, bias = TRUE)

Arguments

Value

Module whose forward(x, cond) returns x normalized and then scaled and shifted by the conditioning embedding, a tensor of the same shape as x.

FLUX adaLN-Zero modulation (double-stream)

Description

Projects the conditioning embedding to six modulation vectors and returns the msa-modulated input plus the remaining parameters. Reference: diffusers AdaLayerNormZero.

Usage

flux_ada_layer_norm_zero(dim)

Arguments

Value

Module whose forward(x, emb) returns list(x_norm, gate_msa, shift_mlp, scale_mlp, gate_mlp).

FLUX adaLN-Zero modulation (single-stream)

Description

Three modulation vectors: shift, scale, gate. Reference: diffusers AdaLayerNormZeroSingle.

Usage

flux_ada_layer_norm_zero_single(dim)

Arguments

Value

Module whose forward(x, emb) returns list(x_norm, gate).

Apply FLUX rotary embeddings to a per-head tensor

Description

Rotates adjacent element pairs of the last dimension: out = x * cos + rotate_half(x) * sin with pairs interleaved (elements 1,2 form the first complex pair). Math in float32, result cast back to the input dtype. Reference: apply_rotary_emb with use_real_unbind_dim = -1.

Usage

flux_apply_rotary_emb(x, freqs)

Arguments

Value

Tensor with the same shape and dtype as x.

FLUX joint attention

Description

Multi-head attention with per-head RMS q/k norms and rotary position embeddings. With added_kv = TRUE (double-stream blocks) the text stream gets its own q/k/v projections and both streams attend jointly (text tokens first); the outputs are split back and projected per stream. With pre_only = TRUE (single-stream blocks) there is no output projection. Reference: diffusers FluxAttention + FluxAttnProcessor.

Usage

flux_attention(query_dim, heads, dim_head, added_kv = FALSE, pre_only = FALSE,
               eps = 1e-06, bias = TRUE)

Arguments

Value

Module whose forward(hidden_states, encoder_hidden_states, image_rotary_emb, chunk_size) returns the attended image stream [B, S, query_dim]. When encoder_hidden_states is supplied (double-stream blocks) it returns list(image, text) instead, each projected by its own output layer.

FLUX double-stream (MMDiT) transformer block

Description

Image and text streams each get adaLN-Zero modulation and a feed-forward; attention is joint across both streams. Reference: diffusers FluxTransformerBlock.

Usage

flux_double_block(dim, num_attention_heads, attention_head_dim)

Arguments

Value

Module whose forward(hidden_states, encoder_hidden_states, temb, image_rotary_emb) returns list(encoder_hidden_states, hidden_states).

Test whether a FLUX key is in the quantization cast set

Description

Test whether a FLUX key is in the quantization cast set

Usage

flux_is_quant_key(key)

Arguments

Value

Logical vector.

Load the FLUX.1-schnell pipeline

Description

Loads the quantized transformer artifact plus the VAE decoder, CLIP and T5 text encoders, tokenizer, and scheduler config (from the HuggingFace cache populated by download_flux1). Components load to the CPU when phase_offload is on and move to the GPU only for their phase of the generation.

Usage

flux_load_pipeline(model_dir = NULL, device = "cuda", precision = NULL,
                   text_device = NULL, attn_chunk = NULL, phase_offload = TRUE,
                   pin = NULL, verbose = TRUE)

Arguments

Value

A flux_pipeline list.

Load a FLUX transformer from any checkpoint format

Description

Builds flux_transformer from the checkpoint's embedded config and loads the weights. Dispatches on the checkpoint format:

Usage

flux_load_transformer(ckpt, device = "cuda", dtype = "bfloat16", pin = TRUE,
                      fp8_resident = FALSE, verbose = TRUE, ...)

Arguments

Details

• full precision (flux_open_checkpoint): weights stream into the model in dtype on device.

• "nf4" (flux_open_quantized): cast-set linears become ltx23_nf4_linear; the whole model (packed weights included) moves to device and stays resident.

• "fp8": cast-set linears become ltx23_fp8_linear; fp8 weights stay CPU-resident (optionally pinned) and stream to device inside each forward.

Value

The loaded flux_transformer in eval mode.

Resolve a FLUX memory profile

Description

A thin adapter over recommend for the FLUX.1 pipeline, kept for back-compatibility. recommend("flux1") is the policy; this reshapes it into the legacy profile fields the loader consumes. Precision now rises with VRAM (nf4 default, fp8 GPU-resident on 14 GB+ cards when safetensors can read float8, bf16 on 24 GB+); the old bands, which put fp8 in a narrow low-VRAM slot it can no longer fit, were backwards.

Usage

flux_memory_profile(vram_gb = NULL)

Arguments

Value

List with name, precision ("nf4"/"fp8"/"bf16"), attn_chunk, text_device, phase_offload, max_pixels, and (advisory) fork_suggested and note.

Open a FLUX transformer checkpoint directory

Description

Opens a diffusers-layout transformer directory lazily (headers only). Sharded checkpoints are resolved through the index.json weight map; single-file checkpoints are opened directly. The transformer config.json is attached as $config.

Usage

flux_open_checkpoint(transformer_dir)

Arguments

Value

An object of class ltx23_checkpoint (shared checkpoint interface): list with handle$get_tensor, keys, config, and path.

Open a quantized FLUX artifact directory

Description

Opens the sharded NF4/fp8 artifact written by flux_quantize through the shared checkpoint interface. The manifest's embedded transformer config and format ride along, so flux_load_transformer needs nothing else.

Usage

flux_open_quantized(dir)

Arguments

Value

An ltx23_checkpoint with $format set.

Pack FLUX latents into a patch sequence

Description

Packs a [B, C, H, W] latent into 2x2 patches, giving a sequence [B, (H/2) * (W/2), C * 4]. Reference: FluxPipeline._pack_latents.

Usage

flux_pack_latents(latents)

Arguments

Value

Tensor of shape [B, (H/2) * (W/2), C * 4].

Compute FLUX rotary frequencies from position ids

Description

Per-axis 1D rotary frequencies (interleaved-real convention), computed in float64 on CPU and concatenated over the axes. Reference: FluxPosEmbed with get_1d_rotary_pos_embed(repeat_interleave_real=TRUE, use_real=TRUE, freqs_dtype=float64).

Usage

flux_pos_embed(ids, axes_dim = c(16L, 56L, 56L), theta = 10000)

Arguments

Value

List of two tensors (cos, sin), each [S, sum(axes_dim)], float32, on the device of ids.

Build FLUX latent image position ids

Description

Position ids over the packed latent grid (latent height/2 x width/2). Channel 1 is always zero, channel 2 holds the row index, channel 3 the column index. Reference: FluxPipeline._prepare_latent_image_ids.

Usage

flux_prepare_latent_image_ids(height, width, device = "cpu")

Arguments

Value

Float tensor of shape [height * width, 3].

Quantize a FLUX transformer to NF4 or fp8 shards

Description

Streams the bf16 diffusers checkpoint tensor by tensor. Cast-set weights (see flux_is_quant_key) are stored as NF4 (packed uint8 + <key>_absmax float32 blocks) or as float8_e4m3fn with an absmax/448 per-tensor <key>_scale; everything else is copied through unchanged. The manifest embeds the transformer config, so the source checkpoint is not needed again after quantization.

Usage

flux_quantize(transformer_dir, output_dir = NULL, format = c("nf4", "fp8"),
              shard_bytes = 1.9e+09, force = FALSE, verbose = TRUE)

Arguments

Value

Invisibly, the manifest list.

FLUX single-stream transformer block

Description

Parallel attention + MLP over the joint [text; image] sequence with a shared gate: x + gate * proj_out(cat(attn, gelu(mlp))). The reference concatenates the streams inside every block and splits after; here the caller concatenates once before the single-block stack, which is numerically identical. Reference: diffusers FluxSingleTransformerBlock.

Usage

flux_single_block(dim, num_attention_heads, attention_head_dim, mlp_ratio = 4)

Arguments

Value

Module whose forward(hidden_states, temb, image_rotary_emb) returns the joint hidden states.

FLUX transformer model

Description

19 double-stream (MMDiT) blocks followed by 38 single-stream blocks over the joint [text; image] sequence, with adaLN-Zero conditioning on timestep + pooled CLIP text. Rotary embeddings are precomputed by the caller with flux_pos_embed (they are static across denoise steps). Defaults are the FLUX.1-schnell configuration.

Usage

flux_transformer(in_channels = 64L, num_layers = 19L, num_single_layers = 38L,
                 attention_head_dim = 128L, num_attention_heads = 24L,
                 joint_attention_dim = 4096L, pooled_projection_dim = 768L,
                 axes_dims_rope = c(16L, 56L, 56L), out_channels = NULL)

Arguments

Value

Module whose forward(hidden_states, encoder_hidden_states, pooled_projections, timestep, image_rotary_emb) returns the predicted velocity for the image tokens [B, S_img, out_channels]. timestep is in sigma space (0-1); it is scaled by 1000 internally, matching the reference.

Unpack a FLUX patch sequence back into latents

Description

Inverse of flux_pack_latents. Height and width are the target image dimensions in pixels; the latent grid is derived via the VAE scale factor and the 2x2 patch size. Reference: FluxPipeline._unpack_latents.

Usage

flux_unpack_latents(latents, height, width, vae_scale_factor = 8L)

Arguments

Value

Tensor of shape [B, C_packed / 4, height / 8, width / 8].

FP8 Weight Storage for the LTX-2.3 Transformer

Description

GPU-poor weight handling: the large attention/FFN linears of the DiT are stored as float8_e4m3fn with per-tensor scales (the official LTX quantization policy), kept CPU-resident (optionally pinned), and dequantized on the compute device inside each forward. Everything else (norms, embeddings, modulation tables, biases) stays bfloat16. Requires a safetensors build with F8 dtype support.

Gemma3 Attention

Description

Multi-head attention with Grouped Query Attention (GQA) and optional sliding window attention.

Usage

gemma3_attention(config, layer_idx = 0L)

Arguments

Create Gemma3 configuration for LTX-2

Description

Returns the default configuration used by LTX-2's text encoder.

Usage

gemma3_config_ltx2()

Value

List with model configuration parameters.

Gemma3 Decoder Layer

Description

Single transformer block with pre-norm attention and MLP.

Usage

gemma3_decoder_layer(config, layer_idx = 0L)

Arguments

Batch-encode prompts with Gemma3, cached to disk

Description

Encodes a vector of prompts in sub-batches (bounding activation VRAM) and optionally caches each prompt's result under cache_dir, keyed by the prompt text and sequence length. Already-cached prompts are skipped, so an interrupted batch resumes where it stopped. Embeddings land on the CPU either way; the renderer moves them per phase.

Usage

gemma3_encode_batch(prompts, model = NULL, tokenizer = NULL, batch_size = 4L,
                    cache_dir = NULL, max_sequence_length = 1024L,
                    device = "cuda", verbose = TRUE)

Arguments

Details

Budget note: each prompt's result is the full hidden-state stack the LTX connectors consume - roughly 0.4 GB at 1024 tokens - so caching N prompts needs ~0.4N GB of disk.

Value

Character vector of cache paths (with cache_dir) or a list of per-prompt embedding results (without).

Gemma3 MLP

Description

Feed-forward network with gated linear units and GELU activation.

Usage

gemma3_mlp(config)

Arguments

Quantize a Gemma3 text encoder to NF4 shards

Description

Streams the HuggingFace Gemma3 checkpoint tensor by tensor. The language model's projection weights (q/k/v/o and gate/up/down, ~11B of the 12B parameters) are stored as NF4 (packed uint8 + <key>_absmax float32 blocks); embeddings and norms are copied at the resident dtype. Vision-tower and projector weights are dropped - the text encoder never uses them. The result is a ~8 GB artifact that fits a 16 GB card during the encode phase (vs 45 GB of host RAM for the fp32 CPU path).

Usage

gemma3_quantize_nf4(model_path, output_dir = NULL, shard_bytes = 1.9e+09,
                    force = FALSE, verbose = TRUE)

Arguments

Details

Keys are stored normalized (language_model. / model. prefixes stripped), matching the module tree of gemma3_text_model.

Value

Invisibly, the manifest list.

Gemma3 RMS Normalization

Description

RMSNorm with optional addition of 1 to weights (Gemma-style).

Usage

gemma3_rms_norm(dim, eps = 1e-06)

Arguments

Gemma3 Rotary Position Embeddings

Description

Standard RoPE with optional scaling factor for extended context.

Usage

gemma3_rotary_embedding(dim, max_position_embeddings = 8192L, base = 10000,
                        scaling_factor = 1)

Arguments

Gemma3 Text Model

Description

Full Gemma3 text encoder model.

Usage

gemma3_text_model(config)

Arguments

Value

Module whose forward(input_ids, ...) returns list(last_hidden_state, hidden_states): the final hidden state [B, S, hidden_size] and the list of per-layer hidden states.

Gemma3 Tokenizer

Description

Native R tokenizer for Gemma3 using BPE. Loads from HuggingFace tokenizer.json format.

Usage

gemma3_tokenizer(tokenizer_path)

Arguments

Value

A gemma3_tokenizer object (extends bpe_tokenizer).

Get BOS token ID

Description

Get BOS token ID

Usage

get_bos_id(tokenizer)

Get EOS token ID

Description

Get EOS token ID

Usage

get_eos_id(tokenizer)

Get padding token ID

Description

Get padding token ID

Usage

get_pad_id(tokenizer)

Get required components for each model type

Description

This function returns a list of required components for each supported model type.

Usage

get_required_components(model_name)

Arguments

Value

A character vector of required components for the specified model.

Greedy longest match tokenization

Description

Greedy longest match tokenization

Usage

greedy_tokenize(text, vocab, byte_fallback = FALSE, unk_token = NULL)

Group Normalization (32 groups)

Description

Group Normalization (32 groups)

Usage

group_norm_32(channels)

Download a single TorchScript component via hfhub

Description

Internal helper that wraps hfhub::hub_download() with the cornball-ai dataset repo naming convention. Falls back to legacy R_user_dir() paths if the file exists there but not yet in the hfhub cache.

Usage

hf_download_pt(model_name, filename, download = TRUE)

Arguments

Value

The local file path (character string).

Image-to-Image Generation with Stable Diffusion

Description

This function generates an image based on an input image and a text prompt using the Stable Diffusion model. It allows for various configurations such as model name, device, scheduler, and more.

Usage

img2img(input_image, prompt, negative_prompt = NULL, img_dim = 512,
        model_name = c("sd21", "sdxl"), pipeline = NULL, devices = "auto",
        unet_dtype_str = "float16", download_models = FALSE,
        scheduler = "ddim", num_inference_steps = 50, strength = 0.8,
        guidance_scale = 7.5, seed = NULL, save_file = TRUE, filename = NULL,
        metadata_path = NULL, use_native_decoder = FALSE,
        use_native_text_encoder = FALSE, use_native_unet = FALSE, ...)

Arguments

Value

An image array and metadata

Check if GPU is Blackwell Architecture

Description

Blackwell GPUs (RTX 50xx) may need special handling.

Usage

is_blackwell_gpu()

Value

Logical. TRUE if Blackwell GPU detected.

Examples

# Soft probe: FALSE on any machine without a Blackwell card, and on
# machines where torch has no lantern binaries.
is_blackwell_gpu()

LTX-2.3 JIT Block Stack

Description

TorchScript compilation of the 48-block NF4 transformer step (cf. the torch skill's JIT-decode pattern proven in whisper and chatterbox). Eager execution crosses R -> lantern per op (~190 us each) and leaves every intermediate as an R tensor handle that only dies at gc(); at high resolution that forces a per-block gc() costing the vast majority of step time. Compiled, the whole block stack is one crossing: intermediates are freed eagerly by libtorch, no R garbage accumulates, no per-block gc is needed, and attention runs through the fused scaled_dot_product_attention kernel instead of a materialized score matrix.

Details

Weights are passed per call as a flat List[Tensor] (borrowed by reference, no copies) with a fixed per-block layout; the packer and the TorchScript indices must stay in lockstep (parity-tested).

JIT-Traced Decode for the LTX-2.3 VAEs and Vocoder

Description

The video/audio decoders and the vocoder are static feed-forward graphs, so torch::jit_trace converts them wholesale: one R-to-libtorch crossing per forward, intermediates freed eagerly by libtorch instead of accumulating as R handles until gc. Traces are shape-specialized (runtime sizes bake into the graph as constants; a mismatched input errors), so they are cached per instance, input shape, dtype, device, and call tag, and re-traced on a miss.

Details

A trace captures the module's weight tensors, which would pin them on the GPU across phase offloads; the pipeline releases all traces whenever a component offloads (.ltx23_release_vae_traces).

Tracing hazard on this lantern build: if the allocator callback runs R's gc during trace recording (memory pressure), the recorded graph can capture garbage argument values (observed as corrupted narrow starts on the full-size decoder; verified 5/5 clean once gc cannot fire mid-trace). Defenses, in order: a gc + cache flush right before each trace so pressure starts near zero, tryCatch around trace and replay, and a one-time validation of every fresh trace against the eager output — any mismatch permanently blacklists that shape and runs eager. With those in place the traced path cannot corrupt output — but per render it measured slower than eager (traces are released on phase offload, so every render re-pays trace + validation), so it stays opt-in: options(diffuseR.jit_vae = TRUE).

Create Video from Latents (Helper)

Description

Convenience function to decode latents and save video in one step.

Usage

latents_to_video(latents, vae, file, fps = 24, ...)

Arguments

Value

Invisibly returns the output file path.

Load HF safetensors VAE weights into the native decoder

Description

Loads the decoder half of a diffusers AutoencoderKL safetensors file (e.g. FLUX.1-schnell's vae/diffusion_pytorch_model.safetensors). Keys under decoder. map to the native module 1:1; encoder and quant-conv keys are skipped (the FLUX VAE has no quant convs, and txt2img needs no encoder).

Usage

load_decoder_safetensors(native_decoder, path, verbose = TRUE)

Arguments

Value

The native decoder with loaded weights (invisibly)

Load weights from TorchScript decoder into native decoder

Description

Load weights from TorchScript decoder into native decoder

Usage

load_decoder_weights(native_decoder, torchscript_path, verbose = TRUE)

Arguments

Value

The native decoder with loaded weights (invisibly)

Load the FLUX.2 VAE decoder from safetensors

Description

Loads the decoder half plus post_quant_conv and the BatchNorm running statistics; encoder and quant_conv keys are skipped (txt2img needs no encoder).

Usage

load_flux2_vae_decoder(path, latent_channels = 32L,
                       block_channels = c(512L, 512L, 256L, 128L),
                       norm_groups = 32L, verbose = TRUE)

Arguments

Value

The loaded flux2_vae_decoder in eval mode.

Load a Gemma3 text encoder from an NF4 artifact

Description

Builds the model as a skeleton at the compute dtype, swaps the projection linears for NF4 modules filled from the artifact (dequantized per forward through the shared byte-LUT), copies the residents, and hard-errors on any parameter the artifact does not fill.

Usage

load_gemma3_nf4(artifact_dir, device = "cuda", dtype = "bfloat16",
                pin = getOption("diffuseR.pin_staging", TRUE), verbose = TRUE)

Arguments

Value

A gemma3_text_model ready for encode_with_gemma3.

Load Gemma3 Text Model from safetensors

Description

Loads pre-trained Gemma3 weights from HuggingFace safetensors files. An NF4 artifact directory (from gemma3_quantize_nf4) dispatches to load_gemma3_nf4.

Usage

load_gemma3_text_encoder(model_path, device = "cpu", dtype = "float16",
                         pin = getOption("diffuseR.pin_staging", TRUE),
                         verbose = TRUE)

Arguments

Value

Initialized gemma3_text_model with loaded weights.

Load weights into Gemma3 model

Description

Load weights into Gemma3 model

Usage

load_gemma3_weights(model, weights, verbose = TRUE)

Load a specific component of a diffusion model

Description

Loads a TorchScript model component (UNet, decoder, or text encoder) from the hfhub cache or legacy model directory, downloading it first if necessary.

Usage

load_model_component(component, model_name = "sd21", device = "cpu",
                     unet_dtype_str = NULL, download = TRUE, use_native = FALSE)

Arguments

Value

A torch model object.

Examples

## Not run: 
unet <- load_model_component("unet", "sd21", "cpu")

## End(Not run)

Load a diffusion model pipeline

Description

This function loads a diffusion model pipeline consisting of a UNet, VAE decoder, and text encoder. It initializes the models and sets up the environment for inference.

Usage

load_pipeline(model_name, m2d, i2i = FALSE, unet_dtype_str,
              use_native_decoder = FALSE, use_native_text_encoder = FALSE,
              use_native_unet = FALSE, ...)

Arguments

Value

An environment containing the loaded models and configuration.

Examples

## Not run: 
pipeline <- load_pipeline("my_model", device = "cuda")

## End(Not run)

Load a Qwen3 encoder from a transformers directory

Description

Streams the (possibly sharded) safetensors weights into qwen3_encoder. The LM head is tied to the embeddings and skipped.

Usage

load_qwen3_text_encoder(model_path, device = "cpu", dtype = "float32",
                        verbose = TRUE, ...)

Arguments

Value

The loaded qwen3_encoder in eval mode.

Load a T5 encoder from a transformers directory

Description

Streams the (possibly sharded) safetensors weights into t5_encoder, stripping the encoder. key prefix and aliasing embed_tokens to the shared embedding.

Usage

load_t5_text_encoder(model_path, device = "cpu", dtype = "float32",
                     verbose = TRUE, ...)

Arguments

Value

The loaded t5_encoder in eval mode.

Load HF safetensors weights into the native SDXL text encoder 2

Description

Loads a HuggingFace CLIPTextModelWithProjection model.safetensors (SDXL's OpenCLIP ViT-bigG text_encoder_2) into text_encoder2_native. Same key layout as encoder 1, plus the top-level text_projection.weight that produces the pooled text_embeds.

Usage

load_text_encoder2_safetensors(native_encoder, path, verbose = TRUE)

Arguments

Value

The native encoder with loaded weights (invisibly)

Load weights from TorchScript text encoder 2 into native encoder

Description

Load weights from TorchScript text encoder 2 into native encoder

Usage

load_text_encoder2_weights(native_encoder, torchscript_path, verbose = TRUE)

Arguments

Value

The native encoder with loaded weights (invisibly)

Load HF safetensors weights into the native CLIP text encoder

Description

Loads a HuggingFace CLIPTextModel model.safetensors (e.g. FLUX.1-schnell's text_encoder or SDXL's text_encoder) into text_encoder_native, reusing the TorchScript key remaps minus the export prefixes.

Usage

load_text_encoder_safetensors(native_encoder, path, verbose = TRUE)

Arguments

Value

The native encoder with loaded weights (invisibly)

Load weights from TorchScript text encoder into native encoder

Description

Load weights from TorchScript text encoder into native encoder

Usage

load_text_encoder_weights(native_encoder, torchscript_path, verbose = TRUE)

Arguments

Value

The native encoder with loaded weights (invisibly)

Load Module to GPU

Description

Moves a torch module and all its parameters to CUDA.

Usage

load_to_gpu(module, device = "cuda")

Arguments

Value

The module (modified in place).

Examples

if (torch::torch_is_installed()) {
  model <- torch::nn_linear(4, 2)
  # "cuda" needs a GPU; "cpu" is the portable round trip.
  load_to_gpu(model, device = "cpu")
}

Load HF safetensors weights into the native SD21 UNet

Description

Load HF safetensors weights into the native SD21 UNet

Usage

load_unet_safetensors(native_unet, path, verbose = TRUE)

Arguments

Value

The native UNet with weights loaded (invisibly).

Load HF safetensors weights into the native SDXL UNet

Description

Load HF safetensors weights into the native SDXL UNet

Usage

load_unet_sdxl_safetensors(native_unet, path, verbose = TRUE)

Arguments

Value

The native UNet with weights loaded (invisibly).

Load weights from TorchScript SDXL UNet into native SDXL UNet

Description

Load weights from TorchScript SDXL UNet into native SDXL UNet

Usage

load_unet_sdxl_weights(native_unet, torchscript_path, verbose = TRUE)

Arguments

Value

The native UNet with loaded weights (invisibly)

Load weights from TorchScript UNet into native UNet

Description

Load weights from TorchScript UNet into native UNet

Usage

load_unet_weights(native_unet, torchscript_path, verbose = TRUE)

Arguments

Value

The native UNet with loaded weights (invisibly)

Adaptive layer norm single (adaLN-single)

Description

Embeds a timestep/sigma and projects it to a configurable number of modulation parameter vectors.

Usage

ltx23_ada_layer_norm_single(embedding_dim, num_mod_params = 6L)

Arguments

Value

Module whose forward returns list(mod_params [N, num_mod_params * dim], embedded_timestep [N, dim]).

Adaptive instance normalization between latent tensors

Description

Matches each (batch, channel) slice's mean/std to the reference latents, blended by factor (cf. diffusers LTX2LatentUpsamplePipeline.adain_filter_latent).

Usage

ltx23_adain_filter_latent(latents, reference_latents, factor = 1)

Arguments

Value

Filtered latents.

Anti-aliased activation

Description

Upsample 2x, apply the activation, downsample 2x.

Usage

ltx23_antialias_act1d(channels, ratio = 2L, kernel_size = 12L)

Arguments

Value

Module whose forward(x) returns the activation applied at 2x rate (upsample, activate, downsample), a tensor of the same shape as x, with the aliasing the raw activation would introduce filtered out.

Apply interleaved rotary embeddings

Description

Rotates adjacent element pairs of the last dimension: out = x * cos + rotate_half(x) * sin with pairs interleaved (elements 1,2 form the first complex pair).

Usage

ltx23_apply_interleaved_rotary_emb(x, freqs)

Arguments

Value

Tensor with the same shape and dtype as x.

Apply split rotary embeddings

Description

Rotates element pairs formed by splitting the last dimension in half: element i pairs with element i + d/2. The cos/sin tensors carry half the head dimension.

Usage

ltx23_apply_split_rotary_emb(x, freqs)

Arguments

Value

Tensor with the same shape and dtype as x.

LTX-2 attention layer

Description

Attention with RMS q/k norms across heads, optional per-head output gating (LTX-2.3), separate query/key RoPE (for a2v/v2a cross attention), and optional STG perturbation (skip attention, use the value projection).

Usage

ltx23_attention(query_dim, heads = 8L, kv_heads = NULL, dim_head = 64L,
                bias = TRUE, cross_attention_dim = NULL, out_bias = TRUE,
                norm_eps = 1e-06, norm_elementwise_affine = TRUE,
                rope_type = "split", apply_gated_attention = FALSE)

Arguments

Value

Module whose forward(hidden_states, ...) returns the attended states [B, S, query_dim] after the output projection.

Causal 2D convolution for audio spectrograms

Description

Pads asymmetrically along the causal axis ("height" = time frames for LTX audio) before an unpadded Conv2d.

Usage

ltx23_audio_causal_conv2d(in_channels, out_channels, kernel_size = 3L,
                          stride = 1L, causality_axis = "height")

Arguments

Value

Module whose forward(x) returns the convolved tensor, padded so that each output frame depends only on current and earlier input frames.

LTX-2.3 audio VAE decoder

Description

Latents [B, 8, L, 16] -> mel spectrogram [B, 2, 4L - 3, 64].

Usage

ltx23_audio_decoder(base_channels = 128L, output_channels = 2L,
                    num_res_blocks = 2L, latent_channels = 8L,
                    ch_mult = c(1L, 2L, 4L), causality_axis = "height",
                    mel_bins = 64L)

Arguments

Value

Module whose forward(x) returns the decoded mel spectrogram reconstructed from an audio latent.

LTX audio downsampler

Description

Causal zero-pad followed by a plain stride-2 conv (reference LTX2AudioDownsample; note the conv is unwrapped, so its checkpoint key is downsample.conv.*).

Usage

ltx23_audio_downsample(in_channels, causality_axis = "height")

Arguments

Value

Module whose forward(x) returns the strided convolution of x, halving the downsampled axes.

LTX-2.3 audio VAE encoder

Description

Mel spectrogram [B, 2, T, 64] -> latent distribution moments [B, 2 * latent_channels, ceil(T/4), 16]. Structure mirrors the decoder: causal convs, parameterless pixel norms, ResNet stages with stride-2 downsampling between levels (reference LTX2AudioEncoder).

Usage

ltx23_audio_encoder(base_channels = 128L, in_channels = 2L,
                    num_res_blocks = 2L, latent_channels = 8L,
                    ch_mult = c(1L, 2L, 4L), causality_axis = "height")

Arguments

Value

Module whose forward(x) returns the encoded audio latent, a tensor downsampled along time and mel axes with the configured latent channel count.

Build the 16 kHz log-mel frontend for audio conditioning

Description

An ltx23_mel_stft whose STFT and mel bases are constructed (not checkpoint-loaded) with the audio VAE's preprocessing spec.

Usage

ltx23_audio_mel_frontend(filter_length = 1024L, hop_length = 160L,
                         n_mels = 64L, sample_rate = 16000L, fmin = 0,
                         fmax = 8000)

Arguments

Value

An ltx23_mel_stft module.

LTX audio ResNet block

Description

PixelNorm -> SiLU -> causal conv, twice, with a 1x1 causal conv shortcut (nin_shortcut) on channel change.

Usage

ltx23_audio_resnet_block(in_channels, out_channels = NULL,
                         causality_axis = "height")

Arguments

Value

Module whose forward(x) returns x plus the residual branch, a tensor of the same shape as x.

LTX audio upsampler

Description

Nearest 2x interpolation, causal conv, then a crop of the first element along the causal axis.

Usage

ltx23_audio_upsample(in_channels, causality_axis = "height")

Arguments

Value

Module whose forward(x) returns the tensor upsampled 2x by nearest-neighbour interpolation and convolved.

LTX-2.3 audio VAE

Description

Encoder + decoder plus the per-channel latent statistics loaded from the checkpoint. Encoding is used for audio-conditioned generation (lip sync); decoding for generated audio.

Usage

ltx23_audio_vae(base_channels = 128L, output_channels = 2L,
                num_res_blocks = 2L, latent_channels = 8L,
                ch_mult = c(1L, 2L, 4L), causality_axis = "height",
                mel_bins = 64L, in_channels = 2L)

Arguments

Value

Module bundling the audio encoder and decoder. Its forward(z) is decode(z), returning the mel spectrogram for a latent; $encode() and $decode() are callable separately.

Causal 3D convolution

Description

Spatial padding is handled by the convolution; temporal padding replicates the first frame (causal) or both edge frames (non-causal), chosen at call time.

Usage

ltx23_causal_conv3d(in_channels, out_channels, kernel_size = 3L, stride = 1L,
                    spatial_padding_mode = "zeros")

Arguments

Value

Module whose forward(hidden_states, causal) returns the 3-D convolution of the input. With causal = TRUE the temporal axis is left-padded by replicating the first frame, so no output frame sees a later input frame.

Summarize checkpoint key coverage

Description

Summarize checkpoint key coverage

Usage

ltx23_census(ckpt)

Arguments

Value

A data.frame with one row per component group and its key count.

1D connector transformer

Description

Replaces padded positions with learnable registers (valid tokens are front-aligned in their original order; the tail is filled with registers indexed by absolute position, after which the attention mask is cleared), then runs 1D transformer blocks with rotary embeddings.

Usage

ltx23_connector_transformer_1d(num_attention_heads = 32L,
                               attention_head_dim = 128L, num_layers = 8L,
                               num_learnable_registers = 128L,
                               rope_base_seq_len = 4096L, rope_theta = 10000,
                               rope_double_precision = TRUE, eps = 1e-06,
                               rope_type = "split", gated_attention = TRUE)

Arguments

Value

Module whose forward(hidden_states, attention_mask, attn_mask_binarize_threshold) returns list(hidden_states, attention_mask): the transformed sequence and the (possibly binarized) mask that accompanies it.

Denormalize latents with the VAE's per-channel statistics

Description

Denormalize latents with the VAE's per-channel statistics

Usage

ltx23_denormalize_latents(latents, latents_mean, latents_std)

Arguments

Value

Denormalized latents ready for the decoder.

Official distilled sigma schedule

Description

The distilled LTX sigma values (with terminal zero appended), as published in the Apache-2.0 diffusers reference (pipelines/ltx2/utils.py).

Usage

ltx23_distilled_sigmas()

Value

Numeric vector of length 9.

Anti-aliasing 1D downsampler (low-pass then stride)

Description

Anti-aliasing 1D downsampler (low-pass then stride)

Usage

ltx23_downsample1d(ratio = 2L, kernel_size = NULL)

Arguments

Value

Module whose forward(x) returns x low-pass filtered and decimated by ratio along the time axis.

Encode audio into normalized, packed conditioning latents

Description

Pads or trims the waveform so the latent length equals audio_num_frames (mel frames 4L - 3, mirroring the decoder's target_frames), computes the log-mel, encodes in argmax mode, packs, and normalizes with the checkpoint statistics.

Usage

ltx23_encode_audio(audio_vae, wav, audio_num_frames, frontend = NULL)

Arguments

Value

Packed normalized latents [1, audio_num_frames, 128] (float32).

Encode pixel frames to normalized video latents

Description

VAE encode in "argmax" mode (the distribution mean), then normalize with the checkpoint's per-channel statistics — the exact inverse of the decode path.

Usage

ltx23_encode_video_frames(vae, frames)

Arguments

Value

Normalized latents [1, 128, F', H/32, W/32] (float32).

LTX feed-forward layer

Description

Linear -> GELU (tanh approximation) -> Linear with 4x hidden dim, matching diffusers FeedForward(activation_fn="gelu-approximate") state-dict names (net.0.proj, net.2).

Usage

ltx23_feed_forward(dim, mult = 4L)

Arguments

Value

Module whose forward(x) returns the projected states passed through a tanh-approximated GELU, a tensor of the same shape as x.

FP8 linear layer

Description

Weight lives as float8_e4m3fn plus a float32 scale in plain module fields (so $to(device) moves only the bias); the forward pass ships 1 byte/param to the input's device, upcasts, rescales, and runs nnf_linear.

Usage

ltx23_fp8_linear(out_features, in_features, bias = TRUE)

Arguments

Value

Module whose forward(x) returns the linear projection of x, with the fp8 weight bytes transferred and cast up to the compute dtype for the matmul. Same result as an nn_linear of the same shape, at a quarter of the resident weight bytes.

Sinusoidal timestep embedding

Description

DDPM-style sinusoidal embedding. LTX uses flip_sin_to_cos=TRUE (cos first) and downscale_freq_shift=0.

Usage

ltx23_get_timestep_embedding(timesteps, embedding_dim, flip_sin_to_cos = TRUE,
                             downscale_freq_shift = 0, max_period = 10000)

Arguments

Value

Tensor [N, embedding_dim].

Test whether a mapped DiT key is in the official fp8 cast set

Description

Test whether a mapped DiT key is in the official fp8 cast set

Usage

ltx23_is_fp8_cast_key(mapped_key)

Arguments

Value

Logical vector.

Kaiser sinc low-pass filter kernel

Description

Kaiser sinc low-pass filter kernel

Usage

ltx23_kaiser_sinc_filter1d(cutoff, half_width, kernel_size)

Arguments

Value

Tensor [kernel_size].

LTX-2.3 latent upsampler model

Description

Latents [B, 128, F, H, W] -> [B, 128, F, 2H, 2W].

Usage

ltx23_latent_upsampler(in_channels = 128L, mid_channels = 1024L,
                       num_blocks_per_stage = 4L)

Arguments

Value

Module whose forward(hidden_states) returns the 2x spatially upscaled latent, a tensor with the same batch, channel and frame counts and doubled height and width.

Stream a checkpoint key group into a module

Description

Reads tensors one at a time from an open checkpoint and copies them into the matching parameters/buffers of module. Destination names are derived by map_key; $copy_() handles any dtype/device conversion, so the module may already live on its target device in its target dtype.

Usage

ltx23_load_group(ckpt, keys, module, map_key = identity, verbose = TRUE,
                 gc_every = 50L)

Arguments

Value

Invisibly, a list with unmapped (checkpoint keys that found no destination), skipped (keys the mapper declined), and unfilled (module parameters/buffers never written). A perfectly loaded group has zero unmapped and zero unfilled.

Load the LTX-2.3 generation components from a single-file checkpoint

Description

Builds the transformer, connectors, video VAE, audio VAE, and vocoder with the LTX 2.3 configuration and streams the checkpoint weights into them. The Gemma3 text encoder ships separately (see load_gemma3_text_encoder).

Usage

ltx23_load_pipeline(checkpoint_path, device = "cuda", dtype = "bfloat16",
                    transformer_device = "cpu",
                    components = c("dit", "connectors", "vae", "audio_vae", "vocoder"),
                    pin = TRUE, attn_chunk = NULL, phase_offload = TRUE,
                    verbose = TRUE)

Arguments

Value

A list with the loaded modules and the checkpoint config, class ltx23_pipeline.

Load the LTX-2.3 transformer with FP8 weights

Description

Builds the transformer, swaps the official cast-set linears for ltx23_fp8_linear, loads fp8 weights CPU-side (optionally pinned) and everything else as bfloat16 on device. Sets options(diffuseR.block_gc = TRUE) so the transformer runs per-block garbage collection over the dequantized temporaries.

Usage

ltx23_load_transformer_fp8(ckpt, device = "cuda", pin = TRUE, verbose = TRUE,
                           ...)

Arguments

Value

The loaded ltx23_transformer.

Load the LTX-2.3 transformer with resident NF4 weights

Description

Builds the transformer, swaps the cast-set linears for ltx23_nf4_linear, and loads everything onto device: at ~4.5 bits/parameter the whole 22B transformer stays GPU-resident, avoiding per-step weight transfers.

Usage

ltx23_load_transformer_nf4(ckpt, device = "cuda", verbose = TRUE, ...)

Arguments

Value

The loaded ltx23_transformer.

Load the LTX-2.3 spatial upscaler weights

Description

The checkpoint keys match this module tree directly.

Usage

ltx23_load_upsampler(path, device = "cuda", dtype = "bfloat16", verbose = TRUE)

Arguments

Value

The loaded ltx23_latent_upsampler.

Map an official audio VAE checkpoint key to the R module name

Description

Map an official audio VAE checkpoint key to the R module name

Usage

ltx23_map_audio_vae_key(key)

Arguments

Value

Character.

Map an official connectors checkpoint key to the R module name

Description

Map an official connectors checkpoint key to the R module name

Usage

ltx23_map_connector_key(key)

Arguments

Value

Character. Module parameter name.

Map an official DiT checkpoint key to the R module name

Description

Applies the official-to-diffusers renames for the LTX-2.3 transformer (cf. diffusers scripts/convert_ltx2_to_diffusers.py). Our module tree matches the diffusers names, so this is the full mapping.

Usage

ltx23_map_dit_key(key)

Arguments

Value

Character. Module parameter/buffer name.

Map an official VAE checkpoint key to the R module name

Description

The official checkpoint stores the encoder/decoder as flat block lists (down_blocks.0-8 / up_blocks.0-8) where downsamplers/upsamplers and the mid block are separate entries; diffusers (and this port) nest them. Index mapping per diffusers convert_ltx2_to_diffusers.py.

Usage

ltx23_map_vae_key(key)

Arguments

Value

Character. Module parameter/buffer name.

Map an official vocoder checkpoint key to the R module name

Description

Map an official vocoder checkpoint key to the R module name

Usage

ltx23_map_vocoder_key(key)

Arguments

Value

Character. Module parameter/buffer name.

Causal log-mel spectrogram with checkpoint-loaded bases

Description

Causal log-mel spectrogram with checkpoint-loaded bases

Usage

ltx23_mel_stft(filter_length = 512L, hop_length = 80L, window_length = 512L,
               num_mel_channels = 64L)

Arguments

Value

Module whose forward(waveform) returns the log-mel spectrogram [B, n_mels, frames], clamped at 1e-5 before the log.

Get an LTX-2.3 memory profile

Description

Selects transformer precision, component placement, and attention chunking for the available VRAM. Measured on an RTX 5060 Ti (16 GB): fp8 streaming peaks ~11.6 GB (without phase offloading) at 512x320x49; NF4 keeps the whole 22B transformer resident (~12.5 GB) and removes the ~21 GB/step PCIe weight streaming. The NF4 profile renders 1280x704x121 with audio in ~23 min at a 15.7 GB peak (tiled VAE decode, in-place feed-forward GELU, and the default diffuseR.attn_budget of 1.5e8 all required at that size).

Usage

ltx23_memory_profile(vram_gb = NULL)

Arguments

Details

precision "nf4"

Weights resident on the GPU; fastest steps; about 9 percent weight round-trip error.

precision "fp8"

Weights CPU-resident, streamed per linear; near-bf16 quality; each step pays the PCIe transfer.

Value

Named list with device/dtype placement, attn_chunk, pin_weights, and resolution caps.

Dequantize NF4 data to a float tensor

Description

Dequantize NF4 data to a float tensor

Usage

ltx23_nf4_dequantize(packed, absmax, shape, dtype = torch::torch_bfloat16(),
                     chunk_elements = 8388608L, out = NULL)

Arguments

Value

Tensor of shape in dtype on the input's device.

NF4 linear layer

Description

Packed weights and per-block scales are registered as buffers, so they move with the module (uint8 packs are untouched by dtype conversions). The forward pass dequantizes on the weight's device.

Usage

ltx23_nf4_linear(out_features, in_features, bias = TRUE)

Arguments

Value

Module whose forward(x) returns the linear projection of x, dequantizing the NF4 weight into a reusable buffer first. Same result as an nn_linear of the same shape, at roughly an eighth of the resident weight bytes.

Quantize a tensor to NF4

Description

Quantize a tensor to NF4

Usage

ltx23_nf4_quantize(x)

Arguments

Value

List with packed (uint8, two indices per byte) and absmax (float32, one per 64-element block).

Normalize latents with the VAE's per-channel statistics

Description

Normalize latents with the VAE's per-channel statistics

Usage

ltx23_normalize_latents(latents, latents_mean, latents_std)

Arguments

Value

Normalized latents.

Open an LTX-2.3 checkpoint

Description

Opens a single-file LTX checkpoint lazily (header only), validates the model_version metadata, and parses the embedded component configuration.

Usage

ltx23_open_checkpoint(path, require_version = "2.3")

Arguments

Value

An object of class ltx23_checkpoint: a list with handle (safetensors reader), keys, version, config (parsed component configs, or NULL), and path.

Examples

## Not run: 
ckpt <- ltx23_open_checkpoint("ltx-2.3-22b-distilled-1.1.safetensors")
str(ltx23_split_keys(ckpt$keys), max.level = 1)

## End(Not run)

Open an FP8 shard directory as a checkpoint

Description

Presents the sharded fp8 artifact through the same interface as ltx23_open_checkpoint so the group loaders work unchanged.

Usage

ltx23_open_fp8_checkpoint(dir)

Arguments

Value

An ltx23_checkpoint.

Per-channel RMS normalization

Description

Normalizes by the root-mean-square across the channel dimension (dim 2 of [B, C, F, H, W]); no learned parameters.

Usage

ltx23_per_channel_rms_norm(eps = 1e-08)

Arguments

Value

Module whose forward(x) returns x divided by its per-channel root mean square, a tensor of the same shape.

Per-token RMS normalization over the channel axis

Description

Per-token RMS normalization over the channel axis

Usage

ltx23_per_token_rms_norm(x, eps = 1e-06)

Arguments

Value

Tensor of the same shape.

Build conditioned initial latents and the conditioning mask

Description

i2v (cond_latents has one latent frame): the encoded frame is repeated across all latent frames and only latent frame 0 is marked conditioned. Continuation (k latent frames): the prefix tokens are replaced and marked. Unconditioned positions start as pure noise.

Usage

ltx23_prepare_conditioned_latents(cond_latents, latent_frames, latent_height,
                                  latent_width, noise, cond_noise_scale = 0)

Arguments

Value

list(latents [1, S, 128] float32 packed, conditioning_mask [1, S] float32 packed).

Preprocess an image (or frame stack) for VAE encoding

Description

Mirrors the diffusers VideoProcessor: bilinear resize so the shorter relative side matches, center-crop to the exact target, and scale to [-1, 1].

Usage

ltx23_preprocess_frames(x, height, width)

Arguments

Value

Float32 tensor [1, 3, F, height, width] in [-1, 1].

Quantize an LTX-2.3 checkpoint to FP8 shards

Description

Streams the single-file bf16 checkpoint tensor by tensor. DiT attention/FFN linear weights are stored as float8_e4m3fn with a float32 absmax/448 per-tensor scale (<key>_scale sibling); everything else is copied through unchanged. Output shards carry the original key names plus a manifest for skip-if-exists.

Usage

ltx23_quantize_fp8(checkpoint_path, output_dir = NULL, shard_bytes = 1.9e+09,
                   force = FALSE, verbose = TRUE)

Arguments

Value

Invisibly, the manifest list.

Quantize an LTX-2.3 checkpoint to NF4 shards

Description

Same streaming layout and cast policy as ltx23_quantize_fp8, but cast-set weights are stored as NF4 (<key> packed uint8 + <key>_absmax float32 blocks + the original shape recovered from the model config at load time). Non-cast tensors are copied through unchanged. The manifest carries format = "nf4".

Usage

ltx23_quantize_nf4(checkpoint_path, output_dir = NULL, shard_bytes = 1.9e+09,
                   force = FALSE, verbose = TRUE)

Arguments

Value

Invisibly, the manifest list.

Read an audio file as 16 kHz stereo PCM

Description

Decodes MP3/WAV/etc. via av to 16-bit PCM at the target rate and parses the RIFF container in base R.

Usage

ltx23_read_audio(path, sample_rate = 16000L)

Arguments

Value

Matrix [2, samples] in [-1, 1].

Read the trailing frames of a video file

Description

Extracts the last n frames of an MP4 (via av) for use as continuation conditioning.

Usage

ltx23_read_tail_frames(path, n = 9L)

Arguments

Value

Array [n, H, W, 3] in [0, 1].

Release the NF4 dequantization buffers

Description

Frees the cached per-shape weight buffers (e.g. before decoding at high resolution).

Usage

ltx23_release_dequant_buffers()

Value

Invisibly, NULL.

RMS normalization

Description

Variance is computed in float32; the result is cast back to the input dtype (or the weight dtype when elementwise affine).

Usage

ltx23_rms_norm(dim, eps = 1e-06, elementwise_affine = TRUE)

Arguments

Value

Module whose forward(x) returns x RMS-normalized over the last axis and cast back to the input dtype, a tensor of the same shape.

LTX-2.3 audio/video rotary position embedder

Description

Computes RoPE cos/sin frequency tensors from spatiotemporal patch coordinates. Video coordinates are 3D (frames scaled to seconds via fps, height, width in pixel space); audio coordinates are 1D (seconds). Coordinates are patch boundaries [start, end); the midpoint is used as the position.

Usage

ltx23_rotary_pos_embed(dim, patch_size = 1L, patch_size_t = 1L,
                       base_num_frames = 20L, base_height = 2048L,
                       base_width = 2048L, sampling_rate = 16000L,
                       hop_length = 160L, scale_factors = c(8L, 32L, 32L),
                       theta = 10000, causal_offset = 1L, modality = "video",
                       double_precision = TRUE, rope_type = "split",
                       num_attention_heads = 32L)

Arguments

Value

Module whose forward(coords, device) returns list(cos_freqs, sin_freqs), the two rotary tables to apply to queries and keys.

1D rotary embeddings for the text connectors

Description

1D rotary embeddings for the text connectors

Usage

ltx23_rotary_pos_embed_1d(dim, base_seq_len = 4096L, theta = 10000,
                          double_precision = TRUE, rope_type = "split",
                          num_attention_heads = 32L)

Arguments

Value

Module whose forward(batch_size, pos, device) returns list(cos_freqs, sin_freqs), the 1-D rotary tables for a sequence of length pos.

Set the attention query-chunk size across a transformer

Description

R torch has no fused attention, so the [B, H, S, S] matrix materializes; chunking queries bounds the peak. NULL disables chunking.

Usage

ltx23_set_attn_chunk(transformer, chunk)

Arguments

Value

Invisibly, the transformer.

SnakeBeta activation

Description

x + (1 / (beta + eps)) * sin(x * alpha)^2 with per-channel log-scale alpha/beta parameters.

Usage

ltx23_snake_beta(channels, eps = 1e-09)

Arguments

Value

Module whose forward(hidden_states) returns the Snake activation x + sin(alpha * x)^2 / beta, a tensor of the same shape as the input.

Split checkpoint keys by component

Description

Splits the flat key space of an LTX single-file checkpoint into its component groups. Connector tensors live under the model.diffusion_model. prefix alongside the transformer, plus a top-level text_embedding_projection. group; both are routed to the connectors component.

Usage

ltx23_split_keys(keys)

Arguments

Value

Named list of character vectors: dit, connectors, vae, audio_vae, vocoder, and other (anything unrecognized; should be empty).

Stage-2 distilled sigma schedule (two-stage refinement)

Description

Stage-2 distilled sigma schedule (two-stage refinement)

Usage

ltx23_stage2_distilled_sigmas()

Value

Numeric vector of length 4.

Slice the trailing latent frames of a generation for chaining

Description

Cuts the last k latent frames out of a result's video latents, in the [1, 128, k, H', W'] layout that txt2vid_ltx2(condition_latents = ) consumes, so one chunk can seed the next without leaving latent space: no decode, no re-encode, no video round-trip.

Usage

ltx23_tail_latents(result, k = 2L, latent_shape = NULL)

Arguments

Details

Semantics caveat: a latent frame sliced from inside a sequence represents 8 pixel frames, while a fresh VAE encode of a k-frame tail represents 1 + 8(k - 1) pixel frames with its first latent in first-frame form. The frozen prefix the next generation sees is therefore not identical to the pixel-path prefix; compare both on real content before relying on latent-only joins.

Value

Normalized latents [1, 128, k, H', W'] (float32), ready for txt2vid_ltx2(condition_latents = ).

LTX-2.3 text connectors

Description

Takes raw stacked per-layer text encoder hidden states and produces the video and audio text embeddings for the DiT: per-token RMS norm, per-modality sqrt(dim ratio) rescaling and projection, then a per-modality 1D connector transformer.

Usage

ltx23_text_connectors(caption_channels = 3840L, text_proj_in_factor = 49L,
                      video_connector_num_attention_heads = 32L,
                      video_connector_attention_head_dim = 128L,
                      video_connector_num_layers = 8L,
                      video_connector_num_learnable_registers = 128L,
                      video_gated_attn = TRUE,
                      audio_connector_num_attention_heads = 32L,
                      audio_connector_attention_head_dim = 64L,
                      audio_connector_num_layers = 8L,
                      audio_connector_num_learnable_registers = 128L,
                      audio_gated_attn = TRUE,
                      connector_rope_base_seq_len = 4096L, rope_theta = 10000,
                      rope_double_precision = TRUE, rope_type = "split",
                      video_hidden_dim = 4096L, audio_hidden_dim = 2048L,
                      proj_bias = TRUE)

Arguments

Value

Module whose forward(text_encoder_hidden_states, attention_mask) returns list(video_text_embedding, audio_text_embedding, attention_mask): the caption states adapted for the video and audio cross-attention streams, plus the binary mask to use with them.

Sigmoid tone mapping for latents

Description

Compresses the latent dynamic range (cf. diffusers tone_map_latents). compression 0 is identity, 1 is the full effect.

Usage

ltx23_tone_map_latents(latents, compression)

Arguments

Value

Tone-mapped latents.

LTX-2.3 video transformer model

Description

Dual-stream audio/video DiT. Text embeddings arrive already projected to the video (inner_dim) and audio (audio_inner_dim) dimensions by the connector modules.

Usage

ltx23_transformer(in_channels = 128L, out_channels = 128L, patch_size = 1L,
                  patch_size_t = 1L, num_attention_heads = 32L,
                  attention_head_dim = 128L, cross_attention_dim = 4096L,
                  vae_scale_factors = c(8L, 32L, 32L), pos_embed_max_pos = 20L,
                  base_height = 2048L, base_width = 2048L, gated_attn = TRUE,
                  cross_attn_mod = TRUE, audio_in_channels = 128L,
                  audio_out_channels = 128L, audio_patch_size = 1L,
                  audio_patch_size_t = 1L, audio_num_attention_heads = 32L,
                  audio_attention_head_dim = 64L,
                  audio_cross_attention_dim = 2048L, audio_scale_factor = 4L,
                  audio_pos_embed_max_pos = 20L, audio_sampling_rate = 16000L,
                  audio_hop_length = 160L, audio_gated_attn = TRUE,
                  audio_cross_attn_mod = TRUE, num_layers = 48L,
                  norm_eps = 1e-06, rope_theta = 10000,
                  rope_double_precision = TRUE, causal_offset = 1L,
                  timestep_scale_multiplier = 1000,
                  cross_attn_timestep_scale_multiplier = 1000,
                  rope_type = "split", perturbed_attn = TRUE)

Arguments

Value

Module whose forward(hidden_states, ...) returns list(sample, audio_sample): the predicted velocity for the video latent tokens and, when the audio branch is active, for the audio latent tokens (audio_sample is NULL otherwise).

LTX-2 transformer block

Description

Dual-stream (video + audio) block: modulated self-attention per modality, text cross-attention per modality (with LTX-2.3 query and key/value modulation), bidirectional audio-video cross-attention with global+per-block modulation, and modulated feed-forward.

Usage

ltx23_transformer_block(dim, num_attention_heads, attention_head_dim,
                        cross_attention_dim, audio_dim,
                        audio_num_attention_heads, audio_attention_head_dim,
                        audio_cross_attention_dim, video_gated_attn = TRUE,
                        video_cross_attn_adaln = TRUE, audio_gated_attn = TRUE,
                        audio_cross_attn_adaln = TRUE, eps = 1e-06,
                        elementwise_affine = FALSE, rope_type = "split",
                        perturbed_attn = TRUE)

Arguments

Value

Module whose forward(hidden_states, ...) returns list(hidden_states, audio_hidden_states), the video and audio streams after self-attention, cross-attention and the feed-forward, each the same shape as its input.

Tune the torch CUDA allocator for large-resident inference

Description

Stops the allocator GC storm (cf. ~/skills/torch torch-jit-gc-performance.md): lantern proactively calls R's gc() whenever reserved memory exceeds torch.cuda_allocator_reserved_rate (default 0.20) of the card. With ~75\ weights that fires on nearly every allocation. Raising the rate to the actual footprint is safe here because the LTX hot loops compute into persistent scratch buffers (near-zero per-step garbage). Also raises the host-allocation GC threshold and defaults PYTORCH_CUDA_ALLOC_CONF to expandable segments. User-set options win.

Usage

ltx23_tune_gc(footprint_gb = 12, total_gb = NULL)

Arguments

Details

start_torch() reads the gate options exactly once, so setting them after torch has started is inert on its own. The three CUDA gates are therefore also pushed into the live allocator here (the .flux_gc_gates pattern), which makes this function effective whenever it runs. The host-side torch.threshold_call_gc has no live setter; the package defaults it in .onLoad so torch reads it at init in any session that loads diffuseR before running torch ops.

Value

Invisibly, the applied reserved rate (NULL if skipped).

Anti-aliasing 1D upsampler (transposed low-pass)

Description

Anti-aliasing 1D upsampler (transposed low-pass)

Usage

ltx23_upsample1d(ratio = 2L, kernel_size = NULL, window_type = "kaiser",
                 persistent = TRUE)

Arguments

Value

Module whose forward(x) returns x interpolated up by ratio along the time axis, with the filter padding trimmed off.

LTX-2.3 video decoder

Description

Latents [B, 128, F, H, W] -> pixel video [B, 3, 8F - 7, 32H, 32W]. Block channel lists are given encoder-side (as in the config) and reversed internally; upsample_type is indexed directly.

Usage

ltx23_video_decoder3d(in_channels = 128L, out_channels = 3L,
                      block_out_channels = c(256L, 512L, 512L, 1024L),
                      spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE),
                      layers_per_block = c(4L, 6L, 4L, 2L, 2L),
                      upsample_type = NULL, patch_size = 4L, patch_size_t = 1L,
                      resnet_norm_eps = 1e-06, is_causal = FALSE,
                      upsample_residual = c(FALSE, FALSE, FALSE, FALSE),
                      upsample_factor = c(2L, 2L, 1L, 2L),
                      spatial_padding_mode = "zeros")

Arguments

Value

Module whose forward(hidden_states, causal) returns the decoded pixel tensor [B, 3, F, H, W], with the final patch axes flattened back into height and width.

LTX video down block

Description

ResNet stack (at the input channel count) followed by a pixel-unshuffle downsampler that also changes the channel count.

Usage

ltx23_video_down_block3d(in_channels, out_channels = NULL, num_layers = 1L,
                         resnet_eps = 1e-06, spatio_temporal_scale = TRUE,
                         downsample_type = "spatiotemporal",
                         spatial_padding_mode = "zeros")

Arguments

Value

Module whose forward(hidden_states, causal) returns the stage output: the resnet stack applied in sequence, then the optional downsampler.

Pixel-unshuffle 3D downsampler

Description

Conv followed by space/time-to-channel rearrangement, plus a grouped channel-mean residual of the same rearrangement.

Usage

ltx23_video_downsampler3d(in_channels, out_channels, stride = c(1L, 1L, 1L),
                          spatial_padding_mode = "zeros")

Arguments

Value

Module whose forward(hidden_states, causal) returns the space-to-depth downsampled tensor plus its residual: spatial and temporal extents shrink by stride, channels grow to match.

LTX-2.3 video encoder

Description

Pixel video [B, 3, F, H, W] -> latent statistics [B, 2 * latent_channels, F/8, H/32, W/32] (mean and a uniform log-var channel broadcast across the latent channels).

Usage

ltx23_video_encoder3d(in_channels = 3L, out_channels = 128L,
                      block_out_channels = c(256L, 512L, 1024L, 1024L),
                      spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE),
                      layers_per_block = c(4L, 6L, 4L, 2L, 2L),
                      downsample_type = NULL, patch_size = 4L,
                      patch_size_t = 1L, resnet_norm_eps = 1e-06,
                      is_causal = TRUE, spatial_padding_mode = "zeros")

Arguments

Value

Module whose forward(hidden_states, causal) returns the encoded video latent [B, C, F, H, W], with the last channel repeated to carry the per-channel scale expected downstream.

LTX video mid block

Description

A plain ResNet stack at a fixed channel count.

Usage

ltx23_video_mid_block3d(in_channels, num_layers = 1L, resnet_eps = 1e-06,
                        spatial_padding_mode = "zeros")

Arguments

Value

Module whose forward(hidden_states, causal) returns the bottleneck output, a tensor of the same shape as the input.

LTX 3D ResNet block

Description

PerChannelRMSNorm -> SiLU -> causal conv, twice, with a LayerNorm + 1x1 Conv3d shortcut when the channel count changes.

Usage

ltx23_video_resnet_block3d(in_channels, out_channels = NULL, eps = 1e-06,
                           spatial_padding_mode = "zeros")

Arguments

Value

Module whose forward(inputs, causal) returns inputs plus the residual branch, a tensor of the same shape.

LTX video up block

Description

Optional channel-changing conv-in ResNet, pixel-shuffle upsampler, then a ResNet stack at the output channel count.

Usage

ltx23_video_up_block3d(in_channels, out_channels = NULL, num_layers = 1L,
                       resnet_eps = 1e-06, spatio_temporal_scale = TRUE,
                       upsample_type = "spatiotemporal",
                       upsample_residual = FALSE, upscale_factor = 1L,
                       spatial_padding_mode = "zeros")

Arguments

Value

Module whose forward(hidden_states, causal) returns the stage output: the optional input projection and upsampler, then the resnet stack applied in sequence.

Pixel-shuffle 3D upsampler

Description

Conv followed by channel-to-space/time rearrangement, with an optional channel-repeat residual and an upscale factor that divides the conv output channels.

Usage

ltx23_video_upsampler3d(in_channels, stride = c(1L, 1L, 1L), residual = FALSE,
                        upscale_factor = 1L, spatial_padding_mode = "zeros")

Arguments

Value

Module whose forward(hidden_states, causal) returns the depth-to-space upsampled tensor: spatial and temporal extents grow by stride, channels shrink to match.

LTX-2.3 video VAE

Description

Encoder + decoder + per-channel latent statistics (loaded from the checkpoint's per_channel_statistics). The checkpoint's scaling_factor is 1.0, so latent (de)normalization is purely the per-channel affine map.

Usage

ltx23_video_vae(in_channels = 3L, out_channels = 3L, latent_channels = 128L,
                block_out_channels = c(256L, 512L, 1024L, 1024L),
                decoder_block_out_channels = c(256L, 512L, 512L, 1024L),
                layers_per_block = c(4L, 6L, 4L, 2L, 2L),
                decoder_layers_per_block = c(4L, 6L, 4L, 2L, 2L),
                spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE),
                decoder_spatio_temporal_scaling = c(TRUE, TRUE, TRUE, TRUE),
                downsample_type = NULL, upsample_type = NULL,
                upsample_residual = c(FALSE, FALSE, FALSE, FALSE),
                upsample_factor = c(2L, 2L, 1L, 2L), patch_size = 4L,
                patch_size_t = 1L, resnet_norm_eps = 1e-06,
                encoder_causal = TRUE, decoder_causal = FALSE,
                encoder_spatial_padding_mode = "zeros",
                decoder_spatial_padding_mode = "zeros")

Arguments

Value

Module bundling the video encoder and decoder. Its forward(z) is decode(z), returning pixels for a latent; $encode() and $decode() are callable separately.

LTX-2.3 vocoder stage

Description

Mel spectrogram [B, C, T, M] -> waveform [B, out_channels, samples]. Channel and mel dims are flattened into conv channels; each upsample stage halves the channel count and averages three parallel ResNet branches.

Usage

ltx23_vocoder(in_channels = 128L, hidden_channels = 1536L, out_channels = 2L,
              upsample_kernel_sizes = c(11L, 4L, 4L, 4L, 4L, 4L),
              upsample_factors = c(5L, 2L, 2L, 2L, 2L, 2L),
              resnet_kernel_sizes = c(3L, 7L, 11L),
              resnet_dilations = list(c(1L, 3L, 5L), c(1L, 3L, 5L), c(1L, 3L, 5L)),
              antialias_ratio = 2L, antialias_kernel_size = 12L,
              final_bias = FALSE)

Arguments

Value

Module whose forward(hidden_states, time_last) returns the synthesized waveform [B, 1, samples] for a mel spectrogram.

Vocoder ResNet block (AMP)

Description

Dilated conv pairs, each preceded by an anti-aliased SnakeBeta activation, with residual connections.

Usage

ltx23_vocoder_resblock(channels, kernel_size = 3L, dilations = c(1L, 3L, 5L),
                       antialias_ratio = 2L, antialias_kernel_size = 12L)

Arguments

Value

Module whose forward(x) returns x after the dilated convolution pairs have been added back as residuals, a tensor of the same shape.

LTX-2.3 vocoder with bandwidth extension

Description

Full mel [B, 2, T, 64] -> 48 kHz stereo waveform pipeline: 16 kHz vocoder, causal mel re-analysis, BWE vocoder residual added to a Hann-resampled skip path, clamped to [-1, 1].

Usage

ltx23_vocoder_with_bwe(in_channels = 128L, hidden_channels = 1536L,
                       out_channels = 2L,
                       upsample_kernel_sizes = c(11L, 4L, 4L, 4L, 4L, 4L),
                       upsample_factors = c(5L, 2L, 2L, 2L, 2L, 2L),
                       resnet_kernel_sizes = c(3L, 7L, 11L),
                       resnet_dilations = NULL, bwe_in_channels = 128L,
                       bwe_hidden_channels = 512L,
                       bwe_upsample_kernel_sizes = c(12L, 11L, 4L, 4L, 4L),
                       bwe_upsample_factors = c(6L, 5L, 2L, 2L, 2L),
                       bwe_resnet_kernel_sizes = c(3L, 7L, 11L),
                       bwe_resnet_dilations = NULL, filter_length = 512L,
                       hop_length = 80L, window_length = 512L,
                       num_mel_channels = 64L, input_sampling_rate = 16000L,
                       output_sampling_rate = 48000L)

Arguments

Value

Module whose forward(mel_spec) returns the bandwidth-extended waveform [B, 1, samples], trimmed to the sample count implied by the input frames and the rate ratio.

FLUX Memory Profiles

Description

VRAM-based execution profiles for the FLUX.1-schnell pipeline. The 12B transformer runs NF4 (~7 GB) or fp8 (~12 GB), phase-onloaded to the GPU for denoise; the T5-XXL text encoder phase-onloads to the GPU (bfloat16, pinned) on 14 GB+ cards and computes on the CPU (float32) below that, where its ~9.8 GB encode phase does not fit.

LTX-2.3 Memory Profiles and CUDA GC Tuning

Description

Memory management for running the 22B LTX-2.3 transformer on limited VRAM, built on the patterns proven in the whisper and chatterbox packages: torch allocator GC tuning before the first CUDA op, fp8 CPU-resident streaming weights, query-chunked attention, and phase-sequential component placement.

models2devices

Description

This function sets up the model directory, device configuration, and data types for diffusion models. It checks the validity of the model name and devices, detects model type, and downloads the model if necessary.

Usage

models2devices(model_name, devices = "cpu", unet_dtype_str = NULL,
               download_models = FALSE)

Arguments

Value

A list containing the device configuration, UNet data type, and CPU/CUDA devices.

NF4 Weight Storage for the LTX-2.3 Transformer

Description

4-bit NormalFloat quantization (the QLoRA scheme: per-block absmax normalization against a 16-level quantile code, two indices packed per byte). At ~4.5 bits/parameter the 22B transformer fits in about 12.5 GB, small enough to stay resident on a 16 GB GPU: no per-step PCIe weight streaming, at a small quality cost relative to fp8. Quantization and dequantization are pure torch ops (bucketize, index_select) - no custom kernels.

Offload Module to CPU

Description

Moves a torch module and all its parameters to CPU.

Usage

offload_to_cpu(module, gc = TRUE)

Arguments

Value

The module (modified in place).

Examples

if (torch::torch_is_installed()) {
  model <- torch::nn_linear(4, 2)
  offload_to_cpu(model)
}

Post Quant Conv

Description

This function applies a quantized convolution operation to an input tensor. It is typically used in the context of image post processing, particularly in generative models like Stable Diffusion XL.

Usage

post_quant_conv(x, dtype, device)

Arguments

Value

Processed tensor after applying the quantized convolution.

Preprocess image for Stable Diffusion

Description

Preprocess image for Stable Diffusion

Usage

preprocess_image(input, device = "cpu", width = 512, height = 512)

Arguments

Value

Torch tensor of shape c(1, 3, 512, 512), scaled to c(-1, 1)

Print BPE Tokenizer

Description

Print BPE Tokenizer

Usage

## S3 method for class 'bpe_tokenizer'
print(x, ...)

Arguments

Value

Invisibly returns x. Called for the side effect of printing a summary of the tokenizer to the console.

Print a resident handle

Description

Print a resident handle

Usage

## S3 method for class 'diffuseR_resident'
print(x, ...)

Arguments

Value

Invisibly x. Called for the side effect of printing a one-block summary to the console.

Quant Conv

Description

This function applies a quantized convolution operation to an input tensor. It is typically used in the context of image processing, particularly in generative models like Stable Diffusion.

Usage

quant_conv(x, dtype, device)

Arguments

Value

Processed tensor after applying the quantized convolution.

FLUX Transformer Quantization and Loading

Description

Quantize the 12B FLUX transformer to NF4 (~7 GB, GPU-resident on 16 GB cards) or fp8 (~12 GB, CPU-resident and streamed per forward), and load any format back into flux_transformer. Reuses the LTX-2.3 quantization machinery (ltx23_nf4_quantize, ltx23_nf4_linear, ltx23_fp8_linear); only the cast set and the diffusers directory layout are FLUX-specific.

QuickGELU activation

Description

GELU approximation used by OpenAI CLIP: x * sigmoid(1.702 * x)

Usage

quick_gelu(x)

Arguments

Qwen3 encoder stack

Description

Defaults are the Qwen3-4B configuration used by FLUX.2 klein. The module tree mirrors the checkpoint keys (model.embed_tokens, model.layers.*, model.norm); the tied LM head carries no weights of its own and is not implemented.

Usage

qwen3_encoder(vocab_size = 151936L, hidden_size = 2560L,
              intermediate_size = 9728L, num_hidden_layers = 36L,
              num_attention_heads = 32L, num_key_value_heads = 8L,
              head_dim = 128L, rope_theta = 1e+06, rms_norm_eps = 1e-06)

Arguments