maxtext.multimodal.utils module

General utility functions for multimodal processing.

class maxtext.multimodal.utils.PreprocessorOutput(pixel_values=None, pixel_mask=None, aspect_ratios=None, num_images=0, audio_values=None, audio_mask=None)

Bases: object

Holds the output of a multimodal preprocessor.

Parameters:

• pixel_values (None | ndarray) – A JAX array containing the processed image pixel data. The shape and format depend on the specific model and preprocessing steps (e.g., [H, W, C] for Gemma3 or [NUM_TILES, C, TILE_SIZE, TILE_SIZE] for Llama4).

• pixel_mask (None | ndarray) – An optional JAX array of shape (NUM_TILES,) indicating valid tiles in the image.

• aspect_ratios (None | ndarray) – An optional JAX array of shape (batch_size, 2) representing the aspect ratio [ratio_h, ratio_w] of the processed image(s). This is particularly relevant for models like Llama4 that handle images by tiling.

• num_images (int) – Number of images in the output.

• audio_values (None | ndarray) – An optional array containing processed audio features.

• audio_mask (None | ndarray) – An optional array indicating valid audio segments.

pixel_values: None | ndarray = None

pixel_mask: None | ndarray = None

aspect_ratios: None | ndarray = None

num_images: int = 0

audio_values: None | ndarray = None

audio_mask: None | ndarray = None

maxtext.multimodal.utils.convert_to_RGB(image)

Convert image to RGB format.

maxtext.multimodal.utils.load_image_from_path(image_path)

Loads an image from a given file path and returns a np.array.

maxtext.multimodal.utils.normalize_images(images, mean, std)

Normalize the image by subtracting mean and dividing by std.

Parameters:

• images – The images to normalize (np.ndarray).

• mean – Scalar float or tuple/array of floats for per-channel normalization. E.g., 127.5 or (127.5, 127.5, 127.5) for RGB.

• std – Scalar float or tuple/array of floats for per-channel normalization. E.g., 127.5 or (127.5, 127.5, 127.5) for RGB.

Returns:

The normalized images (modifies input in-place and returns it).

maxtext.multimodal.utils.merge_mm_embeddings(text_embeddings, multimodal_embeddings, mask, token_masks=None)

Merges text and multimodal (vision/audio) embeddings based on a mask.

This function handles two primary formats for multimodal embeddings: 1. Tiled Format (e.g., Llama4 vision): Embeddings are provided as a batch of

images and their tiles, with shape (B * N, T, K, D). These are flattened into a single sequence of tokens per batch item.

2. Simple Format (e.g., Gemma3 vision, Qwen3-Omni audio): Embeddings are provided as (B, N, K, D) and are flattened into a sequence of tokens.

Parameters:

• text_embeddings (ndarray | Array) – (B, S, D) array of text embeddings.

• multimodal_embeddings (ndarray | Array) –

  Multimodal embeddings (vision/audio) in one of two formats: - (B * N, T, K, D) for tiled inputs. - (B, N, K, D) for simple inputs. (B=batch_size, S=seq_len, D=embedding_dim, N=num_images/audio_chunks,

  T=num_tiles, K=toks_per_image/audio_chunk)

• mask – (B, S) boolean or integer array where non-zero positions indicate where multimodal embeddings should be placed.

• token_masks (ndarray | Array | None) – (Optional) A mask for the multimodal tokens. - (B * N, T) for tiled inputs, indicating valid tiles. - If None, all multimodal embeddings are assumed to be valid.

Returns:

A (B, S, D) array of merged embeddings.

Return type:

ndarray | Array

maxtext.multimodal.utils.amplitude_to_db(spectrogram_array, reference=1.0, min_value=1e-05, db_range=None)

Converts an amplitude spectrogram to the decibel scale. This computes 20 * log10(spectrogram / reference), using basic logarithm properties for numerical stability.

The motivation behind applying the log function on the (mel) spectrogram is that humans do not hear loudness on a linear scale. Generally to double the perceived volume of a sound we need to put 8 times as much energy into it. This means that large variations in energy may not sound all that different if the sound is loud to begin with. This compression operation makes the (mel) spectrogram features match more closely what humans actually hear.

Parameters:

• spectrogram (np.ndarray) – The input amplitude (mel) spectrogram.

• reference (float, optional, defaults to 1.0) – Sets the input spectrogram value that corresponds to 0 dB. For example, use np.max(spectrogram) to set the loudest part to 0 dB. Must be greater than zero.

• min_value (float, optional, defaults to 1e-5) – The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking log(0). The default of 1e-5 corresponds to a minimum of -100 dB. Must be greater than zero.

• db_range (float, optional) – Sets the maximum dynamic range in decibels. For example, if db_range = 80, the difference between the peak value and the smallest value will never be more than 80 dB. Must be greater than zero.

• spectrogram_array (ndarray)

Returns:

the spectrogram in decibels

Return type:

np.ndarray

maxtext.multimodal.utils.power_to_db(spectrogram_array, reference=1.0, min_value=1e-10, db_range=None)

Converts a power spectrogram to the decibel scale. This computes 10 * log10(spectrogram / reference), using basic logarithm properties for numerical stability.

The motivation behind applying the log function on the (mel) spectrogram is that humans do not hear loudness on a linear scale. Generally to double the perceived volume of a sound we need to put 8 times as much energy into it. This means that large variations in energy may not sound all that different if the sound is loud to begin with. This compression operation makes the (mel) spectrogram features match more closely what humans actually hear.

Based on the implementation of librosa.power_to_db.

Parameters:

• spectrogram (np.ndarray) – The input power (mel) spectrogram. Note that a power spectrogram has the amplitudes squared!

• reference (float, optional, defaults to 1.0) – Sets the input spectrogram value that corresponds to 0 dB. For example, use np.max(spectrogram) to set the loudest part to 0 dB. Must be greater than zero.

• min_value (float, optional, defaults to 1e-10) – The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking log(0). The default of 1e-10 corresponds to a minimum of -100 dB. Must be greater than zero.

• db_range (float, optional) – Sets the maximum dynamic range in decibels. For example, if db_range = 80, the difference between the peak value and the smallest value will never be more than 80 dB. Must be greater than zero.

• spectrogram_array (ndarray)

Returns:

the spectrogram in decibels

Return type:

np.ndarray

maxtext.multimodal.utils.spectrogram(waveform, window, frame_length, hop_length, fft_length=None, power=1.0, center=True, pad_mode='reflect', onesided=True, dither=0.0, preemphasis=None, mel_filters=None, mel_floor=1e-10, log_mel=None, reference=1.0, min_value=1e-10, db_range=None, remove_dc_offset=False, dtype=<class 'numpy.float32'>)

Calculates a spectrogram over one waveform using the Short-Time Fourier Transform.

This function can create the following kinds of spectrograms:

• amplitude spectrogram (power = 1.0)

• power spectrogram (power = 2.0)

• complex-valued spectrogram (power = None)

• log spectrogram (use log_mel argument)

• mel spectrogram (provide mel_filters)

• log-mel spectrogram (provide mel_filters and log_mel)

How this works:

1. The input waveform is split into frames of size frame_length that are partially overlapping by frame_length - hop_length samples.

2. Each frame is multiplied by the window and placed into a buffer of size fft_length.

3. The DFT is taken of each windowed frame.

4. The results are stacked into a spectrogram.

We make a distinction between the following “blocks” of sample data, each of which may have a different lengths:

• The analysis frame. This is the size of the time slices that the input waveform is split into.

• The window. Each analysis frame is multiplied by the window to avoid spectral leakage.

• The FFT input buffer. The length of this determines how many frequency bins are in the spectrogram.

In this implementation, the window is assumed to be zero-padded to have the same size as the analysis frame. A padded window can be obtained from window_function(). The FFT input buffer may be larger than the analysis frame, typically the next power of two.

Note: This function is not optimized for speed yet. It should be mostly compatible with librosa.stft and torchaudio.functional.transforms.Spectrogram, although it is more flexible due to the different ways spectrograms can be constructed.

Parameters:

• waveform (np.ndarray of shape (length,)) – The input waveform. This must be a single real-valued, mono waveform.

• window (np.ndarray of shape (frame_length,)) – The windowing function to apply, including zero-padding if necessary. The actual window length may be shorter than frame_length, but we’re assuming the array has already been zero-padded.

• frame_length (int) – The length of the analysis frames in samples. With librosa this is always equal to fft_length but we also allow smaller sizes.

• hop_length (int) – The stride between successive analysis frames in samples.

• fft_length (int, optional) – The size of the FFT buffer in samples. This determines how many frequency bins the spectrogram will have. For optimal speed, this should be a power of two. If None, uses frame_length.

• power (float, optional, defaults to 1.0) – If 1.0, returns the amplitude spectrogram. If 2.0, returns the power spectrogram. If None, returns complex numbers.

• center (bool, optional, defaults to True) – Whether to pad the waveform so that frame t is centered around time t * hop_length. If False, frame t will start at time t * hop_length.

• pad_mode (str, optional, defaults to “reflect”) – Padding mode used when center is True. Possible values are: “constant” (pad with zeros), “edge” (pad with edge values), “reflect” (pads with mirrored values).

• onesided (bool, optional, defaults to True) – If True, only computes the positive frequencies and returns a spectrogram containing fft_length // 2 + 1 frequency bins. If False, also computes the negative frequencies and returns fft_length frequency bins.

• dither (float, optional, defaults to 0.0) – Adds dithering. In other words, adds a small Gaussian noise to each frame. E.g. use 4.0 to add dithering with a normal distribution centered around 0.0 with standard deviation 4.0, 0.0 means no dithering. Dithering has similar effect as mel_floor. It reduces the high log_mel_fbank values for signals with hard-zero sections, when VAD cutoff is present in the signal.

• preemphasis (float, optional) – Coefficient for a low-pass filter that applies pre-emphasis before the DFT.

• mel_filters (np.ndarray of shape (num_freq_bins, num_mel_filters), optional) – The mel filter bank. If supplied, applies a this filter bank to create a mel spectrogram.

• mel_floor (float, optional, defaults to 1e-10) – Minimum value of mel frequency banks.

• log_mel (str, optional) – How to convert the spectrogram to log scale. Possible options are: None (don’t convert), “log” (take the natural logarithm) “log10” (take the base-10 logarithm), “dB” (convert to decibels). Can only be used when power is not None.

• reference (float, optional, defaults to 1.0) – Sets the input spectrogram value that corresponds to 0 dB. For example, use np.max(spectrogram) to set the loudest part to 0 dB. Must be greater than zero.

• min_value (float, optional, defaults to 1e-10) – The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking log(0). For a power spectrogram, the default of 1e-10 corresponds to a minimum of -100 dB. For an amplitude spectrogram, the value 1e-5 corresponds to -100 dB. Must be greater than zero.

• db_range (float, optional) – Sets the maximum dynamic range in decibels. For example, if db_range = 80, the difference between the peak value and the smallest value will never be more than 80 dB. Must be greater than zero.

• remove_dc_offset (bool, optional) – Subtract mean from waveform on each frame, applied before pre-emphasis. This should be set to true in order to get the same results as torchaudio.compliance.kaldi.fbank when computing mel filters.

• dtype (np.dtype, optional, defaults to np.float32) – Data type of the spectrogram tensor. If power is None, this argument is ignored and the dtype will be np.complex64.

Returns:

nd.array containing a spectrogram of shape (num_frequency_bins, length) for a regular spectrogram or shape (num_mel_filters, length) for a mel spectrogram.

Return type:

ndarray

maxtext.multimodal.utils.hertz_to_mel(freq, mel_scale='htk')

Convert frequency from hertz to mels.

Parameters:

• freq (float or np.ndarray) – The frequency, or multiple frequencies, in hertz (Hz).

• mel_scale (str, optional, defaults to “htk”) – The mel frequency scale to use, “htk”, “kaldi” or “slaney”.

Returns:

The frequencies on the mel scale.

Return type:

float or np.ndarray

maxtext.multimodal.utils.mel_to_hertz(mels, mel_scale='htk')

Convert frequency from mels to hertz.

Parameters:

• mels (float or np.ndarray) – The frequency, or multiple frequencies, in mels.

• mel_scale (str, optional, “htk”) – The mel frequency scale to use, “htk”, “kaldi” or “slaney”.

Returns:

The frequencies in hertz.

Return type:

float or np.ndarray

maxtext.multimodal.utils.mel_filter_bank(num_frequency_bins, num_mel_filters, min_frequency, max_frequency, sampling_rate, norm=None, mel_scale='htk', triangularize_in_mel_space=False)

Creates a frequency bin conversion matrix used to obtain a mel spectrogram. This is called a mel filter bank, and various implementation exist, which differ in the number of filters, the shape of the filters, the way the filters are spaced, the bandwidth of the filters, and the manner in which the spectrum is warped. The goal of these features is to approximate the non-linear human perception of the variation in pitch with respect to the frequency.

Different banks of mel filters were introduced in the literature. The following variations are supported:

• MFCC FB-20: introduced in 1980 by Davis and Mermelstein, it assumes a sampling frequency of 10 kHz and a speech bandwidth of [0, 4600] Hz.

• MFCC FB-24 HTK: from the Cambridge HMM Toolkit (HTK) (1995) uses a filter bank of 24 filters for a speech bandwidth of [0, 8000] Hz. This assumes sampling rate ≥ 16 kHz.

• MFCC FB-40: from the Auditory Toolbox for MATLAB written by Slaney in 1998, assumes a sampling rate of 16 kHz and speech bandwidth of [133, 6854] Hz. This version also includes area normalization.

• HFCC-E FB-29 (Human Factor Cepstral Coefficients) of Skowronski and Harris (2004), assumes a sampling rate of 12.5 kHz and speech bandwidth of [0, 6250] Hz.

This code is adapted from torchaudio and librosa. Note that the default parameters of torchaudio’s melscale_fbanks implement the “htk” filters while librosa uses the “slaney” implementation.

Parameters:

• num_frequency_bins (int) – Number of frequency bins (should be the same as n_fft // 2 + 1 where n_fft is the size of the Fourier Transform used to compute the spectrogram).

• num_mel_filters (int) – Number of mel filters to generate.

• min_frequency (float) – Lowest frequency of interest in Hz.

• max_frequency (float) – Highest frequency of interest in Hz. This should not exceed sampling_rate / 2.

• sampling_rate (int) – Sample rate of the audio waveform.

• norm (str, optional) – If “slaney”, divide the triangular mel weights by the width of the mel band (area normalization).

• mel_scale (str, optional, defaults to “htk”) – The mel frequency scale to use, “htk”, “kaldi” or “slaney”.

• triangularize_in_mel_space (bool, optional, defaults to False) – If this option is enabled, the triangular filter is applied in mel space rather than frequency space. This should be set to true in order to get the same results as torchaudio when computing mel filters.

Returns:

Triangular filter bank matrix. This is a projection matrix to go from a spectrogram to a mel spectrogram.

Return type:

np.ndarray of shape (num_frequency_bins, num_mel_filters)

maxtext.multimodal.utils.window_function(window_length, name='hann', periodic=True, frame_length=None, center=True)

Returns an array containing the specified window. This window is intended to be used with stft.

The following window types are supported:

• “boxcar”: a rectangular window

• “hamming”: the Hamming window

• “hann”: the Hann window

• “povey”: the Povey window

Parameters:

• window_length (int) – The length of the window in samples.

• name (str, optional, defaults to “hann”) – The name of the window function.

• periodic (bool, optional, defaults to True) – Whether the window is periodic or symmetric.

• frame_length (int, optional) – The length of the analysis frames in samples. Provide a value for frame_length if the window is smaller than the frame length, so that it will be zero-padded.

• center (bool, optional, defaults to True) – Whether to center the window inside the FFT buffer. Only used when frame_length is provided.

Returns:

np.ndarray of shape (window_length,) or (frame_length,) containing the window.

Return type:

ndarray

maxtext.multimodal.utils.load_audio(data_path, sample_rate=16000)

Load audio from a file path.

Parameters:

• data_path (str) – The path to the audio file or video file.

• sample_rate (int) – The target sample rate in Hz. Default is 16000.

Returns:

The loaded audio waveform.

Return type:

np.ndarray

Raises:

• FileNotFoundError – If the audio file does not exist.

• RuntimeError – If the audio file cannot be loaded.