Skip to content

Overview

Frame sampling processor family.

This package provides processors for sampling frames from video streams at specified intervals or based on scene changes.

Exported Classes

FrameSamplingProcessor()

Bases: BackendSelectableProcessor[Attachment, Attachment], ABC

Family base for resampling video attachments to a target frame rate.

This class serves as a unified entry point for frame sampling operations. It automatically routes requests to the most appropriate, available backend implementation based on your system environment.

Why use this base class?

  • Portability: Your code will run regardless of which underlying libraries are installed on the host machine.
  • Simplicity: No need to handle fallback logic or conditional imports yourself.
  • Future-proofing: New backends can be added to the library without requiring changes to your application code.

Usage Example

from gllm_multimodal.media_toolkit.processor.frame_sampling_processor import (
    FrameSamplingProcessor,
    GstFrameSamplingConfig,
)
from gllm_inference.schema import Attachment

# Instantiates the best available backend automatically
processor = FrameSamplingProcessor.build(
    config=GstFrameSamplingConfig(default_target_fps=2)
)

attachment = Attachment(url="file:///path/to/video.mp4")
sampled_video = await processor.process(attachment)
from gllm_multimodal.media_toolkit.processor.frame_sampling_processor import FrameSamplingProcessor
from gllm_inference.schema import Attachment

# Explicitly select the gstreamer backend
processor = FrameSamplingProcessor.build(backend="gstreamer")

attachment = Attachment(url="file:///path/to/video.mp4")
sampled_video = await processor.process(attachment)

GstFrameSamplingConfig

Bases: GstBaseConfig

Stable configuration for GstFrameSamplingProcessor.

Attributes:

Name Type Description
min_size_mb int

Minimum video size in MB to process. 0 disables the check. Defaults to 0.

default_target_fps int

Default target FPS used when GstFrameSamplingProcessor.process is called without a process_config. Must be greater than 0. Defaults to 1.

validate_default_target_fps(default_target_fps) classmethod

Validate that default_target_fps is positive.

Parameters:

Name Type Description Default
default_target_fps int

The candidate default FPS to validate.

required

Returns:

Name Type Description
int int

default_target_fps unchanged when valid.

Raises:

Type Description
ValueError

If default_target_fps is not greater than 0.

GstFrameSamplingProcessConfig

Bases: ProcessorProcessConfig

Per-invocation configuration for [process][gllm_multimodal.media_toolkit.media_toolkit.MediaToolkit.process].

Attributes:

Name Type Description
target_fps int

Target frames-per-second for the output video.

validate_target_fps(target_fps) classmethod

Validate that target_fps is positive.

Parameters:

Name Type Description Default
target_fps int

The candidate target FPS to validate.

required

Returns:

Name Type Description
int int

target_fps unchanged when valid.

Raises:

Type Description
ValueError

If target_fps is not greater than 0.

GstFrameSamplingProcessor(config=None)

Bases: BaseGstreamerProcessor[GstFrameSamplingConfig], FrameSamplingProcessor

Resamples a video attachment to a target frame-rate using GStreamer.

The processor is format-agnostic: it handles MP4, MKV, MOV, and similar containers and preserves any audio track (passthrough by default).

Attributes:

Name Type Description
config GstFrameSamplingConfig

Runtime configuration.

Raises:

Type Description
ComponentConfigError

If the constructor configuration is invalid.

MediaOperationConfigError

If the per-call process_config is invalid.

MediaInputInvalidError

If an attachment has no MIME type or a malformed one.

MediaInputUnsupportedError

If an attachment is not a video.

MediaResultOperationError

If the pipeline completes without producing an output file.

MediaOperationTimeoutError

If the pipeline does not finish before config.timeout.

MediaOperationFailedError

If processing fails with an error outside the GLLM taxonomy, for example when the pipeline fails or a temporary file cannot be written. GLLM errors raised by the backend are re-raised unchanged.

MediaBackendUnavailableError

If GStreamer is unavailable.

RuntimeError

If no suitable encoder is found.

Initialise and select encoders.

Parameters:

Name Type Description Default
config dict[str, Any] | GstFrameSamplingConfig | None

Optional stable configuration.

None

Raises:

Type Description
ComponentConfigError

If config contains an unknown option or an invalid value, such as a non-positive default_target_fps, or is not a GstFrameSamplingConfig, dict, or None.

MediaBackendUnavailableError

If GStreamer is unavailable.

RuntimeError

If no suitable encoder is found.

config_model() classmethod

Return the stable configuration model for this processor.

Returns:

Type Description
type[GstFrameSamplingConfig]

type[GstFrameSamplingConfig]: The stable configuration model for this processor.

process(attachment, **kwargs) async

Resample one video attachment to the configured or requested FPS.

The method preserves the regular media-toolkit process contract while exposing this concrete processor's runtime semantics in API docs: constructor defaults establish baseline FPS behavior, and per-call overrides can be supplied through process_config.

Parameters:

Name Type Description Default
attachment Attachment

Input video attachment to resample.

required
**kwargs Any

Runtime options, typically process_config as GstFrameSamplingProcessConfig or dict.

{}
Notes
  1. Delegates shared mimetype validation and dispatch to MediaToolkit.process.

Returns:

Name Type Description
Attachment Attachment

Resampled video attachment, or the original attachment

Attachment

when the input is skipped by validation rules.

Raises:

Type Description
MediaInputUnsupportedError

If the attachment is not a video.

MediaInputInvalidError

If the attachment has no MIME type or a malformed one.

MediaOperationConfigError

If process_config is invalid.

MediaResultOperationError

If the pipeline completes without producing an output file.

MediaOperationTimeoutError

If the pipeline does not finish before config.timeout.

MediaOperationFailedError

If processing fails with an error outside the GLLM taxonomy.

Example
processor = GstFrameSamplingProcessor(
    config={"default_target_fps": 2},
)
sampled_attachment = await processor.process(
    attachment=video_attachment,
    process_config={"target_fps": 1},
)

process_config_model() classmethod

Return the per-invocation configuration model for this processor.

Returns:

Type Description
type[GstFrameSamplingProcessConfig]

type[GstFrameSamplingProcessConfig]: The configuration class

type[GstFrameSamplingProcessConfig]

accepted by

type[GstFrameSamplingProcessConfig]
type[GstFrameSamplingProcessConfig]

for per-call overrides.