Skip to content

Base

Frame decode processor family.

Decodes every video frame into PNG image attachments (dense decode), in contrast to FrameExtractionProcessor, which extracts sparse frames at caller-given timestamps. Dense decode feeds algorithm consumers such as shot segmenters.

Concrete backends (FFmpeg, GStreamer) register under this family so callers only change backend=. Each output attachment carries decode metadata:

  • frame_index (int): Zero-based position in decode order.
  • timestamp (float | None): Frame presentation time in seconds (frame_index / fps when the effective rate is known).
  • fps (float | None): Effective sampling rate (requested sample_fps or the stream's native framerate).
  • frame_decode_backend (str): Backend key that produced the frame.

Shared constructor / process knobs live on FrameDecodeConfig and FrameDecodeProcessConfig. Backend modules inherit those bases and add engine-specific fields.

FrameDecodeConfig

Bases: FrameDecodeFieldsMixin

Backend-agnostic stable configuration for dense frame decoding.

FrameDecodeFieldsMixin

Bases: BaseModel

Shared sample_fps / target_width fields for frame-decode configs.

Attributes:

Name Type Description
sample_fps int | None

Downsample decoded frames to this rate. None keeps the native framerate. Lower rates (2–5 fps) are the standard choice for shot detection over long videos. Defaults to None.
target_width int | None

Downscale decoded frames to this width (aspect preserved). None keeps native resolution. Defaults to None.

validate_sample_fps(value) classmethod

Reject non-positive sample rates.

Parameters:

Name Type Description Default
value int | None

Candidate rate.

 required

Returns:

Type Description
int | None

int | None: The rate unchanged when valid.

Raises:

Type Description
ValueError

If not positive.

validate_target_width(value) classmethod

Reject non-positive downscale widths.

Parameters:

Name Type Description Default
value int | None

Candidate width.

 required

Returns:

Type Description
int | None

int | None: The width unchanged when valid.

Raises:

Type Description
ValueError

If not positive.

FrameDecodeProcessConfig

Bases: ProcessorProcessConfig

Backend-agnostic per-invocation frame decode config.

Attributes:

Name Type Description
sample_fps int | None

Optional rate override.
target_width int | None

Optional width override.

FrameDecodeProcessor()

Bases: BackendSelectableProcessor[Attachment, list[Attachment]], ABC

Family base for dense video-frame decoding.

Why use this base class?

  • Portability: Swap FFmpeg vs GStreamer without changing call sites.
  • FIPS choice: backend="gstreamer" decodes with system plugins (no bundled-FFmpeg wheels); backend="ffmpeg" shells out to the system ffmpeg binary.
  • Separation: Frame decoding stays here; frame scoring (shot detection, keyframes) lives in segmenters/extractors.

Usage

from gllm_multimodal.media_toolkit.processor.frame_decode_processor import (
    FrameDecodeProcessor,
)

processor = FrameDecodeProcessor.build(backend="gstreamer")
frames = await processor.process(video_attachment)

frame_metadata(frame_index, fps)

Build the metadata dict attached to every decoded frame.

Parameters:

Name Type Description Default
frame_index int

Zero-based position in decode order.

 required
fps float | None

Effective sampling rate, if known.

 required

Returns:

Type Description
dict[str, Any]

dict[str, Any]: frame_index / timestamp / fps / frame_decode_backend mapping.

iter_rgb_frames_sync(attachment, *, sample_fps=None, target_width=None, as_rgb=True)

Yield decoded frames after the backend writes the full PNG sequence.

The decoder subprocess/pipeline completes first, so peak temp-disk usage is every sampled PNG at once. After that, this generator opens each file in order, yields the payload, and deletes the PNG so Python RAM stays O(1) in frames. Prefer this over process when callers only need a scored stream and can tolerate the peak-disk cost.

Parameters:

Name Type Description Default
attachment Attachment

Source video.

 required
sample_fps int | None

Override downsample rate. Defaults to the constructor config.

 None
target_width int | None

Override downscale width. Defaults to the constructor config.

 None
as_rgb bool

When True (default), yield RGB arrays. When False, yield raw PNG bytes (used by process).

 True

Yields:

Type Description
tuple[Any, dict[str, Any]]

tuple[Any, dict[str, Any]]: (rgb_or_png_bytes, frame_metadata).

Raises:

Type Description
RuntimeError

If decoding fails or yields no frames.
FileNotFoundError

If a required decoder binary is missing.

estimate_fps_from_container(data, filename, frame_count)

Estimate the native framerate from container duration metadata.

Parameters:

Name Type Description Default
data bytes

Raw video bytes.

 required
filename str | None

Filename hint for the container parser.

 required
frame_count int

Decoded frame count.

 required

Returns:

Type Description
float | None

float | None: frame_count / duration when hachoir reports a positive duration, else None.