Overview

Frame decode processor family.

This package provides dense video-frame decoding into PNG image attachments, with ffmpeg and gstreamer backend implementations.

Exported Classes

• FrameDecodeProcessor -- Abstract base for dense frame decoding.
• FrameDecodeConfig -- Shared stable configuration.
• FrameDecodeProcessConfig -- Shared per-invocation configuration.
• FFmpegFrameDecodeProcessor -- FFmpeg CLI-based frame decoding.
• GstFrameDecodeProcessor -- GStreamer-based frame decoding (default, FIPS-friendly).

FFmpegFrameDecodeConfig

Bases: FrameDecodeFieldsMixin, FFmpegBaseConfig

FFmpeg-specific stable configuration for dense frame decoding.

FFmpegFrameDecodeProcessConfig

Bases: FrameDecodeProcessConfig

Per-invocation FFmpeg overrides (inherits shared process config).

FFmpegFrameDecodeProcessor(config=None)

Bases: BaseFFmpegProcessor, FrameDecodeProcessor

Decode every video frame to PNG attachments with the FFmpeg CLI.

Attributes:

Name	Type	Description
config	FFmpegFrameDecodeConfig	Stable constructor configuration.

Initialize the FFmpeg frame decode processor.

Parameters:

Name	Type	Description	Default
config	dict[str, Any] | FrameDecodeConfig | FFmpegFrameDecodeConfig | None	Shared or FFmpeg-specific config. Defaults to None.	None

build_filter(*, sample_fps, target_width)

Build the -vf filter chain for the requested knobs.

Parameters:

Name	Type	Description	Default
sample_fps	int | None	Downsample rate, if any.	required
target_width	int | None	Downscale width, if any.	required

Returns:

Type	Description
str | None	str | None: Comma-joined filter chain, or None for native decode.

config_model() classmethod

Return the stable configuration model.

process(attachment, **kwargs) async

Decode every frame of one video attachment.

Parameters:

Name	Type	Description	Default
attachment	Attachment	Source video.	required
**kwargs	Any	May include process_config.	{}

Returns:

Type	Description
list[Attachment]	list[Attachment]: PNG image attachments in decode order.

process_config_model() classmethod

Return the per-invocation configuration model.

FrameDecodeConfig

Bases: FrameDecodeFieldsMixin

Backend-agnostic stable configuration for dense frame decoding.

FrameDecodeProcessConfig

Bases: ProcessorProcessConfig

Backend-agnostic per-invocation frame decode config.

Attributes:

Name	Type	Description
sample_fps	int | None	Optional rate override.
target_width	int | None	Optional width override.

FrameDecodeProcessor()

Bases: BackendSelectableProcessor[Attachment, list[Attachment]], ABC

Family base for dense video-frame decoding.

Why use this base class?

• Portability: Swap FFmpeg vs GStreamer without changing call sites.
• FIPS choice: backend="gstreamer" decodes with system plugins (no bundled-FFmpeg wheels); backend="ffmpeg" shells out to the system ffmpeg binary.
• Separation: Frame decoding stays here; frame scoring (shot detection, keyframes) lives in segmenters/extractors.

Usage

from gllm_multimodal.media_toolkit.processor.frame_decode_processor import (
    FrameDecodeProcessor,
)

processor = FrameDecodeProcessor.build(backend="gstreamer")
frames = await processor.process(video_attachment)

frame_metadata(frame_index, fps)

Build the metadata dict attached to every decoded frame.

Parameters:

Name	Type	Description	Default
frame_index	int	Zero-based position in decode order.	required
fps	float | None	Effective sampling rate, if known.	required

Returns:

Type	Description
dict[str, Any]	dict[str, Any]: frame_index / timestamp / fps / frame_decode_backend mapping.

iter_rgb_frames_sync(attachment, *, sample_fps=None, target_width=None, as_rgb=True)

Yield decoded frames after the backend writes the full PNG sequence.

The decoder subprocess/pipeline completes first, so peak temp-disk usage is every sampled PNG at once. After that, this generator opens each file in order, yields the payload, and deletes the PNG so Python RAM stays O(1) in frames. Prefer this over process when callers only need a scored stream and can tolerate the peak-disk cost.

Parameters:

Name	Type	Description	Default
attachment	Attachment	Source video.	required
sample_fps	int | None	Override downsample rate. Defaults to the constructor config.	None
target_width	int | None	Override downscale width. Defaults to the constructor config.	None
as_rgb	bool	When True (default), yield RGB arrays. When False, yield raw PNG bytes (used by process).	True

Yields:

Type	Description
tuple[Any, dict[str, Any]]	tuple[Any, dict[str, Any]]: (rgb_or_png_bytes, frame_metadata).

Raises:

Type	Description
RuntimeError	If decoding fails or yields no frames.
FileNotFoundError	If a required decoder binary is missing.

GstFrameDecodeConfig

Bases: FrameDecodeFieldsMixin, GstBaseConfig

Stable configuration for GstFrameDecodeProcessor.

GstFrameDecodeProcessConfig

Bases: FrameDecodeProcessConfig

Per-invocation GStreamer decode overrides (inherits shared process config).

GstFrameDecodeProcessor(config=None)

Bases: BaseGstreamerProcessor[GstFrameDecodeConfig], FrameDecodeProcessor

Decode every video frame to PNG attachments with system GStreamer.

Attributes:

Name	Type	Description
config	GstFrameDecodeConfig	Runtime configuration.

Raises:

Type	Description
RuntimeError	If GStreamer is not available.

Initialise without encoder selection (decode needs no encoder).

Parameters:

Name	Type	Description	Default
config	dict[str, Any] | FrameDecodeConfig | GstFrameDecodeConfig | None	Shared or GStreamer-specific config. Defaults to None.	None

config_model() classmethod

Return the stable configuration model for this processor.

Returns:

Type	Description
type[GstFrameDecodeConfig]	type[GstFrameDecodeConfig]: The stable configuration model.

process(attachment, **kwargs) async

Decode every frame of one video attachment.

Parameters:

Name	Type	Description	Default
attachment	Attachment	Input video attachment to decode.	required
**kwargs	Any	Runtime options, typically process_config as GstFrameDecodeProcessConfig or dict.	{}

Returns:

Type	Description
list[Attachment]	list[Attachment]: PNG image attachments in decode order.

process_config_model() classmethod

Return the per-invocation configuration model for this processor.

Returns:

Type	Description
type[GstFrameDecodeProcessConfig]	type[GstFrameDecodeProcessConfig]: The configuration class accepted by process for per-call overrides.

estimate_fps_from_container(data, filename, frame_count)

Estimate the native framerate from container duration metadata.

Parameters:

Name	Type	Description	Default
data	bytes	Raw video bytes.	required
filename	str | None	Filename hint for the container parser.	required
frame_count	int	Decoded frame count.	required

Returns:

Type	Description
float | None	float | None: frame_count / duration when hachoir reports a positive duration, else None.