Skip to content

Overview

Frame extraction processor family.

Decode one or more still frames from a video at given timestamps.

Exported Classes

DeinterlaceMode

Bases: StrEnum

How frame extraction applies deinterlace (e.g. FFmpeg yadif).

Attributes:

Name Type Description
OFF

Never deinterlace.
FORCE

Always deinterlace (ignore field-order metadata).
AUTO

Deinterlace only when ffprobe reports an interlaced field order.

FFmpegFrameExtractionConfig

Bases: FrameExtractionConfig

FFmpeg-specific stable config (inherits shared FrameExtractionConfig).

Yadif fields apply when deinterlace is force or auto (and the probe selects yadif for auto).

Attributes:

Name Type Description
yadif_mode int

Yadif mode. Defaults to 0.
yadif_parity int

Field parity (-1=auto). Defaults to -1.
yadif_deint int

Deinterlace all (0) or flagged-only (1). Defaults to 0.
filter_override str | None

Full -vf string when set. Defaults to None.

yadif_filter()

Build the -vf filter string for deinterlaced extract.

Returns:

Name Type Description
str str

Filter graph for FFmpeg -vf.

FFmpegFrameExtractionProcessConfig

Bases: FrameExtractionProcessConfig

Per-invocation FFmpeg overrides (inherits shared process config).

FFmpegFrameExtractionProcessor(config=None)

Bases: BaseFFmpegProcessor[FFmpegFrameExtractionConfig], FrameExtractionProcessor

Extract image frames with FFmpeg (optional yadif).

Attributes:

Name Type Description
config FFmpegFrameExtractionConfig

Stable constructor configuration.

Initialize the FFmpeg frame extraction processor.

Parameters:

Name Type Description Default
config dict[str, Any] | FrameExtractionConfig | FFmpegFrameExtractionConfig | None

Shared or FFmpeg-specific config. Defaults to None.

 None

config_model() classmethod

Return the stable configuration model.

extract_frames(video_path, timestamps, *, params=None)

Extract encoded image bytes at each timestamp from a video path.

Parameters:

Name Type Description Default
video_path str

Source video path.

 required
timestamps list[float]

Non-empty non-negative timestamps.

 required
params FFmpegFrameExtractionConfig | None

Effective options. Defaults to self.config.

 None

Returns:

Type Description
list[bytes]

list[bytes]: Encoded frames in the same order as timestamps.

Raises:

Type Description
FileNotFoundError

If ffmpeg is unavailable.
RuntimeError

If any extraction fails.
ValueError

If timestamps are invalid.

process(attachment, **kwargs) async

Extract frames from one video attachment.

Parameters:

Name Type Description Default
attachment Attachment

Source video.

 required
**kwargs Any

May include process_config.

 {}

Returns:

Type Description
list[Attachment]

list[Attachment]: One image attachment per timestamp.

process_config_model() classmethod

Return the per-invocation configuration model.

set_timestamps(timestamps)

Update constructor-level default timestamps.

Parameters:

Name Type Description Default
timestamps list[float]

Non-empty non-negative timestamps.

 required

stream_frames(attachment, sample_fps, deinterlace=None) async

Stream sampled BGR frames from a single FFmpeg process.

Parameters:

Name Type Description Default
attachment Attachment

Source video.

 required
sample_fps float

Positive sampling rate in frames per second.

 required
deinterlace DeinterlaceMode | None

Deinterlace override. Defaults to None.

 None

Yields:

Type Description
AsyncIterator[tuple[float, ndarray]]

tuple[float, np.ndarray]: Timestamp and BGR uint8 frame at source size.

Raises:

Type Description
ValueError

If sample_fps is not finite and positive.
FileNotFoundError

If ffmpeg or ffprobe is unavailable.
RuntimeError

If probing or decoding fails.

FrameExtractionConfig

Bases: BaseModel

Backend-agnostic stable configuration for frame extraction.

Yadif knobs are shared so FFmpeg can inline -vf yadif and GStreamer can forward the same values to DeinterlaceProcessor.

Attributes:

Name Type Description
output_format str

Image encode format (JPEG/PNG). Defaults to JPEG.
deinterlace DeinterlaceMode

Deinterlace policy (off / force / auto). Bool aliases True→force, False→off are accepted. Defaults to off.
yadif_mode int

Yadif mode (0=send_frame, 1=send_field, 2=send_frame_nospatial, 3=send_field_nospatial). Defaults to 0.
yadif_parity int

Field parity (-1=auto, 0=tff, 1=bff). Defaults to -1.
yadif_deint int

Deinterlace all frames (0) or flagged-only (1). Defaults to 0.
default_timestamps list[float] | None

Used when process is called without process_config. Defaults to None (must pass per-call).

validate_default_timestamps(value) classmethod

Validate optional constructor-level timestamps.

Parameters:

Name Type Description Default
value list[float] | None

Candidate timestamps.

 required

Returns:

Type Description
list[float] | None

list[float] | None: Validated timestamps or None.

validate_deinterlace_mode(value) classmethod

Coerce bool / string deinterlace values.

Parameters:

Name Type Description Default
value Any

Candidate mode.

 required

Returns:

Name Type Description
DeinterlaceMode DeinterlaceMode

Normalized mode.

validate_output_format(value) classmethod

Reject blank output format strings.

Parameters:

Name Type Description Default
value str

Candidate format.

 required

Returns:

Name Type Description
str str

Normalized upper-case format.

Raises:

Type Description
ValueError

If blank.

yadif_filter()

Build the shared yadif=mode:parity:deint filter description.

Returns:

Name Type Description
str str

Yadif filter string consumed by FFmpeg -vf and nested GStreamer deinterlace metadata.

FrameExtractionProcessConfig

Bases: ProcessorProcessConfig

Backend-agnostic per-invocation frame extraction config.

Attributes:

Name Type Description
timestamps list[float]

Required non-empty timestamps in seconds.
output_format str | None

Optional format override.
deinterlace DeinterlaceMode | None

Optional deinterlace override.
yadif_mode int | None

Optional yadif mode override.
yadif_parity int | None

Optional field-parity override.
yadif_deint int | None

Optional flagged-only override.

validate_optional_deinterlace_mode(value) classmethod

Coerce optional bool / string deinterlace overrides.

Parameters:

Name Type Description Default
value Any

Candidate mode or None.

 required

Returns:

Type Description
DeinterlaceMode | None

DeinterlaceMode | None: Normalized mode or None.

validate_optional_output_format(value) classmethod

Normalize optional format override.

Parameters:

Name Type Description Default
value str | None

Candidate format.

 required

Returns:

Type Description
str | None

str | None: Upper-case format or None.

validate_process_timestamps(value) classmethod

Validate per-call timestamps.

Parameters:

Name Type Description Default
value list[float]

Candidate timestamps.

 required

Returns:

Type Description
list[float]

list[float]: Validated timestamps.

FrameExtractionProcessor()

Bases: BackendSelectableProcessor[Attachment, list[Attachment]], ABC

Family base for extracting image frames at timestamps.

Why use this base class?

  • Portability: Swap FFmpeg vs GStreamer without changing call sites.
  • Batch I/O: Extract many keyframes in one process call.
  • Separation: Keyframe planning stays in extractors; decode is here.

Usage

from gllm_multimodal.media_toolkit.processor.frame_extraction_processor import (
    FrameExtractionProcessConfig,
    FrameExtractionProcessor,
)

processor = FrameExtractionProcessor.build()
frames = await processor.process(
    video_attachment,
    process_config=FrameExtractionProcessConfig(timestamps=[1.5, 4.0]),
)

set_timestamps(timestamps) abstractmethod

Configure default timestamps used when process_config is omitted.

Parameters:

Name Type Description Default
timestamps list[float]

Non-empty list of non-negative seconds.

 required

stream_frames(attachment, sample_fps, deinterlace=None) abstractmethod

Stream frames sampled at a fixed rate from a single decode pass.

Parameters:

Name Type Description Default
attachment Attachment

Source video.

 required
sample_fps float

Positive sampling rate in frames per second.

 required
deinterlace DeinterlaceMode | None

Deinterlace override. Defaults to None.

 None

Returns:

Type Description
AsyncIterator[tuple[float, ndarray]]

AsyncIterator[tuple[float, np.ndarray]]: Timestamps and BGR uint8 frames at source size.

GstFrameExtractionConfig

Bases: FrameExtractionConfig, GstBaseConfig

GStreamer-specific stable config (inherits shared FrameExtractionConfig).

Shared yadif knobs are forwarded to DeinterlaceProcessor when deinterlace runs. Encode settings remain on GstDeinterlaceConfig.

Attributes:

Name Type Description
output_format str

Image encode format (JPEG/PNG). Defaults to JPEG.
deinterlace DeinterlaceMode

Whether to run the nested deinterlace family before extracting frames. Defaults to off.
yadif_mode int

Shared frame setting used by the FFmpeg backend; GStreamer uses its native deinterlace mode. Defaults to 0.
yadif_parity int

Shared field parity used by the FFmpeg backend. GStreamer reads field order from caps. Defaults to -1.
yadif_deint int

Shared flagged-only setting used by the FFmpeg backend. GStreamer reads interlace flags from caps. Defaults to 0.
default_timestamps list[float] | None

Used when process is called without process_config. Defaults to None.

validate_supported_image_format(value) classmethod

Reject still-image formats this backend cannot encode.

Parameters:

Name Type Description Default
value str

Candidate format (already upper-cased by the parent).

 required

Returns:

Name Type Description
str str

Validated format.

Raises:

Type Description
ValueError

If the format is not JPEG/JPG/PNG.

GstFrameExtractionProcessConfig

Bases: FrameExtractionProcessConfig

Per-invocation GStreamer overrides (inherits shared process config).

validate_optional_supported_image_format(value) classmethod

Reject optional still-image format overrides this backend cannot encode.

Parameters:

Name Type Description Default
value str | None

Candidate format or None.

 required

Returns:

Type Description
str | None

str | None: Validated format or None.

Raises:

Type Description
ValueError

If the format is not JPEG/JPG/PNG.

GstFrameExtractionProcessor(config=None)

Bases: BaseGstreamerProcessor[GstFrameExtractionConfig], FrameExtractionProcessor

Extract image frames with GStreamer.

When deinterlace is force or auto (and the source is interlaced), frames are taken from the progressive output of DeinterlaceProcessor rather than inlining yadif in the still-frame pipeline.

Attributes:

Name Type Description
config GstFrameExtractionConfig

Stable constructor configuration.

Initialize the GStreamer frame extraction processor.

Parameters:

Name Type Description Default
config dict[str, Any] | FrameExtractionConfig | GstFrameExtractionConfig | None

Shared or GStreamer-specific config. Defaults to None.

 None

config_model() classmethod

Return the stable configuration model.

extract_frames(video_path, timestamps, *, params=None)

Extract encoded image bytes at each timestamp from a video path.

Parameters:

Name Type Description Default
video_path str

Source video path.

 required
timestamps list[float]

Non-empty non-negative timestamps.

 required
params GstFrameExtractionConfig | None

Effective options. Defaults to self.config.

 None

Returns:

Type Description
list[bytes]

list[bytes]: Encoded frames in the same order as timestamps.

Raises:

Type Description
RuntimeError

If any extraction fails.
ValueError

If timestamps are invalid.

process(attachment, **kwargs) async

Extract frames from one video attachment.

Parameters:

Name Type Description Default
attachment Attachment

Source video.

 required
**kwargs Any

May include process_config.

 {}

Returns:

Type Description
list[Attachment]

list[Attachment]: One image attachment per timestamp.

process_config_model() classmethod

Return the per-invocation configuration model.

set_timestamps(timestamps)

Update constructor-level default timestamps.

Parameters:

Name Type Description Default
timestamps list[float]

Non-empty non-negative timestamps.

 required

stream_frames(attachment, sample_fps, deinterlace=None) async

Stream sampled BGR frames from a single GStreamer decode pass.

Frames are resampled by videorate, auto-rotated by videoflip and pulled from a bounded appsink, so decoding pauses while the consumer is busy. Closing the generator early stops the pipeline.

Parameters:

Name Type Description Default
attachment Attachment

Source video.

 required
sample_fps float

Positive sampling rate in frames per second.

 required
deinterlace DeinterlaceMode | None

Deinterlace override. Defaults to None.

 None

Yields:

Type Description
AsyncIterator[tuple[float, ndarray]]

tuple[float, np.ndarray]: Timestamp and BGR uint8 frame at source size.

Raises:

Type Description
ValueError

If sample_fps is not finite and positive.
RuntimeError

If decoding fails, the source has no video stream, or the pipeline stalls.

coerce_frame_extraction_config(config)

Normalize optional shared frame-extraction config.

Parameters:

Name Type Description Default
config FrameExtractionConfig | dict[str, object] | None

Raw config.

 required

Returns:

Name Type Description
FrameExtractionConfig FrameExtractionConfig

Shared configuration shape.