Skip to content

Overview

Audio extraction processor family.

This package provides processors for extracting audio tracks from video files for audio-to-text conversion.

Exported Classes

AudioExtractionProcessor()

Bases: BackendSelectableProcessor[Attachment, Attachment], ABC

Family base for extracting audio tracks from video attachments.

This class serves as a unified entry point for audio extraction operations. It automatically routes requests to the most appropriate, available backend implementation based on your system environment.

Why use this base class?

  • Portability: Your code will run regardless of which underlying libraries are installed on the host machine.
  • Simplicity: No need to handle fallback logic or conditional imports yourself.
  • Future-proofing: New backends can be added to the library without requiring changes to your application code.

Usage Example

from gllm_multimodal.media_toolkit.processor.audio_extraction_processor import AudioExtractionProcessor
from gllm_inference.schema import Attachment

# Instantiates the best available backend automatically
processor = AudioExtractionProcessor.build()

attachment = Attachment(url="file:///path/to/video.mp4")
audio_attachment = await processor.process(attachment)

from gllm_multimodal.media_toolkit.processor.audio_extraction_processor import AudioExtractionProcessor
from gllm_inference.schema import Attachment

# Explicitly force the ffmpeg backend
processor = AudioExtractionProcessor.build(backend="ffmpeg")

attachment = Attachment(url="file:///path/to/video.mp4")
audio_attachment = await processor.process(attachment)

from gllm_multimodal.media_toolkit.processor.audio_extraction_processor import AudioExtractionProcessor
from gllm_inference.schema import Attachment

# Explicitly force the moviepy backend
processor = AudioExtractionProcessor.build(backend="moviepy")

attachment = Attachment(url="file:///path/to/video.mp4")
audio_attachment = await processor.process(attachment)

GstAudioExtractionConfig

Bases: GstBaseConfig

Configuration for GstAudioExtractionProcessor.

Attributes:

Name Type Description
sample_rate int | None

Force output sample rate in Hz (e.g. 16000). None preserves the source rate. Defaults to None.
channels int | None

Force output channel count (e.g. 1 for mono). None preserves the source count. Defaults to None.
output_format str | None

Preferred container/extension to extract to ("wav", "mp3", "m4a", or "ogg"). When set, only encoders for that format are considered. Defaults to None (auto-select from all candidates, preferring WAV when available).
audio_encoder str | None

Pin a specific GStreamer audio encoder element (e.g. "lamemp3enc"). When both audio_encoder and output_format are set, the candidate list is first narrowed to the pinned encoder, then further filtered to the requested format; an incompatible combination raises ValueError. Defaults to None.

GstAudioExtractionProcessConfig

Bases: ProcessorProcessConfig

Per-invocation configuration for [process][gllm_multimodal.media_toolkit.media_toolkit.MediaToolkit.process].

Attributes:

Name Type Description
sample_rate int | None

Force output sample rate in Hz. None is a legitimate per-call override meaning "do not force a sample rate" — it suppresses the constructor self.config.sample_rate so the source rate is preserved.
channels int | None

Force output channel count. None is a legitimate per-call override meaning "do not force a channel count" — it suppresses the constructor self.config.channels.

GstAudioExtractionProcessor(config=None)

Bases: BaseGstreamerProcessor[GstAudioExtractionConfig], AudioExtractionProcessor

Extracts the audio track from a video Attachment using GStreamer.

The processor demuxes the input video, re-encodes (or passes through) the audio into the best available format, and returns the result as an Attachment whose mime_type reflects the audio container.

If the video has no audio track the original Attachment is returned unchanged, so callers do not need to handle None.

Attributes:

Name Type Description
audio_encoder_info EncoderFormatInfo | None

Selected audio encoder format info.
config GstAudioExtractionConfig

Runtime configuration.

Raises:

Type Description
RuntimeError

If GStreamer is unavailable or no suitable audio encoder is found.
ValueError

If output_format or audio_encoder is unsupported.
Example 
processor = GstAudioExtractionProcessor(config={"output_format": "mp3"})
audio_attachment = await processor.process(video_attachment)
# audio_attachment.mime_type == "audio/mpeg"
# audio_attachment.filename  == "audio_my_video.mp3"

Initialise GStreamer and select the audio encoder / output format.

Parameters:

Name Type Description Default
config dict[str, Any] | GstAudioExtractionConfig | None

Optional configuration. A plain dict is coerced into GstAudioExtractionConfig. None uses defaults. Use output_format (e.g. "mp3") or audio_encoder (e.g. "lamemp3enc") to override auto-selection.

 None

Raises:

Type Description
RuntimeError

If GStreamer is unavailable or no audio encoder is found.
ValueError

If output_format or audio_encoder is unsupported.

config_model() classmethod

Return the stable configuration model for this processor.

Returns:

Type Description
type[GstAudioExtractionConfig]

type[GstAudioExtractionConfig]: The configuration class used at
type[GstAudioExtractionConfig]

construction time for stable (per-instance) settings.

process(attachment, **kwargs) async

Extract the audio track of one video attachment.

This public entrypoint keeps configuration ergonomics simple for callers: stable defaults can be supplied in the constructor, while per-call overrides (for sample rate/channels) can be passed through process_config in kwargs.

Parameters:

Name Type Description Default
attachment Attachment

Input video attachment whose audio should be extracted.

 required
**kwargs Any

Additional runtime options, typically process_config as GstAudioExtractionProcessConfig or dict.

 {}
Notes
  1. Delegates shared mimetype validation and dispatch to MediaToolkit.process.

Returns:

Name Type Description
Attachment Attachment

Extracted audio attachment, or the original attachment when
Attachment

no audio stream is detected.
Example 
processor = GstAudioExtractionProcessor(
    config={"output_format": "mp3"},
)
audio_attachment = await processor.process(
    attachment=video_attachment,
    process_config={"sample_rate": 16000, "channels": 1},
)

process_config_model() classmethod

Return the per-invocation configuration model for this processor.

Returns:

Type Description
type[GstAudioExtractionProcessConfig]

type[GstAudioExtractionProcessConfig]: The configuration class
type[GstAudioExtractionProcessConfig]

accepted by
type[GstAudioExtractionProcessConfig]

process
type[GstAudioExtractionProcessConfig]

for per-call overrides.