Overview

Video-to-text conversion module providing video captioning and transcription.

Submodules

• video_to_text -- Abstract base class for video-to-text converters.
• video_to_caption -- Video captioning implementations.

HybridVideoToCaption(video_captioner=None, image_captioner=None, segmenter=None, transcriber=None, keyframe_extractor=None, pipeline_mode=PipelineMode.DIRECT_LM_CAPTION, audio_extractor_config=None, **kwargs)

Bases: BaseVideoToCaption

Hybrid video captioning component with a pluggable flow registry.

Wraps an inner captioner (a BaseVideoToCaption implementation) and orchestrates optional segmenter, transcriber, and keyframe_extractor components into one of several named pipeline flows.

The active flow is selected by pipeline_mode, which maps to a registered handler function. Built-in handlers are pre-registered at module load time. New flows can be added without modifying this class via register_flow.

Attributes:

Name	Type	Description
video_captioner	BaseVideoToCaption | None	Optional video captioner.
image_captioner	BaseImageToCaption | None	Optional image captioner.
segmenter	BaseSegmenter | None	Optional segmenter processor.
transcriber	BaseAudioToText | None	Optional audio-to-text transcriber.
keyframe_extractor	BaseKeyframeExtractor | None	Optional keyframe extractor processor.
pipeline_mode	str	Active pipeline mode key.

Example — built-in mode::

hybrid = HybridVideoToCaption(
    video_captioner=my_video_captioner,
    pipeline_mode=PipelineMode.DIRECT_LM_CAPTION,
)
result = await hybrid.convert("video.mp4", title="My Video")

Example — custom flow::

async def my_flow(ctx: FlowContext, attachment, caption_data, **kwargs):
    ...
    return await ctx.video_captioner.convert(attachment, **kwargs)

HybridVideoToCaption.register_flow(
    mode="my_flow",
    handler=my_flow,
    required_components=["segmenter"],
)

hybrid = HybridVideoToCaption(
    video_captioner=my_video_captioner,
    segmenter=my_segmenter,
    pipeline_mode="my_flow",
)
result = await hybrid.convert("video.mp4")

Initialise the hybrid captioner.

Parameters:

Name	Type	Description	Default
video_captioner	BaseVideoToCaption | None	The video captioner used to generate captions directly from video (or segment) attachments. Defaults to None.	None
image_captioner	BaseImageToCaption | None	The image captioner used when captioning individual frames or images. Defaults to None.	None
segmenter	BaseSegmenter | None	Segmenter processor. Defaults to None.	None
transcriber	BaseAudioToText | None	Audio-to-text transcriber. Defaults to None.	None
keyframe_extractor	BaseKeyframeExtractor | None	Frame extraction processor. Defaults to None.	None
pipeline_mode	PipelineMode | str	The pipeline mode to use. Defaults to PipelineMode.DIRECT_LM_CAPTION.	DIRECT_LM_CAPTION
audio_extractor_config	dict[str, Any] | None	Config forwarded to GstAudioExtractionProcessor (e.g. {"output_format": "mp3"} or {"audio_encoder": "lamemp3enc"}). Defaults to None (auto-select format, typically WAV).	None
**kwargs	Any	Forwarded to BaseVideoToCaption.	{}

Raises:

Type	Description
ValueError	If pipeline_mode is not registered in the flow registry.
ValueError	If a component required by pipeline_mode is None.

__init_subclass__(**kwargs)

Give every subclass its own independent registry copy.

from_preset(preset_name=HybridPreset.E2E_LM_ONLY, **kwargs) classmethod

Create a HybridVideoToCaption using a preset configuration.

Delegates preset_name to the preset registry in preset.py to retrieve the appropriate component composition and pipeline mode, then wraps it in a HybridVideoToCaption. Extra arguments can be injected via kwargs.

Parameters:

Name	Type	Description	Default
preset_name	HybridPreset | str | None	Preset name forwarded to the registry. Defaults to HybridPreset.E2E_LM_ONLY.	E2E_LM_ONLY
**kwargs	Any	Overrides forwarded to HybridVideoToCaption.__init__. Also supports lm_invoker_kwargs, prompt_builder_kwargs, and transcriber_kwargs which are passed down to the preset factory.	{}

Returns:

Name	Type	Description
HybridVideoToCaption	'HybridVideoToCaption'	A fully initialised hybrid captioner.

register_flow(mode, handler, required_components=None) classmethod

Register a new pipeline flow.

After registration the mode string can be passed as pipeline_mode to any new HybridVideoToCaption instance.

Parameters:

Name	Type	Description	Default
mode	str	Unique key for this flow (e.g. "my_custom_flow"). Can be a PipelineMode value or any arbitrary string.	required
handler	FlowHandler	An async callable with the signature:: async def handler( ctx: FlowContext, video_attachment: Attachment, caption_data: Caption, **kwargs: Any, ) -> VideoCaptionMetadata | None: ...	required
required_components	list[str | tuple[str, ...]] | None	Names of HybridVideoToCaption attributes (e.g. ["segmenter", "transcriber"]) that must not be None when this mode is selected. Validation happens at __init__ time. Use a tuple (e.g. [("captioner", "transcriber")]) to specify that at least one of the components in the tuple is required. Defaults to [].	None

Example

Register a custom flow

async def my_flow(ctx: FlowContext, attachment, caption_data, **kwargs):
    ...
    return await ctx.captioner.convert(attachment, **kwargs)

HybridVideoToCaption.register_flow(
    mode="my_flow",
    handler=my_flow,
    required_components=["segmenter"],
)

LMBasedVideoToCaption(lm_request_processor, transform=None, max_retries=2, **kwargs)

Bases: BaseVideoToCaption, UsesLM

Video captioning implementation using Language Models.

This class implements the VideoToCaption interface using LMs for generating natural language captions.

Initialize the LM based video captioning component.

Parameters:

Name	Type	Description	Default
lm_request_processor	LMRequestProcessor	Language model request processor instance that supports multimodal inputs.	required
transform	list[MediaToolkit] | None	Processor(s) to transform the video attachment.	None
max_retries	int	Maximum number of retries for LM invocations. Defaults to 2.	2
**kwargs	Any	Additional keyword arguments to pass to the parent constructor.	{}

from_preset(preset_name='default', lm_invoker_kwargs=None, prompt_builder_kwargs=None, **kwargs) classmethod

Initialize the LM based video captioning component using preset model configurations.

Parameters:

Name	Type	Description	Default
preset_name	str	Name of the preset to use.	'default'
lm_invoker_kwargs	dict | None	Keyword arguments to pass to the LM invoker.	None
prompt_builder_kwargs	dict | None	Keyword arguments to pass to the prompt builder.	None
**kwargs	Any	Additional keyword arguments to pass to from_lm_components().	{}

Returns:

Name	Type	Description
LMBasedVideoToCaption	LMBasedVideoToCaption	Initialized video captioning component using preset model.