Modality Converter Builder

Defines a convenience function to build a modality converter.

The build_modality_converter factory is the main entry point for constructing any BaseModalityConverter subclass. It resolves the appropriate converter class from the combination of source_modality, target_modality, task_type, and approach_type, then delegates to one of three build strategies.

Quick start

from gllm_multimodal.builder.modality_converter_builder import build_modality_converter
from gllm_multimodal.constants import Modality, ModalityConverterTask, ModalityConverterApproach

converter = build_modality_converter(
    source_modality=Modality.AUDIO,
    target_modality=Modality.TEXT,
    task_type=ModalityConverterTask.TRANSCRIPT,
    approach_type=ModalityConverterApproach.LM_BASED,
    lmrp_config={"model_id": "google/gemini-3.1-flash-lite"},
)
result = await converter.convert(audio_source="path/to/audio.mp3")
print(result)  # list[AudioTranscript]

Supported modality / task combinations

Source	Target	Task	Approaches
`IMAGE`	`TEXT`	`CAPTIONING`	`LM_BASED`
`IMAGE`	`TEXT`	`MERMAID`	`LM_BASED`
`IMAGE`	`TEXT`	`AUTO` (resolves to `CAPTIONING`)	(none required, resolves to `LM_BASED`)
`AUDIO`	`TEXT`	`TRANSCRIPT`	`LM_BASED`, `ASR`, `TRANSCRIPT_FETCH`
`AUDIO`	`TEXT`	`AUTO` (resolves to `TRANSCRIPT`)	(none required, resolves to `ASR` via Qwen)

Audio transcript provider selection

For AUDIO → TEXT with task_type=TRANSCRIPT, use one of the canonical approaches: LM_BASED, ASR, or TRANSCRIPT_FETCH. When model_id is provided, its provider selects the concrete audio converter and its model name or endpoint is forwarded to that converter. For TRANSCRIPT_FETCH, omit model_id to use the default YouTube transcript-fetch provider; YouTube does not require a model name.

provider is not a public keyword argument. Use the provider-qualified model_id to select a provider, together with the canonical approach:

Standard model IDs use the provider/model-name format. For a provider served through a custom endpoint, use the extended provider/custom_url:model-name format, for example prosa/https://api.prosa.ai/v2/speech/stt:v2.

Extended model_id format for custom endpoints

For models served at a custom endpoint, use the extended format provider/custom_url:model_name (e.g., openai/https://my-host.com:whisper-large-v3).

Approach	Model ID	Converter class
`LM_BASED`	`google/gemini-*`	`LMBasedAudioToTranscript`
`ASR`	`qwen/*`	`ASRBasedQwenAudioToText`
`ASR`	`openai/whisper-*`	`ASRBasedOpenAIAudioToText`
`ASR`	`prosa/*`	`ASRBasedProsaAudioToText`
`ASR`	`google_cloud/*`	`ASRBasedGoogleCloudAudioToText`
`ASR`	`snowflake/*`	`ASRBasedSnowflakeAudioToText`

Google Cloud ASR also accepts google-cloud/*. A google/* model ID is routed to Google Cloud when its model name does not contain gemini.

Examples:

# LM-based transcription. The model ID selects the Google/Gemini provider.
converter = build_modality_converter(
    source_modality=Modality.AUDIO,
    target_modality=Modality.TEXT,
    task_type=ModalityConverterTask.TRANSCRIPT,
    approach_type=ModalityConverterApproach.LM_BASED,
    model_id="google/gemini-3.1-flash-lite",
    api_key="...",
)

# ASR with a provider-specific model ID.
converter = build_modality_converter(
    source_modality=Modality.AUDIO,
    target_modality=Modality.TEXT,
    task_type=ModalityConverterTask.TRANSCRIPT,
    approach_type=ModalityConverterApproach.ASR,
    model_id="openai/whisper-1",
    api_key="...",
)

Legacy provider-specific approaches (WHISPER, GEMINI, PROSA, QWEN, GOOGLE_CLOUD, YOUTUBE, SNOWFLAKE) are automatically remapped to their canonical form with the appropriate implicit provider.

For YouTube transcript fetching, do not pass model_id:

converter = build_modality_converter(
    source_modality=Modality.AUDIO,
    target_modality=Modality.TEXT,
    task_type=ModalityConverterTask.TRANSCRIPT,
    approach_type=ModalityConverterApproach.TRANSCRIPT_FETCH,
)

Build strategies

The factory selects a build strategy based on which kwargs you provide. Use the decision guide below to pick the right one quickly.

Quick decision guide:

I want to …	Use strategy
Connect to a language model (Gemini, GPT, etc.)	1 — LMRP (pass `lmrp_config`)
Use a library preset (with optional overrides)	2 — PRESET (pass `preset`)
Inject a fully-constructed processor object	3 — KWARGS (pass the object directly)

1. LMRP strategy

When to use: You want to build a converter backed by a language model (e.g., Gemini, GPT-4o). This strategy only works with LM_BASED approach converters such as LMBasedImageToCaption or LMBasedAudioToTranscript (and its provider subclasses like LMBasedGeminiAudioToText). Pass an lmrp_config dict and the factory will build an LMRequestProcessor and wire it into the converter automatically.

Key kwargs:

kwarg	Type	Description
`lmrp_config`	`dict`	Configuration dictionary passed directly to [`build_lm_request_processor`][gllm_inference.request_processor.build_lm_request_processor]. See that function for all available parameters (e.g., `model_id`, `config`, etc.).
`api_key`	`str`	Convenience option — forwarded to `lmrp_config["api_key"]` if not already set.

build_modality_converter(
    source_modality=Modality.AUDIO,
    target_modality=Modality.TEXT,
    task_type=ModalityConverterTask.TRANSCRIPT,
    approach_type=ModalityConverterApproach.LM_BASED,
    lmrp_config={
        "model_id": "google/gemini-3.1-flash-lite",
        "config": {"temperature": 0.3},
    },
    api_key="...",
)

2. PRESET strategy

When to use: The converter ships named configurations (presets) and you want to use one without providing every constructor argument yourself. Presets encapsulate sensible defaults (model, prompt template, formatter) so you only need to name them.

Note

This strategy only works for converters that implement a from_preset() class method. If the converter does not implement from_preset, the factory falls back to KWARGS with a warning.

Additional kwargs are forwarded to from_preset so you can still override specific settings (e.g., lm_invoker_kwargs, prompt_builder_kwargs, formatter).

Key kwargs:

kwarg	Type	Description
`preset`	`str`	Preset name to pass to `from_preset()`, e.g. `"default"`. Available presets are converter-specific.
(any)	varies	All additional kwargs are forwarded directly to the converter's `from_preset()` method.

Check the documentation of the specific converter class you are building to see which keyword arguments are accepted by its from_preset() method:

build_modality_converter(
    source_modality=Modality.IMAGE,
    target_modality=Modality.TEXT,
    task_type=ModalityConverterTask.CAPTIONING,
    approach_type=ModalityConverterApproach.LM_BASED,
    preset="default",
    lm_invoker_kwargs={"credentials": "..."},
)

3. KWARGS strategy

When to use: You need fine-grained control over the converter's construction. This is the fallback when neither lmrp_config nor preset is provided. The factory forwards all provided kwargs directly to the resolved converter's __init__ method (and validates that they are accepted, unless the constructor takes **kwargs).

Key kwargs:

kwarg	Type	Description
(any)	varies	Passed directly to the converter's `__init__`. Please refer to the specific converter class documentation for its available parameters.

build_modality_converter(
    source_modality=Modality.IMAGE,
    target_modality=Modality.TEXT,
    task_type=ModalityConverterTask.CAPTIONING,
    approach_type=ModalityConverterApproach.LM_BASED,
    lm_request_processor=my_lmrp,
    formatter=my_formatter,
)

`build_modality_converter(source_modality, target_modality, task_type=ModalityConverterTask.AUTO, approach_type=None, preset=None, model_id=None, strategy=None, **kwargs)`

Build and initialize a modality converter instance for a given configuration.

The factory looks up the converter class based on the combination of

source_modality: input modality (e.g., Modality.IMAGE, Modality.AUDIO)
target_modality: output modality (e.g., Modality.TEXT)
task_type: conversion task (e.g., CAPTIONING, TRANSCRIPT, MERMAID, or AUTO)
approach_type: the converter's algorithmic approach; required for non-AUTO tasks, must be None for AUTO

Audio transcript approaches are separated into canonical LM_BASED, ASR, and TRANSCRIPT_FETCH values. For audio transcription, model_id selects the provider implementation and is forwarded to the LM invoker when the resolved approach is LM_BASED. Use this factory directly for audio transcription.

preset has the same meaning as for image converters: a named from_preset config.

Parameters:

Name	Type	Description	Default
`source_modality`	`Modality`	The source modality.	required
`target_modality`	`Modality`	The output modality.	required
`task_type`	`ModalityConverterTask`	The conversion task. Defaults to ModalityConverterTask.AUTO.	`AUTO`
`approach_type`	`ModalityConverterApproach \| None`	The approach for the conversion. Required for non-AUTO tasks; must be None for task_type=AUTO.	`None`
`preset`	`str \| None`	Named `from_preset` config (same semantics as image-to-text). Defaults to None.	`None`
`model_id`	`str \| ModelId \| None`	Provider-qualified audio model identifier used to infer the provider implementation when applicable. Defaults to None, which selects the default provider for the requested approach. For transcript fetching, omit this argument to select YouTube, which does not require a model name.	`None`
`strategy`	`ModalityConverterBuildStrategy \| None`	The build strategy to use. If None, the strategy is determined automatically based on the provided parameters.	`None`
`**kwargs`	`Any`	Additional keyword arguments passed to the converter, including: 1. lmrp_config (dict[str, Any]): Configuration to build an LMRP instance. Should follow the same structure as `build_lm_request_processor`. 2. Any other parameters supported by the converter's initialization method.	`{}`

Returns:

Name	Type	Description
`BaseModalityConverter`	`BaseModalityConverter`	An instance of the matching converter class.

Raises:

Type	Description
`ValueError`	If the configuration is invalid or not registered, including: - (source_modality, target_modality, task_type, approach) not registered - approach_type missing for non-AUTO task_type - approach_type provided when task_type is AUTO - Any dimension unsupported for the given combination

Modality Converter Builder

Quick start

See also

Supported modality / task combinations

Audio transcript provider selection

Build strategies

1. LMRP strategy

2. PRESET strategy

3. KWARGS strategy

build_modality_converter(source_modality, target_modality, task_type=ModalityConverterTask.AUTO, approach_type=None, preset=None, model_id=None, strategy=None, **kwargs)

`build_modality_converter(source_modality, target_modality, task_type=ModalityConverterTask.AUTO, approach_type=None, preset=None, model_id=None, strategy=None, **kwargs)`