Multimodal
Schemas for multimodal content contracts.
AudioTranscript
Bases: BaseModel
A timed transcript of spoken audio.
Attributes:
| Name | Type | Description |
|---|---|---|
text |
str
|
The transcribed text. |
start_time |
float
|
The transcript start time in seconds. |
end_time |
float
|
The transcript end time in seconds. |
lang_id |
str | None
|
The transcript language identifier. Defaults to None. |
Keyframe
Bases: BaseModel
A keyframe extracted from a video segment.
Attributes:
| Name | Type | Description |
|---|---|---|
time_offset |
float
|
Time within the segment where the keyframe occurs. |
caption |
str | None
|
Text description of the keyframe. |
Media
Bases: BaseModel
A media asset associated with multimodal content.
Attributes:
| Name | Type | Description |
|---|---|---|
media_id |
str
|
A stable identifier derived from the media type and content. |
media_type |
MediaType
|
The type of media, such as image, audio, etc. |
media_content |
str
|
Base64 encoded string, URL, or object storage key pointing to the media content. |
media_content_type |
MediaContentType
|
The type of media content, such as base64, url, or object storage key. |
media_id
property
Return a stable identifier derived from the media type and content.
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
A stable identifier derived from the media type and content. |
MediaContentType
Bases: StrEnum
Supported media content sources.
MediaType
Bases: StrEnum
Supported media types.
VideoCaptionMetadata
Bases: BaseModel
Caption metadata for an entire video.
Attributes:
| Name | Type | Description |
|---|---|---|
video_summary |
str
|
A high-level video summary. |
segments |
list[VideoSegment]
|
Captioned video segments. |
ensure_segment_end_time_greater_than_start_time()
Normalize segments whose end time is not after their start time.
Returns:
| Name | Type | Description |
|---|---|---|
VideoCaptionMetadata |
VideoCaptionMetadata
|
The video caption metadata with normalized segment end times. |
VideoSegment
Bases: BaseModel
A timed video segment with captions, transcripts, and keyframes.
Attributes:
| Name | Type | Description |
|---|---|---|
start_time |
float | None
|
The segment start time in seconds. Defaults to None. |
end_time |
float | None
|
The segment end time in seconds. Defaults to None. |
transcripts |
list[AudioTranscript]
|
The segment transcripts. |
segment_caption |
list[str]
|
The segment captions. |
keyframes |
list[Keyframe]
|
The segment keyframes. |
ensure_caption()
Populate an empty caption from valid keyframes or transcripts.
Returns:
| Name | Type | Description |
|---|---|---|
VideoSegment |
VideoSegment
|
The video segment with populated caption. |
ensure_keyframes()
Discard keyframes with negative time offsets.
Returns:
| Name | Type | Description |
|---|---|---|
VideoSegment |
VideoSegment
|
The video segment with keyframes whose time offsets are non-negative. |
ensure_transcripts()
Discard transcripts with invalid timestamps.
Returns:
| Name | Type | Description |
|---|---|---|
VideoSegment |
VideoSegment
|
The video segment with non-negative, non-reversed transcripts. |