Lm invoker
Defines a base class for language model invokers used in Gen AI applications.
BaseLMInvoker(model_id, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, input_transformers=None, output_transformers=None, input_transformer=None, output_transformer=None, thinking=False)
Bases: ABC
A base class for language model invokers used in Gen AI applications.
The BaseLMInvoker class provides a framework for invoking language models.
It handles both standard and streaming invocation.
Attributes:
| Name | Type | Description |
|---|---|---|
model_id |
str
|
The model ID of the language model. |
model_provider |
str
|
The provider of the language model. |
model_name |
str
|
The name of the language model. |
default_config |
dict[str, Any]
|
Default config for invoking the language model. |
tools |
list[Tool]
|
Tools provided to the language model to enable tool calling. |
response_schema |
ResponseSchema | None
|
The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. |
output_analytics |
bool
|
Whether to output the invocation analytics. |
retry_config |
RetryConfig
|
The retry configuration for the language model. |
thinking |
ThinkingConfig
|
The thinking configuration for the language model. |
input_transformers |
list[InputTransformerConfig]
|
The list of input transformer configs to use. |
output_transformers |
list[OutputTransformerConfig]
|
The list of output transformer configs to use. |
Initializes a new instance of the BaseLMInvoker class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_id
|
ModelId
|
The model ID of the language model. |
required |
default_hyperparameters
|
dict[str, Any] | None
|
Default hyperparameters for invoking the language model. Defaults to None, in which case an empty dictionary is used. |
None
|
tools
|
list[LMTool] | None
|
Tools provided to the model to enable tool calling. Defaults to None, in which case an empty list is used. |
None
|
response_schema
|
ResponseSchema | None
|
The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None. |
None
|
output_analytics
|
bool
|
Whether to output the invocation analytics. Defaults to False. |
False
|
retry_config
|
RetryConfig | None
|
The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used. |
None
|
input_transformers
|
list[InputTransformer] | None
|
The list of input transformers to use. Defaults to None, in which case no input transformers will be used. |
None
|
output_transformers
|
list[OutputTransformer] | None
|
The list of output transformers to use. Defaults to None, in which case no output transformers will be used. |
None
|
input_transformer
|
InputTransformerType | None
|
Deprecated param to define input transformer. Will be removed in v0.7. Defaults to None. |
None
|
output_transformer
|
OutputTransformerType | None
|
Deprecated param to define output transformer. Will be removed in v0.7. Defaults to None. |
None
|
thinking
|
bool | ThinkingConfig
|
A boolean or ThinkingConfig object to configure thinking. Defaults to False. |
False
|
batch
cached
property
The batch operations for the language model.
Returns:
| Name | Type | Description |
|---|---|---|
BatchOperations |
BatchOperations
|
The batch operations for the language model. |
data_store
cached
property
The data store operations for the language model.
Returns:
| Name | Type | Description |
|---|---|---|
DataStoreOperations |
DataStoreOperations
|
The data store operations for the language model. |
file
cached
property
The file operations for the language model.
Returns:
| Name | Type | Description |
|---|---|---|
FileOperations |
FileOperations
|
The file operations for the language model. |
model_id
property
The model ID of the language model.
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The model ID of the language model. |
model_name
property
The name of the language model.
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The name of the language model. |
model_provider
property
The provider of the language model.
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The provider of the language model. |
prompt
cached
property
The prompt operations for the language model.
Returns:
| Name | Type | Description |
|---|---|---|
PromptOperations |
PromptOperations
|
The prompt operations for the language model. |
skill
cached
property
The skill operations for the language model.
Returns:
| Name | Type | Description |
|---|---|---|
SkillOperations |
SkillOperations
|
The skill operations for the language model. |
clear_response_schema()
Clears the response schema for the language model.
This method clears the response schema for the language model by calling the set_response_schema method with
None.
clear_tools()
Clears the tools for the language model.
This method clears the tools for the language model by calling the set_tools method with an empty list.
count_input_tokens(messages)
async
Counts the input tokens for an invocation request inputs.
This method counts the input tokens for an invocation request inputs. This method is useful for: 1. Estimating the cost of an invocation request before invoking the language model. 2. Checking if the invocation request is too large to be processed by the language model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
messages
|
LMInput
|
The input messages for the language model. 1. If a list of Message objects is provided, it is used as is. 2. If a list of MessageContent or a string is provided, it is converted into a user message. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
The number of input tokens for the invocation request. |
Raises:
| Type | Description |
|---|---|
TimeoutError
|
If the token counting times out. |
get_context_window()
async
Gets the context window for the language model.
Returns:
| Name | Type | Description |
|---|---|---|
ContextWindow |
ContextWindow
|
The context window for the language model. |
Raises:
| Type | Description |
|---|---|
TimeoutError
|
If the context window retrieval times out. |
invoke(messages, hyperparameters=None, event_emitter=None, max_calls=1)
async
Invokes the language model.
This method processes the language model invocation request. If tool calls are present in the output, the method will invoke the tools and feed the results back to the language model. This process will repeat until the tool calls are no longer present in the output or the maximum number of calls is reached.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
messages
|
LMInput
|
The input messages for the language model. 1. If a list of Message objects is provided, it is used as is. 2. If a list of MessageContent or a string is provided, it is converted into a user message. |
required |
hyperparameters
|
dict[str, Any] | None
|
A dictionary of hyperparameters for the language model. Defaults to None, in which case the default hyperparameters are used. |
None
|
event_emitter
|
EventEmitter | None
|
The event emitter for streaming tokens. If provided, streaming invocation is enabled. Defaults to None. |
None
|
max_calls
|
int
|
The max number of times the language model can be invoked. Multiple invocations enable automatic tool invocation when needed. Defaults to 1. |
1
|
Returns:
| Name | Type | Description |
|---|---|---|
LMOutput |
LMOutput
|
The generated response from the language model. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
set_response_schema(response_schema)
Sets the response schema for the language model.
This method sets the response schema for the language model. Any existing response schema will be replaced.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
response_schema
|
ResponseSchema | None
|
The response schema to be used. |
required |
set_tools(tools)
Sets the tools for the language model.
This method sets the tools for the language model. Any existing tools will be replaced.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tools
|
list[LMTool]
|
The list of tools to be used. |
required |
InputType
Defines valid input types in LM invokers JSON schema.
Key
Defines valid keys in LM invokers JSON schema.