Reference Formatter

Modules concerning the reference formatters used in Gen AI applications.

`DMReferenceFormatter(dm_invoker, threshold=_DEFAULT_THRESHOLD, max_concurrency=_DEFAULT_MAX_CONCURRENCY, relevance_question=None, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None, batch_size=_DEFAULT_BATCH_SIZE)`

Bases: BaseReferenceFormatter, DMComponent

Filter candidate reference chunks using a decisions model.

Chunks are evaluated in batches with one binary relevance question per chunk. Chunks whose positive probability meets the configured threshold are retained, in their original order.

Examples:

Using the from_config class method

Build the decisions-model invoker from a provider-qualified model ID.

formatter = DMReferenceFormatter.from_config(model_id="openrouter/typesafe/jev-1.13")
references = await formatter.format_reference(response=response, chunks=chunks)

Using the class constructor

Useful when supplying a preconfigured decisions-model invoker.

dm_invoker = OpenRouterDMInvoker(model_name="typesafe/jev-1.13")
formatter = DMReferenceFormatter(dm_invoker=dm_invoker)
references = await formatter.format_reference(response=response, chunks=chunks)

Attributes:

Name	Type	Description
`dm_invoker`	`BaseDMInvoker`	The decisions model invoker used to evaluate chunks.
`threshold`	`float`	Minimum positive relevance probability required to retain a chunk.
`batch_size`	`int`	Number of chunks evaluated in each decisions-model call.
`max_concurrency`	`int`	Maximum number of simultaneous decisions-model calls.
`relevance_question`	`DMQuestion`	Binary relevance definition used to assess each chunk. Its instructions and criteria are used to construct one named `noul` question per chunk; chunk targeting is added internally.
`stringify`	`bool`	Whether formatted references are returned as a string. If False, returns chunks.
`format_references_func`	`Callable[[list[str]], str]`	Formats rendered chunk references as a string.
`format_chunk_func`	`Callable[[Chunk], str]`	Formats an individual chunk as a reference.
`streamable`	`bool`	Whether formatted references are streamed when an event emitter is provided.
`context_config`	`ContextConfig \| None`	Configuration for rendering chunk context before evaluation.

Initialize the formatter.

Parameters:

Name	Type	Description	Default
`dm_invoker`	`BaseDMInvoker`	The decisions model invoker used to score each chunk.	required
`threshold`	`float`	Minimum positive probability required to retain a chunk. Defaults to _DEFAULT_THRESHOLD.	`_DEFAULT_THRESHOLD`
`max_concurrency`	`int`	Maximum number of simultaneous decisions-model calls. Defaults to _DEFAULT_MAX_CONCURRENCY.	`_DEFAULT_MAX_CONCURRENCY`
`relevance_question`	`DMQuestion \| None`	A `noul` relevance definition for each chunk. Its instructions and criteria are used to construct one named question per chunk; Defaults to a built-in question that checks whether the chunk supports the response.	`None`
`stringify`	`bool`	Whether formatted references are returned as a string. Defaults to True.	`True`
`format_references_func`	`Callable[[list[str]], str] \| None`	Custom reference string formatter. Defaults to None.	`None`
`format_chunk_func`	`Callable[[Chunk], str] \| None`	Custom formatter for individual chunks. Defaults to None.	`None`
`format_chunk_metadata_map`	`dict[str, str] \| None`	Mapping for default chunk metadata fields. Defaults to None.	`None`
`streamable`	`bool`	Whether to emit reference events when an event emitter is provided. Defaults to True.	`True`
`context_config`	`ContextConfig \| None`	Configuration for rendering chunk context. Defaults to None.	`None`
`batch_size`	`int`	Number of chunks to evaluate in each decisions-model call. Defaults to _DEFAULT_BATCH_SIZE.	`_DEFAULT_BATCH_SIZE`

Raises:

Type	Description
`GenerationValueError`

`LMBasedReferenceFormatter(lm_request_processor, batch_size=DEFAULT_BATCH_SIZE, reference_metadata=DEFAULT_REFERENCE_METADATA, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None)`

Bases: BaseReferenceFormatter, UsesLM

A reference formatter that utilizes a language model to filter the candidate chunks.

This class uses a language model to filter the candidate chunks based on their relevance to the generated response. It then formats the relevant chunks into a reference string or a list of chunks depending on the stringify attribute.

Attributes:

Name	Type	Description
`lm_request_processor`	`LMRequestProcessor`	The request processor used to handle the candidate chunks filtering.
`batch_size`	`int`	The number of chunks to process in each batch.
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks.
`format_references_func`	`Callable[[list[str]], str]`	A function that formats a list of formatted chunks as a string of references.
`format_chunk_func`	`Callable[[Chunk], str]`	A function that formats a chunk as a reference.
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided.
`context_config`	`ContextConfig \| None`	Configuration for rendering context text from chunks.

LM request processor configuration

The LM request processor for the LMBasedReferenceFormatter must take the following variables as its input: 1. response: The synthesized response. 2. context: The formatted context string of the candidate chunks in the following format: reference_metadata: content:

The reference_metadata is the metadata of the chunk that will be referenced by the language model in the filtering process, and can be customized via the reference_metadata parameter. By default, it is set to file_name.

The LM request processor must also output a list of reference metadata of the chunks that are deemed as relevant to the synthesized response, for example:

RelevantChunks(
    reference_metadata=["<reference_metadata_1>", "<reference_metadata_2>"]
)

Using preset

This LMBasedReferenceFormatter has a preset with a predefined system template and response schema. This preset can be used for simple usage as only the model id is required. The preset utilizes the LM invoker's structured output feature and the default reference metadata.

Usage example:

reference_formatter = LMBasedReferenceFormatter.from_preset(model_id="openai/gpt-4.1-mini")
reference = await reference_formatter.format(response=response, chunks=chunks)

Using LM invoker's structured output

Using the LM invoker's structured output feature is the recommended way to use the LMBasedReferenceFormatter with customized LM request processor.

Usage example:

class RelevantChunks(BaseModel):
    file_name: list[str]
lm_request_processor = build_lm_request_processor(
    model_id="...",
    system_template="...",
    config={"response_schema": RelevantChunks}
)
reference_formatter = LMBasedReferenceFormatter(lm_request_processor)
reference = await reference_formatter.format(response=response, chunks=chunks)

Using output parser

An alternative way to use the LMBasedReferenceFormatter is to use a separate output parser. This is useful when the language model does not support native structured output feature.

Usage example:

lm_request_processor = build_lm_request_processor(
    model_id="...",
    system_template="...",
    output_parser=JSONOutputParser()
)
reference_formatter = LMBasedReferenceFormatter(lm_request_processor)
reference = await reference_formatter.format(response=response, chunks=chunks)

Initializes a new instance of the LMBasedReferenceFormatter class.

Parameters:

Name	Type	Description	Default
`lm_request_processor`	`LMRequestProcessor`	The request processor used to handle the candidate chunks filtering.	required
`batch_size`	`int`	The number of chunks to process in each batch. Defaults to DEFAULT_BATCH_SIZE.	`DEFAULT_BATCH_SIZE`
`reference_metadata`	`str`	The metadata of the chunk to be referenced by the language model in the filtering process. Defaults to DEFAULT_REFERENCE_METADATA.	`DEFAULT_REFERENCE_METADATA`
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks. Defaults to True.	`True`
`format_references_func`	`Callable[[list[str]], str] \| None`	A function that formats a list of formatted chunks as a string of references. Defaults to None, in which case `format_references_as_string` will be used.	`None`
`format_chunk_func`	`Callable[[Chunk], str] \| None`	A function that formats a chunk as a reference. Defaults to None, in which case `format_chunk_as_string` will be used.	`None`
`format_chunk_metadata_map`	`dict[str, str] \| None`	A dictionary mapping the metadata keys needed by the `format_chunk_func` to the actual chunk metadata keys. The keys in the dictionary must match the metadata keys needed by the `format_chunk_func`. Defaults to None, in which case `DEFAULT_FORMAT_CHUNK_METADATA_MAP` will be used.	`None`
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided. Defaults to True.	`True`
`context_config`	`ContextConfig \| None`	Configuration for rendering context text from chunks. Defaults to None.	`None`

`from_preset(model_id, **kwargs)` `classmethod`

Creates a reference formatter with the preset system template.

Parameters:

Name	Type	Description	Default
`model_id`	`str`	The model id.	required
`**kwargs`	`Any`	Additional keyword arguments.	`{}`

Returns:

Name	Type	Description
`LMBasedReferenceFormatter`	`LMBasedReferenceFormatter`	A reference formatter with the preset system template.

`LMReferenceFormatter(lm_invoker, batch_size=_DEFAULT_BATCH_SIZE, reference_metadata=_DEFAULT_REFERENCE_METADATA, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, fallback_lms=None, context_config=None)`

Bases: BaseReferenceFormatter, LMComponent

A reference formatter that utilizes a language model to filter the candidate chunks.

This class uses a language model to filter the candidate chunks based on their relevance to the generated response. It then formats the relevant chunks into a reference string or a list of chunks depending on the stringify attribute.

Examples:

Using the from_preset class method

The most straightforward method. Uses the built-in prompt templates and response schema.

formatter = LMReferenceFormatter.from_preset("openai/gpt-5-nano")
references = await formatter.format(response=response, chunks=chunks)

Using the from_config class method

Useful for customizing several configurations, such as prompt templates and response schema.

formatter = LMReferenceFormatter.from_config(
    model_id="openai/gpt-5-nano",
    system_template=SYSTEM_TEMPLATE,
    config={"response_schema": create_model("RelevantChunks", **{"file_name": (list[str], ...)})},
)

Using the class constructor

Useful for passing a pre-defined LM invokers

formatter = LMReferenceFormatter(lm_invoker=lm_invoker)
references = await formatter.format(response=response, chunks=chunks)

Attributes:

Name	Type	Description
`lm_invoker`	`BaseLMInvoker`	The language model invoker used to handle the candidate chunks filtering.
`batch_size`	`int`	The number of chunks to process in each batch.
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks.
`format_references_func`	`Callable[[list[str]], str]`	A function that formats a list of formatted chunks as a string of references.
`format_chunk_func`	`Callable[[Chunk], str]`	A function that formats a chunk as a reference.
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided.
`context_config`	`ContextConfig \| None`	Configuration for rendering context text from chunks.

Initializes a new instance of the LMReferenceFormatter class.

Parameters:

Name	Type	Description	Default
`lm_invoker`	`BaseLMInvoker`	The language model invoker to use for filtering the candidate chunks.	required
`batch_size`	`int`	The number of chunks to process in each batch. Defaults to _DEFAULT_BATCH_SIZE.	`_DEFAULT_BATCH_SIZE`
`reference_metadata`	`str`	The metadata of the chunk to be referenced by the language model in the filtering process. Defaults to _DEFAULT_REFERENCE_METADATA.	`_DEFAULT_REFERENCE_METADATA`
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks. Defaults to True.	`True`
`format_references_func`	`Callable[[list[str]], str] \| None`	A function that formats a list of formatted chunks as a string of references. Defaults to None.	`None`
`format_chunk_func`	`Callable[[Chunk], str] \| None`	A function that formats a chunk as a reference. Defaults to None.	`None`
`format_chunk_metadata_map`	`dict[str, str] \| None`	A dictionary mapping the metadata keys needed by the `format_chunk_func` to the actual chunk metadata keys. The keys in the dictionary must match the metadata keys needed by the `format_chunk_func`. Defaults to None.	`None`
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided. Defaults to True.	`True`
`fallback_lms`	`list[BaseLMInvoker] \| None`	A list of language model invokers to use as fallback if the main language model fails. Defaults to None.	`None`
`context_config`	`ContextConfig \| None`	Configuration for rendering context text from chunks. Defaults to None.	`None`

`from_preset(model_id, **kwargs)` `classmethod`

Creates a reference formatter with the preset prompt templates and response schema.

Parameters:

Name	Type	Description	Default
`model_id`	`str`	The model id.	required
`**kwargs`	`Any`	Additional keyword arguments.	`{}`

Returns:

Name	Type	Description
`Self`	`Self`	A reference formatter with the preset prompt templates and response schema.

`NoOpReferenceFormatter(stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True)`

Bases: BaseReferenceFormatter

A reference formatter that returns all candidate chunks without filtering.

This class extends the BaseReferenceFormatter and returns all candidate chunks without filtering.

Attributes:

Name	Type	Description
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks.
`format_references_func`	`Callable[[list[str]], str]`	A function that formats a list of formatted chunks as a string of references.
`format_chunk_func`	`Callable[[Chunk], str]`	A function that formats a chunk as a reference.
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided.

Initializes a new instance of the NoOpReferenceFormatter class.

Parameters:

Name	Type	Description	Default
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks. Defaults to True.	`True`
`format_references_func`	`Callable[[list[str]], str] \| None`	A function that formats a list of formatted chunks as a string of references. Defaults to None, in which case `format_references_as_string` will be used.	`None`
`format_chunk_func`	`Callable[[Chunk], str] \| None`	A function that formats a chunk as a reference. Defaults to None, in which case `format_chunk_as_string` will be used.	`None`
`format_chunk_metadata_map`	`dict[str, str] \| None`	A dictionary mapping the metadata keys needed by the `format_chunk_func` to the actual chunk metadata keys. The keys in the dictionary must match the metadata keys needed by the `format_chunk_func`. Defaults to None, in which case `DEFAULT_FORMAT_CHUNK_METADATA_MAP` will be used.	`None`
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided. Defaults to True.	`True`

`SimilarityBasedReferenceFormatter(em_invoker, threshold=DEFAULT_THRESHOLD, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None)`

Bases: SimilarityReferenceFormatter

A deprecated compatibility formatter that filters chunks by similarity to the synthesized response.

This class retains the legacy public name and constructor while delegating all behavior to SimilarityReferenceFormatter. New code should use the canonical class instead.

Attributes:

Name	Type	Description
`em_invoker`	`BaseEMInvoker`	The embedding model invoker for generating embeddings for the response and chunk contents.
`threshold`	`float`	The similarity threshold for filtering chunks.
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks.
`format_references_func`	`Callable[[list[str]], str]`	A function that formats a list of formatted chunks as a string of references.
`format_chunk_func`	`Callable[[Chunk], str]`	A function that formats a chunk as a reference.
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided.
`context_config`	`ContextConfig \| None`	Configuration for rendering context text from chunks.

Initializes the deprecated formatter name with the legacy constructor API.

Parameters:

Name	Type	Description	Default
`em_invoker`	`BaseEMInvoker`	The embedding model invoker for generating embeddings for the response and chunk contents.	required
`threshold`	`float`	The similarity threshold for filtering chunks. Defaults to DEFAULT_THRESHOLD.	`DEFAULT_THRESHOLD`
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks. Defaults to True.	`True`
`format_references_func`	`Callable[[list[str]], str] \| None`	A function that formats a list of formatted chunks as a string of references. Defaults to None, in which case `format_references_as_string` will be used.	`None`
`format_chunk_func`	`Callable[[Chunk], str] \| None`	A function that formats a chunk as a reference. Defaults to None, in which case `format_chunk_as_string` will be used.	`None`
`format_chunk_metadata_map`	`dict[str, str] \| None`	A dictionary mapping the metadata keys needed by the `format_chunk_func` to the actual chunk metadata keys. Defaults to None, in which case `DEFAULT_FORMAT_CHUNK_METADATA_MAP` will be used.	`None`
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided. Defaults to True.	`True`
`context_config`	`ContextConfig \| None`	Configuration for rendering context text from chunks. Defaults to None.	`None`

`SimilarityReferenceFormatter(em_invoker, threshold=DEFAULT_THRESHOLD, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None)`

Bases: BaseReferenceFormatter, EMComponent

A reference formatter that filters the candidate chunks based on their relevance to the synthesized response.

This class uses an embedding model to compute the similarity between the synthesized response and the chunk contents. It filters out chunks whose similarity score falls below a specified threshold. Then, it formats the relevant chunks into a reference string or a list of chunks depending on the stringify attribute.

Attributes:

Name	Type	Description
`em_invoker`	`BaseEMInvoker`	The embedding model invoker for generating embeddings for the response and chunk contents.
`threshold`	`float`	The similarity threshold for filtering chunks.
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks.
`format_references_func`	`Callable[[list[str]], str]`	A function that formats a list of formatted chunks as a string of references.
`format_chunk_func`	`Callable[[Chunk], str]`	A function that formats a chunk as a reference.
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided.
`context_config`	`ContextConfig \| None`	Configuration for rendering context text from chunks.

Initializes a new instance of the SimilarityReferenceFormatter class.

Parameters:

Name	Type	Description	Default
`em_invoker`	`BaseEMInvoker`	The embedding model invoker for generating embeddings for the response and chunk contents.	required
`threshold`	`float`	The similarity threshold for filtering chunks. Defaults to DEFAULT_THRESHOLD.	`DEFAULT_THRESHOLD`
`stringify`	`bool`	Whether to format the references as a string. If False, the references will be returned as a list of chunks. Defaults to True.	`True`
`format_references_func`	`Callable[[list[str]], str] \| None`	A function that formats a list of formatted chunks as a string of references. Defaults to None, in which case `format_references_as_string` will be used.	`None`
`format_chunk_func`	`Callable[[Chunk], str] \| None`	A function that formats a chunk as a reference. Defaults to None, in which case `format_chunk_as_string` will be used.	`None`
`format_chunk_metadata_map`	`dict[str, str] \| None`	A dictionary mapping the metadata keys needed by the `format_chunk_func` to the actual chunk metadata keys. The keys in the dictionary must match the metadata keys needed by the `format_chunk_func`. Defaults to None, in which case `DEFAULT_FORMAT_CHUNK_METADATA_MAP` will be used.	`None`
`streamable`	`bool`	A flag to indicate whether the formatted references will be streamed if an event emitter is provided. Defaults to True.	`True`
`context_config`	`ContextConfig \| None`	Configuration for rendering context text from chunks. Defaults to None.	`None`

Reference Formatter

Using the from_config class method

Using the class constructor

LMBasedReferenceFormatter(lm_request_processor, batch_size=DEFAULT_BATCH_SIZE, reference_metadata=DEFAULT_REFERENCE_METADATA, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None)

from_preset(model_id, **kwargs) classmethod

LMReferenceFormatter(lm_invoker, batch_size=_DEFAULT_BATCH_SIZE, reference_metadata=_DEFAULT_REFERENCE_METADATA, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, fallback_lms=None, context_config=None)

Using the from_preset class method

Using the from_config class method

Using the class constructor

from_preset(model_id, **kwargs) classmethod

NoOpReferenceFormatter(stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True)

SimilarityBasedReferenceFormatter(em_invoker, threshold=DEFAULT_THRESHOLD, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None)

SimilarityReferenceFormatter(em_invoker, threshold=DEFAULT_THRESHOLD, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None)

`LMBasedReferenceFormatter(lm_request_processor, batch_size=DEFAULT_BATCH_SIZE, reference_metadata=DEFAULT_REFERENCE_METADATA, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None)`

`from_preset(model_id, **kwargs)` `classmethod`

`LMReferenceFormatter(lm_invoker, batch_size=_DEFAULT_BATCH_SIZE, reference_metadata=_DEFAULT_REFERENCE_METADATA, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, fallback_lms=None, context_config=None)`

`from_preset(model_id, **kwargs)` `classmethod`

`NoOpReferenceFormatter(stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True)`

`SimilarityBasedReferenceFormatter(em_invoker, threshold=DEFAULT_THRESHOLD, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None)`

`SimilarityReferenceFormatter(em_invoker, threshold=DEFAULT_THRESHOLD, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, context_config=None)`