Filter Extractor

Modules concerning the filter extractor used in Gen AI applications.

`FieldDescriptor`

Bases: BaseModel

Describes a single filterable field in the target datastore index.

Attributes:

Name	Type	Description
`name`	`str`	Dot-notation path to the field.
`python_type`	`Any`	Any type expressible as a JSON Schema (validated at construction).
`description`	`str \| None`	Human-readable description of the field for LLM context.
`allowed_values`	`list[Any] \| None`	Enumerated allowed values for enum-like fields.
`allowed_operators`	`list[FilterOperator]`	List of filter operators permitted on this field.
`example_values`	`list[Any] \| None`	Example values to help LLM understand field semantics.

`FieldExtractionRule`

Bases: BaseModel

Bundles a rule and operator for extracting a single filter field.

The rule is applied to the original input string. Regex and str rules produce str | None and go through coercion. Callable rules may return Any: if the return value is non-str and non-None, coercion is skipped but the value is still validated against FieldDescriptor.python_type when a catalog is present.

Attributes:

Name	Type	Description
`rule`	`Pattern[str] \| str \| Callable[[str], Any]`	Rule applied to the input string.
`operator`	`FilterOperator`	The operator used when constructing the `FilterClause`.

`FieldKey`

Keys for field metadata.

`FilterExtractionResult`

Bases: BaseModel

Typed output of BaseFilterExtractor.extract().

Attributes:

Name	Type	Description
`query`	`str`	The extracted query string.
`query_filter`	`QueryFilter \| None`	The extracted query filter.
`query_options`	`QueryOptions`	The extracted query options.
`retriever_params`	`RetrieverParams \| None`	The extracted retriever parameters.

`LMFilterExtractor(lm_invoker, metadata_catalog, retrieval_params_type=None, response_schema=None)`

Bases: BaseFilterExtractor, LMComponent

Extracts typed retrieval parameters using a language model with native structured output.

The process follows these steps: 1. At construction time, the metadata catalog is compiled into a constrained subclass of FilterExtractionResult via pydantic.create_model. 2. Each FieldDescriptor becomes a FilterClause subclass with Literal-constrained key, operator, and optionally value. 3. These clause models are combined into a QueryFilter subclass and wired into the result class, which is passed as response_schema to the LM invoker.

Examples:

catalog = MetadataCatalog(fields=[
    FieldDescriptor(
        name="metadata.department",
        description="Owning department",
        python_type=str,
        allowed_values=["InfoSec", "HR"],
        allowed_operators=[FilterOperator.EQ, FilterOperator.IN],
    )
])

extractor = LMFilterExtractor.from_metadata_catalog(
    model_id="openai/gpt-4.1-mini",
    metadata_catalog=catalog,
    retrieval_params_type=VectorRetrieverParams,
)

params = await extractor.extract("top 5 InfoSec policies")

Attributes:

Name	Type	Description
`lm_invoker`	`BaseLMInvoker`	Invoker configured with the constrained `response_schema` and baked templates.
`metadata_catalog`	`MetadataCatalog`	Filterable-field schema used to generate the constrained subclass.
`retrieval_params_type`	`RetrieverParamsType \| None`	Specific `RetrieverParams` subclass to produce, or `None`.

Initialize LMFilterExtractor.

The constrained schema is resolved in the following priority order: 1. If lm_invoker.response_schema is set, it must be a subclass of FilterExtractionResult (raises ValueError otherwise) and is used as-is. 2. If response_schema is provided, it is used directly. 3. Otherwise the schema is built from metadata_catalog.

Parameters:

Name	Type	Description	Default
`lm_invoker`	`BaseLMInvoker`	If already configured with a `response_schema`, that schema is validated and used. Use `from_metadata_catalog()` to get this wired correctly.	required
`metadata_catalog`	`MetadataCatalog`	Required. Defines filterable fields.	required
`retrieval_params_type`	`RetrieverParamsType`	Optional retriever-specific params type. Defaults to None for agnostic extraction.	`None`
`response_schema`	`type[FilterExtractionResult] \| None`	Optional explicit schema override. Ignored when `lm_invoker.response_schema` is already set. Defaults to None.	`None`

Raises:

Type	Description
`ValueError`	If `lm_invoker.response_schema` or `response_schema` is set but is not a subclass of `FilterExtractionResult`.

`from_metadata_catalog(model_id, metadata_catalog, retrieval_params_type=None, system_template=None, user_template=None)` `classmethod`

Build an extractor from a metadata catalog.

Constructs the constrained FilterExtractionResult subclass from the catalog, configures the LM invoker with it as response_schema, and returns a fully wired instance.

Parameters:

Name	Type	Description	Default
`model_id`	`str`	LLM identifier, e.g. `"openai/gpt-4.1-mini"`.	required
`metadata_catalog`	`MetadataCatalog`	Required. Defines filterable fields for this index.	required
`retrieval_params_type`	`RetrieverParamsType`	Optional retriever-specific params type to produce. Defaults to None for agnostic extraction.	`None`
`system_template`	`str \| None`	Instruction prompt. Defaults to `default_system_template`.	`None`
`user_template`	`str \| None`	Must contain `{query}` placeholder. Defaults to `default_user_template`.	`None`

Returns:

Name	Type	Description
`LMFilterExtractor`	`'LMFilterExtractor'`	An initialized LMFilterExtractor instance.

`MetadataCatalog`

Bases: BaseModel

Schema of filterable fields for a target datastore index.

Attributes:

Name	Type	Description
`fields`	`list[FieldDescriptor]`	List of field descriptors defining the available filterable fields.

`from_pydantic(model, prefix)` `classmethod`

Build a MetadataCatalog by reflecting on a Pydantic document model.

Extracts field metadata by introspecting the Pydantic model's fields. For each field, it resolves the type annotation, maps it to a datastore-compatible type string, and extracts optional field attributes (description, allowed_values, allowed_operators, example_values) from Pydantic field attributes.

Examples:

class Meta(BaseModel):
    status: Literal["active", "archived"]
    score: float

class Doc(BaseModel):
    meta: Meta

catalog = MetadataCatalog.from_pydantic(Doc, prefix="meta")

Args: model (type[BaseModel]): The Pydantic document model class. prefix (str): Dot-notation path to the sub-model to reflect.

Returns:

Name	Type	Description
`MetadataCatalog`	`MetadataCatalog`	Metadata catalog with one FieldDescriptor per reflected field.

Raises:

Type	Description
`ValueError`	If `prefix` does not refer to a BaseModel field in `model`, or if a field type cannot be mapped.

`RuleBasedFilterConfig`

Bases: BaseModel

Pure data holder encoding all extraction rules for RuleBasedFilterExtractor.

Mirrors the shape of FilterExtractionResult — one rule field per output field. Has no behaviour: no compilation, no side effects. All compilation and validation happens in RuleBasedFilterExtractor.__init__.

When query is None, the input is passed through unchanged (result.query == input.strip()).

Attributes:

Name	Type	Description
`query`	`ExtractorRule \| None`	Rule for extracting the query string. `None` means pass-through. Defaults to None.
`query_filter`	`dict[str, FieldExtractionRule] \| None`	Mapping of field names to `FieldExtractionRule` objects. Applied independently to the original input. Defaults to None.
`query_options`	`Callable[[str], QueryOptions] \| None`	Callable receiving the original input string. Must return `QueryOptions`. Defaults to None.

`RuleBasedFilterExtractor(rules, metadata_catalog=None, retrieval_params_type=None)`

Bases: BaseFilterExtractor

Deterministic filter extractor that applies regex, callable, or raw-string rules.

Extracts typed retrieval parameters from a query string without calling an LM. Returns the same FilterExtractionResult as LMFilterExtractor, so callers need not branch on extractor type.

When metadata_catalog is provided: 1. query_filter keys are validated against catalog field names at construction. 2. Each FieldExtractionRule.operator is validated against FieldDescriptor.allowed_operators at construction. 3. Extracted values are validated against allowed_values and coerced to python_type at extraction time.

When metadata_catalog is omitted, no catalog-based validation occurs.

retrieval_params_type is instantiated fresh per _extract call with all defaults to avoid shared mutable state.

Examples:

from gllm_retrieval.filter_extractor import RuleBasedFilterExtractor
from gllm_retrieval.filter_extractor.schema import (
    FieldExtractionRule,
    RuleBasedFilterConfig,
    MetadataCatalog,
)
from gllm_datastore.core.filters.schema import FilterOperator

rules = RuleBasedFilterConfig(
    query=r"(?P<value>.+)",
    query_filter={
        "department": FieldExtractionRule(
            rule=r"dept[:\\s]+(?P<value>\\w+)",
            operator=FilterOperator.EQ,
        ),
    },
)

catalog = MetadataCatalog(fields=[
    FieldDescriptor(
        name="department",
        python_type=str,
        allowed_values=["InfoSec", "HR"],
        allowed_operators=[FilterOperator.EQ],
    ),
])

extractor = RuleBasedFilterExtractor(rules=rules, metadata_catalog=metadata_catalog)
result = await extractor.extract("Find docs dept: InfoSec")

# result.query_filter contains FilterClause(key="department", value="InfoSec", ...)

Initializes a new instance of RuleBasedFilterExtractor.

Compiles all string rules and validates catalog constraints at construction time so that misconfiguration is surfaced before any query is processed.

Parameters:

Name	Type	Description	Default
`rules`	`RuleBasedFilterConfig`	Encoding all extraction logic.	required
`metadata_catalog`	`MetadataCatalog \| None`	When provided, validates field keys and operators at construction, and extracted values at extraction time. Defaults to None.	`None`
`retrieval_params_type`	`RetrieverParamsType`	Type to instantiate per call with all defaults. Defaults to None.	`None`

Raises:

Type	Description
`error`	If a `str` rule in `rules.query` or any `rules.query_filter` value is not a valid regex pattern.
`ValueError`	If `metadata_catalog` is provided and a key in `rules.query_filter` is absent from the catalog. If `metadata_catalog` is provided and a `FieldExtractionRule.operator` is not in `FieldDescriptor.allowed_operators`.

Filter Extractor

FieldDescriptor

FieldExtractionRule

FieldKey

FilterExtractionResult

LMFilterExtractor(lm_invoker, metadata_catalog, retrieval_params_type=None, response_schema=None)

from_metadata_catalog(model_id, metadata_catalog, retrieval_params_type=None, system_template=None, user_template=None) classmethod

MetadataCatalog

from_pydantic(model, prefix) classmethod

RuleBasedFilterConfig

RuleBasedFilterExtractor(rules, metadata_catalog=None, retrieval_params_type=None)

`FieldDescriptor`

`FieldExtractionRule`

`FieldKey`

`FilterExtractionResult`

`LMFilterExtractor(lm_invoker, metadata_catalog, retrieval_params_type=None, response_schema=None)`

`from_metadata_catalog(model_id, metadata_catalog, retrieval_params_type=None, system_template=None, user_template=None)` `classmethod`

`MetadataCatalog`

`from_pydantic(model, prefix)` `classmethod`

`RuleBasedFilterConfig`

`RuleBasedFilterExtractor(rules, metadata_catalog=None, retrieval_params_type=None)`