Index

GLLM Evaluator SDK - A comprehensive evaluation framework for Generative AI applications.

This package provides a robust, extensible framework for evaluating LLM outputs, AI Agent responses, and RAG (Retrieval-Augmented Generation) systems. It includes support for various evaluation metrics, datasets, evaluators, and experiment tracking integrations.

Key features: - Support for multiple evaluation metrics (generation, retrieval, agent-based) - Integration with popular evaluators (RAGAS, DeepEval, LangChain) - Flexible dataset loading from various sources (HuggingFace, spreadsheets, CSV, JSONL) - Experiment tracking integration - Async-first design for high-performance evaluation

is_failed_result = ErrorReporter.is_failed_result module-attribute

Tells whether a metric, evaluator or judge result records a failure. See ErrorReporter.is_failed_result.

AttachmentRef

Bases: BaseModel

Row-level attachment reference for multimodal evaluation inputs.

Attributes:

Name	Type	Description
uri	str	Attachment URI or portable reference location.
mime_type	str | None	MIME type hint for the attachment. Defaults to None.
metadata	dict[str, Any]	Caller-provided metadata for downstream adapters. Defaults to an empty dict.

BaseEvalError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseError

Root of the errors owned by gllm-evals.

Catch gllm_core.exceptions.BaseError to handle every normalized gllm failure, including preserved gllm-inference errors; catch BaseEvalError only for failures owned by gllm-evals.

Every message says what happened and then what to do next. A class's default_message holds both, and a raise site that passes its own message writes both as well.

Attributes:

Name	Type	Description
context	ErrorContext | None	Where the failure happened, if known.
children	tuple[BaseError, ...]	Normalized failures this error summarizes, e.g. every failed judge.

Initializes the error.

Parameters:

Name	Type	Description	Default
message	str | None	The complete message: what happened, then what to do next. It must be as safe to show as the default: never include row content, paths, URLs, credentials or upstream error text. Defaults to None, which uses default_message.	None
context	ErrorContext | None	Where the failure happened. Defaults to None.	None
children	Iterable[BaseError]	Normalized failures this error summarizes. Defaults to an empty tuple.	()
debug_info	DebugInfo | None	Sanitized internal detail. Defaults to None.	None
recovery	RecoveryInfo | None	Recovery replacing the class default for this occurrence, for example an operation that is safe to retry. Defaults to None.	None

origin property

Who owns the original failure.

Returns:

Name	Type	Description
ErrorOrigin	ErrorOrigin	The context's origin, or "gllm-evals" when no context was given.

from_error(raw_error, *, context, status_code=None, recovery=None, message=None) classmethod

Builds this error for a raw failure without copying the raw error's message.

Use it at a raise site that already knows the class. It does not classify: translating an unknown failure is EvalErrorTranslator's job.

Parameters:

Name	Type	Description	Default
raw_error	Exception | None	The raw failure. Only its type name is kept. None when the SDK detected the failure itself, for example from an HTTP status, and no exception was raised.	required
context	ErrorContext	Where the failure happened.	required
status_code	int | None	Numeric status reported by the dependency. Defaults to None.	None
recovery	RecoveryInfo | None	Recovery replacing the class default. Defaults to None.	None
message	str | None	The complete message replacing the class default: what happened, then what to do next. It must not contain row content, paths, URLs or upstream error text. Defaults to None.	None

Returns:

Name	Type	Description
Self	Self	The normalized error, with raw_error as its __cause__ when one is given.

ConversationalTestCase

Bases: BaseModel

Canonical top-level input object for multi-turn evaluation.

Attributes:

Name	Type	Description
turns	list[Turn]	Ordered conversation transcript. Must contain at least one turn.
chatbot_role	str | None	The assistant's intended role / persona. Defaults to None.
expected_output	str | None	Primary top-level reference field, mapped to DeepEval's expected_outcome by the conversion layer. Defaults to None.
scenario	str | None	Conversation scenario description. Defaults to None.
attachments	dict[str, AttachmentRef]	Row-level attachment references for [ATTACHMENT:<id>] placeholders. Defaults to an empty dict.

from_dict(row) classmethod

Build a ConversationalTestCase from a raw mapping.

Parameters:

Name	Type	Description	Default
row	dict[str, Any]	Raw row data.	required

Returns:

Name	Type	Description
ConversationalTestCase	ConversationalTestCase	The constructed test case.

Raises:

Type	Description
ValidationError	If the row cannot be validated as a ConversationalTestCase.

from_messages(messages, *, chatbot_role=None, **conv_fields) classmethod

Build a ConversationalTestCase from a gllm_inference Message transcript.

Mapping applied to each message:

• system -> chatbot_role
• user / assistant -> one Turn each (not merged)
• tool-call request -> Turn.tools_called
• tool result -> folded into the matching ToolCall.output, not its own turn
• Thinking content part -> dropped

Parameters:

Name	Type	Description	Default
messages	list[Message]	Ordered inference transcript.	required
chatbot_role	str | None	Extra system context, newline-joined ahead of any system messages. Defaults to None.	None
**conv_fields	Any	Forwarded verbatim to the constructor -- e.g. expected_output, scenario, attachments.	{}

Returns:

Name	Type	Description
ConversationalTestCase	ConversationalTestCase	The constructed test case.

Raises:

Type	Description
EvalInputError	If messages is not a sequence of Message objects, if turns is passed in conv_fields, if the transcript produces no user/assistant turns, or if the resulting test case fails validation.
EvalUnsupportedOperationError	If a message carries an Attachment content part.

EvalAccessError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, PermissionError

A local or remote operation is denied because authentication or authorization fails.

EvalAggregationError(message=None, *, failures=None, **kwargs)

Bases: BaseEvalError, ValueError

Aggregation cannot compute a result from the available outcomes, although its setup is valid.

Initializes the error.

Parameters:

Name	Type	Description	Default
message	str | None	What happened. Defaults to None, which uses the class default.	None
failures	dict[str, Any] | None	The failed MetricResult of each metric, by metric name, when every metric failed on a row. Defaults to None.	None
**kwargs	Any	The other BaseEvalError arguments.	{}

EvalConfigurationError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

Evaluation setup is invalid, incomplete, ambiguous or inconsistent before any content is processed.

EvalConnectionError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ConnectionError

Connection establishment or transport fails, such as DNS, TLS, refusal or reset.

EvalDependencyUnavailableError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

A required integration is missing or verifiably incompatible with the installed runtime.

EvalExecutionError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, RuntimeError

An unexpected failure or invalid component state that no more specific class describes.

EvalIOError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, OSError

Local storage read, write, replace or cleanup fails for a reason other than absence or access denial.

EvalInputError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

Supplied evaluation content fails its input contract.

EvalJudgeResponseError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

A judge invocation completed but its answer cannot be used.

EvalJudgesExhaustedError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, RuntimeError

At least two judges in a fallback chain were attempted and none produced a usable result.

Pass every attempt's normalized failure as children and chain the last one as the cause. A single failed attempt is re-raised as its own type instead.

EvalRateLimitError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, RuntimeError

A non-inference integration explicitly rejects an operation because request-rate capacity is exhausted.

EvalResourceNotFoundError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, FileNotFoundError

A referenced file, directory, dataset, worksheet, object or session cannot be found.

EvalResultContractError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

A metric, evaluator or adapter violates the SDK result interface.

EvalSerializationError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

Encoding or decoding an SDK-owned persisted or exported artifact fails.

EvalServiceError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, RuntimeError

A non-inference remote service reports a failure or rejection not covered by a more specific class.

EvalServiceUnavailableError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: EvalServiceError

A remote service failed temporarily (HTTP 500, 502, 503 and 504). Retrying the same request can succeed.

EvalSuite

Bases: BaseModel

Evaluation suite defining a data partition and its evaluators.

Attributes:

Name	Type	Description
data	str | BaseDataset | list[LLMTestCase] | list[ConversationalTestCase]	Input data as a string path, BaseDataset, or list of evaluation inputs (single-turn LLMTestCase or multi-turn ConversationalTestCase).
evaluators	list[BaseEvaluator]	List of evaluators to apply to this suite's data.
name	str | None	Optional name for the suite. If not provided, auto-generated as suite_0, suite_1, etc.

from_yaml(source, extra_dataset_kwargs=None) staticmethod

Build an EvalSuite from a single YAML suite document.

See gllm_evals/utils/yaml_suite_loader.py for the full YAML format, evaluator/metric resolution rules, ${VAR} substitution, and dataset resolution behavior.

Parameters:

Name	Type	Description	Default
source	str | Path	Path to the YAML suite document.	required
extra_dataset_kwargs	dict[str, Any] | None	Extra kwargs merged into dataset resolution, for values that can't be expressed in YAML. Defaults to None.	None

Returns:

Name	Type	Description
EvalSuite	EvalSuite	The constructed suite.

Raises:

Type	Description
ValueError	If the document is malformed, or an evaluator/metric/dataset reference is invalid.

from_yaml_dir(directory, pattern='*.y*ml', extra_dataset_kwargs=None) staticmethod

Build one EvalSuite per matching YAML file in a directory.

Parameters:

Name	Type	Description	Default
directory	str | Path	Directory containing one YAML file per suite.	required
pattern	str	Glob pattern for matching files. Defaults to ".yml".	'*.y*ml'
extra_dataset_kwargs	dict[str, Any] | None	Forwarded identically to every file. Defaults to None.	None

Returns:

Type	Description
list[EvalSuite]	list[EvalSuite]: One suite per matching file, sorted by path relative to directory.

Raises:

Type	Description
ValueError	If no files match, or any single file fails to load.

EvalTimeoutError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, TimeoutError

An operation exceeds an execution, network, polling or retry deadline. Distinct from cancellation.

EvalUnsupportedOperationError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, NotImplementedError

A well-formed operation or input capability is not supported by the selected component.

EvalUnsupportedSyncOperationError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: EvalUnsupportedOperationError

The component supports only asynchronous evaluation, but a synchronous path was called.

EvaluatorOutput

Bases: _FailureRecord

Structured output for a single evaluator's result.

The three aggregate fields are required. Per-metric results (e.g. completeness, groundedness) are stored as extra fields via model_config extra="allow".

Attributes:

Name	Type	Description
aggregate_success	bool	Whether the evaluation passed overall.
aggregate_score	float	Aggregate score across all metrics.
aggregate_explanation	str	Human-readable explanation of the result.
evaluator_token_usage	dict[str, TokenUsage] | None	Model-keyed token usage keyed by model_id, summed across this evaluator's metrics. None only when none of its metrics reported usage.
error	ErrorInfo | None	The failure's safe record, set only when the whole row errored. Left out of dumps while None.

LLMTestCase

Bases: BaseModel

Evaluation row model for LLM-based tests.

Use this model for QA, RAG, and agent-style evaluations. All fields are optional so one row can represent different evaluation cases.

Extra fields beyond those listed below are allowed and preserved throughout evaluation. They can be accessed by attribute or via model_dump()::

row = LLMTestCase(input="q", custom_score=0.9)
row.custom_score          # 0.9
row.model_dump()["custom_score"]  # 0.9

Attributes:

Name	Type	Description
input	str | None	Input query or user prompt. Defaults to None.
actual_output	str | None	Model-generated response. Defaults to None.
expected_output	str | None	Reference response. Defaults to None.
retrieved_context	str | list[str] | None	Retrieved context used to answer the query. Defaults to None.
expected_context	str | list[str] | None	Reference context expected to be retrieved. Defaults to None.
agent_trajectory	list[dict[str, Any]] | None	Agent execution trace. Defaults to None.
expected_agent_trajectory	list[dict[str, Any]] | None	Reference agent execution trace. Defaults to None.
tools_called	list[ToolCall] | None	Tools invoked by the agent. Defaults to None.
expected_tools	list[ToolCall] | None	Tools expected to be invoked. Defaults to None.
is_refusal	bool | None	Whether the sample is a refusal case. Defaults to None.
attachments	dict[str, AttachmentRef]	Row-level attachment references used by [ATTACHMENT:<id>] placeholders. Defaults to an empty dict.

from_dict(row) classmethod

Build an LLMTestCase from a raw mapping.

The counterpart of ConversationalTestCase.from_dict. Both exist so that DictDataset.load() can construct whichever type it was configured with through one uniform call.

Parameters:

Name	Type	Description	Default
row	dict[str, Any]	Raw row data.	required

Returns:

Name	Type	Description
LLMTestCase	LLMTestCase	The constructed test case.

Raises:

Type	Description
ValidationError	If the row cannot be validated as an LLMTestCase.

RetrievalData

Bases: BaseModel

Retrieval data.

A data for retrieval evaluation such as ClassicalRetrievalEvaluator, etc.

Attributes:

Name	Type	Description
retrieved_chunks	dict[str, float]	The retrieved chunks and their scores.
ground_truth_chunk_ids	list[str]	The ground truth chunk IDs.

SuiteExperimentResult

Bases: BaseModel

Top-level result from evaluating multiple suites.

Attributes:

Name	Type	Description
run_id	str	Shared run ID across all suites.
dataset_name	str	Base dataset name (before suite namespacing).
timestamp	str	ISO 8601 timestamp of the evaluation.
num_samples	int	Total number of samples across all suites.
run_aggregators_result	dict[str, Any]	Aggregated result from all suites pooled.
experiment_uris	ExperimentUris	URIs for accessing experiment data.
suites	dict[str, SuiteResult]	Dict mapping suite names to their individual results.
total_token_usage	dict[str, TokenUsage] | None	Model-keyed token usage for the run.

SuiteResult

Bases: BaseModel

Result from evaluating a single suite.

Attributes:

Name	Type	Description
dataset_name	str	Namespaced dataset name for this suite.
num_samples	int	Number of samples in this suite.
results	list[list[EvaluatorResult]]	Row-grouped evaluation results from all evaluators.
run_aggregators_result	dict[str, Any]	Aggregated result from run aggregators for this suite.
suite_token_usage	dict[str, TokenUsage] | None	Model-keyed token usage for this suite.

ToolCall

Bases: BaseModel

Structured tool call data for agent evaluation rows.

Attributes:

Name	Type	Description
name	str	Tool name.
description	str | None	Tool description or metadata.
reasoning	str | None	Model reasoning for selecting the tool.
output	Any | None	Tool output.
input_parameters	dict[str, Any] | None	Tool input parameters.

from_dicts(tool_calls) classmethod

Convert canonical tool-call dictionaries to ToolCall objects.

Parameters:

Name	Type	Description	Default
tool_calls	list[dict[str, Any]] | None	Tool-call dictionaries using the public ToolCall field names.	required

Returns:

Type	Description
list[ToolCall] | None	list[ToolCall] | None: Parsed ToolCall objects, or None when the input is None or empty.

Turn

Bases: BaseModel

One ordered item in the conversation transcript.

role is structural ("user" or "assistant"), not speaker identity; attach a label via extra="allow" (e.g. Turn(..., name="AI_A")) if one is needed.

Attributes:

Name	Type	Description
role	TurnRole	"user" or "assistant" only.
content	str	Textual content of the turn. "" is valid.
retrieved_context	list[str] | None	Per-turn retrieval context; a bare string is accepted and wrapped in a single-element list. Defaults to None.
tools_called	list[ToolCall] | None	Per-turn tool invocations. Defaults to None.

build_metric_result(parsed_response)

Build a MetricScore from a raw dict.

Parameters:

Name	Type	Description	Default
parsed_response	dict	The response to build from.	required

Returns:

Name	Type	Description
MetricScore	MetricScore	The validated response as a MetricScore model.

Raises:

Type	Description
EvalResultContractError	If the response is not a valid MetricScore. Neither the submitted values nor their field locations are included.

get_default_retry_config()

Get the default retry configuration for the LM invoker.

Retries rate limits, Google Sheets API errors, connection failures and server errors. LookupError is not retried: it signals a KeyError or IndexError defect, not a transient failure. Timeouts are not retried because core's default non_retryable_exceptions already excludes TimeoutError.

Returns:

Name	Type	Description
RetryConfig	RetryConfig	The default retry configuration for the LM invoker.

load_simple_agent_dataset(current_dir=None, filename='simple_agent_data.csv')

Load the simple agent dataset from the local CSV file.

The dataset contains agent interaction data with trajectories, questions, and responses, suitable for both agent trajectory evaluation and generation quality evaluation.

Reads the CSV directly so that JSON-string columns (e.g. agent_trajectory) are parsed by parse_agent_row before LLMTestCase construction.

Parameters:

Name	Type	Description	Default
current_dir	Path | str | None	The directory containing the CSV file. If None, defaults to the examples/sample_data directory.	None
filename	str	The name of the CSV file. Defaults to "simple_agent_data.csv".	'simple_agent_data.csv'

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded simple agent dataset as list[LLMTestCase].

Raises:

Type	Description
EvalResourceNotFoundError	If the CSV file doesn't exist.
EvalInputError	If the CSV file is empty or malformed, or a row is incompatible with LLMTestCase.
BaseError	A normalized error for any other read failure.

load_simple_qa_dataset(current_dir=None, filename='simple_qa_data.csv')

Load the simple QA dataset from the local CSV file.

The dataset contains question-answer pairs with generated responses and contexts, suitable for RAG evaluation and testing.

Parameters:

Name	Type	Description	Default
current_dir	Path | str | None	The directory containing the CSV file. If None, defaults to the examples/sample_data directory.	None
filename	str	The name of the CSV file. Defaults to "simple_qa_data.csv".	'simple_qa_data.csv'

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded simple QA dataset as LLMTestCase instances with fields: 'input', 'actual_output', 'expected_output', 'retrieved_context'.

Raises:

Type	Description
EvalResourceNotFoundError	If the CSV file doesn't exist.
EvalInputError	If the CSV file is empty or malformed.
BaseError	A normalized error for any other read failure.

load_simple_summarization_dataset(current_dir=None, filename='simple_summarization_data.csv')

Load the simple summarization dataset from the local CSV file.

Parameters:

Name	Type	Description	Default
current_dir	Path | str | None	The directory containing the CSV file. If None, defaults to the examples/sample_data directory.	None
filename	str	The name of the CSV file. Defaults to "simple_summarization_data.csv".	'simple_summarization_data.csv'

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded simple summarization dataset containing LLMTestCase dictionaries with keys: 'input' and 'summary'.

Raises:

Type	Description
EvalResourceNotFoundError	If the CSV file doesn't exist.
EvalInputError	If the CSV file is empty or malformed.
BaseError	A normalized error for any other read failure.

parse_list_strings_in_dict(data)

Parse string representations of lists in dictionary values.

Recursively processes all values in a dictionary, converting string representations of lists (e.g., '["abc", "def"]') into actual Python lists. Also handles CSV-escaped JSON strings (e.g., '[""abc"", ""def""]').

Parameters:

Name	Type	Description	Default
data	dict[str, Any]	Dictionary to process	required

Returns:

Type	Description
dict[str, Any]	dict[str, Any]: Dictionary with parsed list strings converted to actual lists

Examples:

Basic string to list conversion: >>> parse_list_strings_in_dict({"key1": "abc", "key2": '["item1", "item2"]'})

Nested dictionary processing: >>> parse_list_strings_in_dict({"nested": {"list": '["a", "b"]'}}) {"nested": {"list": ["a", "b"]}}

CSV-escaped JSON handling: >>> parse_list_strings_in_dict({"csv_escaped": '[""item1"", ""item2""]'})

Mixed data types: >>> parse_list_strings_in_dict({"str": "hello", "list": '["a", "b"]', "num": 42, "bool": True})

List of dictionaries with nested parsing: >>> parse_list_strings_in_dict({"items": [{"name": '["item1", "item2"]'}, {"id": 123}]}) {"items": [{"name": ["item1", "item2"]}, {"id": 123}]}

Complex nested structure: >>> data = { ... "config": {"features": '["auth", "logging"]'}, ... "users": [{"roles": '["admin", "user"]'}, {"status": "active"}], ... "settings": {"enabled": True, "timeout": 30} ... } >>> parse_list_strings_in_dict(data) { "config": {"features": ["auth", "logging"]}, "users": [{"roles": ["admin", "user"]}, {"status": "active"}], "settings": {"enabled": True, "timeout": 30} }

Single-quoted strings (AST parsing): >>> parse_list_strings_in_dict({"single_quotes": "['a', 'b', 'c']"})

Non-list strings (unchanged): >>> parse_list_strings_in_dict({"not_list": "just a string", "empty": ""})