Types

Types for the evaluator.

This package contains the types for the evaluator. Most of them are defined here in the package __init__ verbatim; three concerns live in submodules so that types/multiturn.py can import what it needs from a sibling rather than from this module, which would be an import cycle:

• types/tool.py -- ToolCall
• types/attachment.py -- AttachmentRef
• types/multiturn.py -- Turn, ConversationalTestCase, TurnRole

All three are re-exported here, so from gllm_evals.types import ToolCall and friends keep working unchanged.

AttachmentConfig

Bases: BaseModel, ABC

Base configuration for loading attachments.

Attributes:

Name	Type	Description
type	str	The type of attachment source.

validate_config() abstractmethod

Validate configuration (can be overridden if needed).

AttachmentRef

Bases: BaseModel

Row-level attachment reference for multimodal evaluation inputs.

Attributes:

Name	Type	Description
uri	str	Attachment URI or portable reference location.
mime_type	str | None	MIME type hint for the attachment. Defaults to None.
metadata	dict[str, Any]	Caller-provided metadata for downstream adapters. Defaults to an empty dict.

ConversationalTestCase

Bases: BaseModel

Canonical top-level input object for multi-turn evaluation.

Attributes:

Name	Type	Description
turns	list[Turn]	Ordered conversation transcript. Must contain at least one turn.
chatbot_role	str | None	The assistant's intended role / persona. Defaults to None.
expected_output	str | None	Primary top-level reference field, mapped to DeepEval's expected_outcome by the conversion layer. Defaults to None.
scenario	str | None	Conversation scenario description. Defaults to None.
attachments	dict[str, AttachmentRef]	Row-level attachment references for [ATTACHMENT:<id>] placeholders. Defaults to an empty dict.

from_dict(row) classmethod

Build a ConversationalTestCase from a raw mapping.

Parameters:

Name	Type	Description	Default
row	dict[str, Any]	Raw row data.	required

Returns:

Name	Type	Description
ConversationalTestCase	ConversationalTestCase	The constructed test case.

Raises:

Type	Description
ValidationError	If the row cannot be validated as a ConversationalTestCase.

from_messages(messages, *, chatbot_role=None, **conv_fields) classmethod

Build a ConversationalTestCase from a gllm_inference Message transcript.

Mapping applied to each message:

• system -> chatbot_role
• user / assistant -> one Turn each (not merged)
• tool-call request -> Turn.tools_called
• tool result -> folded into the matching ToolCall.output, not its own turn
• Thinking content part -> dropped

Parameters:

Name	Type	Description	Default
messages	list[Message]	Ordered inference transcript.	required
chatbot_role	str | None	Extra system context, newline-joined ahead of any system messages. Defaults to None.	None
**conv_fields	Any	Forwarded verbatim to the constructor -- e.g. expected_output, scenario, attachments.	{}

Returns:

Name	Type	Description
ConversationalTestCase	ConversationalTestCase	The constructed test case.

Raises:

Type	Description
EvalInputError	If messages is not a sequence of Message objects, if turns is passed in conv_fields, if the transcript produces no user/assistant turns, or if the resulting test case fails validation.
EvalUnsupportedOperationError	If a message carries an Attachment content part.

DecisionDetails

Bases: BaseModel

What a decisions model answered, for programmatic use.

Attributes:

Name	Type	Description
model_id	str	The decisions model that answered.
choice	str	The chosen label.
confidence	float | None	The model's confidence in the choice, if reported.
probabilities	dict[str, float]	The probability of every offered label.

EvaluatorOutput

Bases: _FailureRecord

Structured output for a single evaluator's result.

The three aggregate fields are required. Per-metric results (e.g. completeness, groundedness) are stored as extra fields via model_config extra="allow".

Attributes:

Name	Type	Description
aggregate_success	bool	Whether the evaluation passed overall.
aggregate_score	float	Aggregate score across all metrics.
aggregate_explanation	str	Human-readable explanation of the result.
evaluator_token_usage	dict[str, TokenUsage] | None	Model-keyed token usage keyed by model_id, summed across this evaluator's metrics. None only when none of its metrics reported usage.
error	ErrorInfo | None	The failure's safe record, set only when the whole row errored. Left out of dumps while None.

ExperimentResult

Bases: BaseModel

Structured result from the evaluate function.

This provides a unified return type that includes both evaluation results and experiment tracker URLs/paths.

Attributes:

Name	Type	Description
run_id	str	The run ID for this evaluation.
results	list[list[EvaluatorResult]]	The evaluation results from all evaluators.
experiment_uris	ExperimentUris	URIs and paths for accessing experiment data.
dataset_name	str | None	The name of the dataset that was evaluated.
timestamp	str	The timestamp of the evaluation in ISO 8601 format.
num_samples	int	The number of samples in the dataset.
metadata	dict[str, Any]	The metadata of the evaluation.
run_aggregators_result	dict[str, Any] | None	Aggregated result from run_aggregators. Defaults to None.

ExperimentUris

Bases: BaseModel

Experiment URIs and paths for different experiment trackers.

This model provides a unified interface for experiment tracker URIs/paths. Different trackers will populate different fields based on their capabilities.

Attributes:

Name	Type	Description
run_uri	str | None	URI to view the experiment results. For Simple: local file path to experiment results. Defaults to None.
leaderboard_uri	str | None	URI to view the leaderboard. For Simple: local file path to leaderboard CSV. Defaults to None.

GoogleDriveAttachmentConfig

Bases: AttachmentConfig

Configuration for loading attachments from Google Drive.

Attributes:

Name	Type	Description
type	Literal[GDRIVE]	Always "gdrive" for this implementation.
client_email	str | None	Google service account client email.
private_key	str | None	Google service account private key.
folder_id	str	Google Drive folder ID.
service_account_file	str | None	Path to service account JSON file (alternative to client_email/private_key).

validate_config()

Google Drive-specific validation.

Raises:

Type	Description
EvalConfigurationError	If neither a service account file nor a client email and private key are provided.

LLMTestCase

Bases: BaseModel

Evaluation row model for LLM-based tests.

Use this model for QA, RAG, and agent-style evaluations. All fields are optional so one row can represent different evaluation cases.

Extra fields beyond those listed below are allowed and preserved throughout evaluation. They can be accessed by attribute or via model_dump()::

row = LLMTestCase(input="q", custom_score=0.9)
row.custom_score          # 0.9
row.model_dump()["custom_score"]  # 0.9

Attributes:

Name	Type	Description
input	str | None	Input query or user prompt. Defaults to None.
actual_output	str | None	Model-generated response. Defaults to None.
expected_output	str | None	Reference response. Defaults to None.
retrieved_context	str | list[str] | None	Retrieved context used to answer the query. Defaults to None.
expected_context	str | list[str] | None	Reference context expected to be retrieved. Defaults to None.
agent_trajectory	list[dict[str, Any]] | None	Agent execution trace. Defaults to None.
expected_agent_trajectory	list[dict[str, Any]] | None	Reference agent execution trace. Defaults to None.
tools_called	list[ToolCall] | None	Tools invoked by the agent. Defaults to None.
expected_tools	list[ToolCall] | None	Tools expected to be invoked. Defaults to None.
is_refusal	bool | None	Whether the sample is a refusal case. Defaults to None.
attachments	dict[str, AttachmentRef]	Row-level attachment references used by [ATTACHMENT:<id>] placeholders. Defaults to an empty dict.

from_dict(row) classmethod

Build an LLMTestCase from a raw mapping.

The counterpart of ConversationalTestCase.from_dict. Both exist so that DictDataset.load() can construct whichever type it was configured with through one uniform call.

Parameters:

Name	Type	Description	Default
row	dict[str, Any]	Raw row data.	required

Returns:

Name	Type	Description
LLMTestCase	LLMTestCase	The constructed test case.

Raises:

Type	Description
ValidationError	If the row cannot be validated as an LLMTestCase.

LocalAttachmentConfig

Bases: AttachmentConfig

Configuration for loading attachments from local directory.

Attributes:

Name	Type	Description
type	Literal[LOCAL]	Always "local" for this implementation.
local_directory	str	Local directory path.

validate_config()

Local-specific validation.

Raises:

Type	Description
EvalResourceNotFoundError	If the local directory does not exist. The path is not included.

MetricResult

Bases: _FailureRecord, MetricScore

GEval-specific metric result with polarity-aware success and diagnostic fields.

Extends MetricScore with fields required for polarity-aware binary scoring and threshold-based pass/fail logic. All GEval-specific fields below are required and non-optional for type-safe aggregation.

Attributes:

Name	Type	Description
score	MetricValue | None	The evaluation score. Can be continuous (float), discrete (int), or categorical (str). None when the metric errored: an errored metric has no score, and scoring it 0.0 would drag the evaluator aggregate down as though the model had failed the check.
explanation	str | None	A detailed explanation of the evaluation result. Inherited from MetricScore; nullable.
error	ErrorInfo | None	The failure's safe record, set only when this metric errored. Left out of dumps while None.
rubric_score	int | float | None	Native rubric value as-is (diagnostic only, not for thresholding). None when the metric errored, for the same reason score is.
success	bool	Pass/fail determination using polarity and threshold.
threshold	float	The threshold used to compute success.
strict_mode	bool	If True, binarizes score to 1.0 or 0.0; else uses raw float score.
higher_is_better	bool	Polarity flag; determines success logic direction.
token_usage	dict[str, TokenUsage] | None	Model-keyed token usage keyed by model_id. Populated on every result; None only when the metric made no billed LLM call or the provider did not report usage for it (e.g. some batch APIs).
decision	DecisionDetails | None	Choice, confidence and probabilities of a decisions-model judge. Set only when a single decisions-model judge produced the score; None otherwise.

create_default_failure(threshold=0.5, strict_mode=False, higher_is_better=True) classmethod

Create a default failure MetricResult.

Used when a metric fails or returns unexpected data.

Parameters:

Name	Type	Description	Default
threshold	float	The threshold to use. Defaults to 0.5.	0.5
strict_mode	bool	Whether strict mode is enabled. Defaults to False.	False
higher_is_better	bool	Whether higher scores are better. Defaults to True.	True

Returns:

Name	Type	Description
MetricResult	MetricResult	A failure result with score 0.0 and success=False.

MetricScore

Bases: BaseModel

Metric Output Pydantic Model.

A structured output for metric results with score and explanation.

Attributes:

Name	Type	Description
score	MetricValue	The evaluation score. Can be continuous (float), discrete (int), or categorical (str).
explanation	str | None	A detailed explanation of the evaluation result.

RetrievalData

Bases: BaseModel

Retrieval data.

A data for retrieval evaluation such as ClassicalRetrievalEvaluator, etc.

Attributes:

Name	Type	Description
retrieved_chunks	dict[str, float]	The retrieved chunks and their scores.
ground_truth_chunk_ids	list[str]	The ground truth chunk IDs.

RunSummaryData

Bases: BaseModel

Summary data for a single run accumulated from batch processing.

This contains all accumulated data and the computed cumulative summary for a run. Stored and managed by experiment trackers that support summary evaluators.

Attributes:

Name	Type	Description
results	list[list[EvaluatorResult]]	Row-grouped evaluation results accumulated from batches. Defaults to empty list.
data	list[LLMTestCase]	All input data accumulated from batches. Defaults to empty list.
summary	dict[str, Any]	Cumulative summary computed by summary evaluators. Defaults to empty dict.

S3AttachmentConfig

Bases: AttachmentConfig

Configuration for loading attachments from S3.

Attributes:

Name	Type	Description
type	Literal[S3]	Always "s3" for this implementation.
s3_bucket	str	S3 bucket name.
s3_prefix	str | None	S3 prefix (optional).
aws_access_key_id	str	AWS access key ID.
aws_secret_access_key	str	AWS secret access key.
aws_region	str	AWS region.

validate_config()

S3-specific validation (if needed beyond Pydantic's).

SuiteExperimentResult

Bases: BaseModel

Top-level result from evaluating multiple suites.

Attributes:

Name	Type	Description
run_id	str	Shared run ID across all suites.
dataset_name	str	Base dataset name (before suite namespacing).
timestamp	str	ISO 8601 timestamp of the evaluation.
num_samples	int	Total number of samples across all suites.
run_aggregators_result	dict[str, Any]	Aggregated result from all suites pooled.
experiment_uris	ExperimentUris	URIs for accessing experiment data.
suites	dict[str, SuiteResult]	Dict mapping suite names to their individual results.
total_token_usage	dict[str, TokenUsage] | None	Model-keyed token usage for the run.

SuiteResult

Bases: BaseModel

Result from evaluating a single suite.

Attributes:

Name	Type	Description
dataset_name	str	Namespaced dataset name for this suite.
num_samples	int	Number of samples in this suite.
results	list[list[EvaluatorResult]]	Row-grouped evaluation results from all evaluators.
run_aggregators_result	dict[str, Any]	Aggregated result from run aggregators for this suite.
suite_token_usage	dict[str, TokenUsage] | None	Model-keyed token usage for this suite.

ToolCall

Bases: BaseModel

Structured tool call data for agent evaluation rows.

Attributes:

Name	Type	Description
name	str	Tool name.
description	str | None	Tool description or metadata.
reasoning	str | None	Model reasoning for selecting the tool.
output	Any | None	Tool output.
input_parameters	dict[str, Any] | None	Tool input parameters.

from_dicts(tool_calls) classmethod

Convert canonical tool-call dictionaries to ToolCall objects.

Parameters:

Name	Type	Description	Default
tool_calls	list[dict[str, Any]] | None	Tool-call dictionaries using the public ToolCall field names.	required

Returns:

Type	Description
list[ToolCall] | None	list[ToolCall] | None: Parsed ToolCall objects, or None when the input is None or empty.

Turn

Bases: BaseModel

One ordered item in the conversation transcript.

role is structural ("user" or "assistant"), not speaker identity; attach a label via extra="allow" (e.g. Turn(..., name="AI_A")) if one is needed.

Attributes:

Name	Type	Description
role	TurnRole	"user" or "assistant" only.
content	str	Textual content of the turn. "" is valid.
retrieved_context	list[str] | None	Per-turn retrieval context; a bare string is accepted and wrapped in a single-element list. Defaults to None.
tools_called	list[ToolCall] | None	Per-turn tool invocations. Defaults to None.

build_metric_result(parsed_response)

Build a MetricScore from a raw dict.

Parameters:

Name	Type	Description	Default
parsed_response	dict	The response to build from.	required

Returns:

Name	Type	Description
MetricScore	MetricScore	The validated response as a MetricScore model.

Raises:

Type	Description
EvalResultContractError	If the response is not a valid MetricScore. Neither the submitted values nor their field locations are included.

create_attachment_config(config_dict)

Factory function to create the appropriate AttachmentConfig.

Parameters:

Name	Type	Description	Default
config_dict	dict[str, Any]	Configuration dictionary with 'type' field.	required

Returns:

Type	Description
AttachmentConfig	The appropriate AttachmentConfig subclass instance.

Raises:

Type	Description
ValidationError	If a configuration field is invalid.
EvalConfigurationError	If the type is unknown or the configuration is incomplete.
EvalResourceNotFoundError	If a local attachment directory does not exist.

Example

config = create_attachment_config({ ... "type": "s3", ... "s3_bucket": "my-bucket", ... "aws_access_key_id": "...", ... "aws_secret_access_key": "...", ... "aws_region": "us-east-1" ... }) isinstance(config, S3AttachmentConfig) True