Exceptions

Normalized errors, context, resolution and failure reporting for gllm-evals.

Catch gllm_core.exceptions.BaseError for general handling and a specific class below for deliberate recovery.

error_reporter = ErrorReporter() module-attribute

The shared reporter every boundary uses.

is_failed_result = ErrorReporter.is_failed_result module-attribute

Tells whether a metric, evaluator or judge result records a failure. See ErrorReporter.is_failed_result.

BaseEvalError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseError

Root of the errors owned by gllm-evals.

Catch gllm_core.exceptions.BaseError to handle every normalized gllm failure, including preserved gllm-inference errors; catch BaseEvalError only for failures owned by gllm-evals.

Every message says what happened and then what to do next. A class's default_message holds both, and a raise site that passes its own message writes both as well.

Attributes:

Name	Type	Description
context	ErrorContext | None	Where the failure happened, if known.
children	tuple[BaseError, ...]	Normalized failures this error summarizes, e.g. every failed judge.

Initializes the error.

Parameters:

Name	Type	Description	Default
message	str | None	The complete message: what happened, then what to do next. It must be as safe to show as the default: never include row content, paths, URLs, credentials or upstream error text. Defaults to None, which uses default_message.	None
context	ErrorContext | None	Where the failure happened. Defaults to None.	None
children	Iterable[BaseError]	Normalized failures this error summarizes. Defaults to an empty tuple.	()
debug_info	DebugInfo | None	Sanitized internal detail. Defaults to None.	None
recovery	RecoveryInfo | None	Recovery replacing the class default for this occurrence, for example an operation that is safe to retry. Defaults to None.	None

origin property

Who owns the original failure.

Returns:

Name	Type	Description
ErrorOrigin	ErrorOrigin	The context's origin, or "gllm-evals" when no context was given.

from_error(raw_error, *, context, status_code=None, recovery=None, message=None) classmethod

Builds this error for a raw failure without copying the raw error's message.

Use it at a raise site that already knows the class. It does not classify: translating an unknown failure is EvalErrorTranslator's job.

Parameters:

Name	Type	Description	Default
raw_error	Exception | None	The raw failure. Only its type name is kept. None when the SDK detected the failure itself, for example from an HTTP status, and no exception was raised.	required
context	ErrorContext	Where the failure happened.	required
status_code	int | None	Numeric status reported by the dependency. Defaults to None.	None
recovery	RecoveryInfo | None	Recovery replacing the class default. Defaults to None.	None
message	str | None	The complete message replacing the class default: what happened, then what to do next. It must not contain row content, paths, URLs or upstream error text. Defaults to None.	None

Returns:

Name	Type	Description
Self	Self	The normalized error, with raw_error as its __cause__ when one is given.

BatchNotSupportedError

Bases: Exception

Internal control signal: a metric has no batch mode, so the caller falls back to row-by-row evaluation.

Not a BaseError: it is never surfaced to users and is not part of the error catalogue.

ErrorContext(component, operation, stage, origin='gllm-evals', integration=None, row_index=None, attempt_index=None, request_id=None) dataclass

Where a failure happened, used to classify and record it.

Every field is a static name or a number. Never put user labels, URLs, file paths, credentials or row and model payloads in a context: it is copied into failure results and diagnostic records.

Attributes:

Name	Type	Description
component	ErrorComponent	Static name of the component that observed the failure, e.g. "dataset".
operation	str	Static name of the operation, e.g. "dataset.load".
stage	ErrorStage	Evaluation stage. Decides how parse and validation failures are classified.
origin	ErrorOrigin	Who owns the original failure, which can differ from the normalized class owner.
integration	ErrorIntegration | None	External integration involved, if any. Defaults to None.
row_index	int | None	Index of the affected row, if known. Defaults to None.
attempt_index	int | None	Index of the affected attempt, e.g. a judge in a fallback chain. Defaults to None.
request_id	str | None	Generated identifier of the affected request, if any. Defaults to None.

__post_init__()

Validates that names are present and indexes are non-negative integers.

Raises:

Type	Description
ValueError	If component or operation is empty, or an index is not a non-negative integer.

ErrorInfo

Bases: BaseModel

Safe, structured record of one failure, used for failed results, exports and diagnostics.

Attributes:

Name	Type	Description
code	str	Stable error code of the failure type.
message	str	Reviewed message that is safe to show to a user.
reference_id	str	Identifier of this occurrence.
error_type	str	Name of the normalized exception class.
origin	ErrorOrigin	Who owns the original failure.
recovery	dict[str, Any]	Copy of the error's RecoveryInfo. Typed as a plain dict because Pydantic rejects typing.TypedDict on Python 3.11.
component	str	Component that observed the failure.
operation	str	Operation that failed.
children	list[ErrorInfo]	Records of the failures this one summarizes. Defaults to an empty list.

ErrorReporter(logger_name=ERRORS_LOGGER_NAME)

Builds safe records of normalized errors and logs each handled occurrence once per evaluation.

One shared instance (error_reporter) is used by every boundary. The dedup scope lives in a ContextVar, so it is per evaluation or task, not per instance.

Attributes:

Name	Type	Description
SERVICE_NAME	ClassVar[str]	Service name written to every diagnostic record.
OMITTED_DETAILS	ClassVar[str]	Placeholder written where an upstream message is left out.
RESERVED_ERROR_KEYS	ClassVar[frozenset[str]]	Keys a failed metric or evaluator result carries.

Initializes the reporter.

The logger is resolved on first use, not here: LoggerManager.get_logger attaches the handlers that exist at that moment, so resolving it at import would miss handlers an application adds before its first evaluation.

Parameters:

Name	Type	Description	Default
logger_name	str	Name of the logger failures are written to. Defaults to "gllm_evals.errors".	ERRORS_LOGGER_NAME

build_diagnostic_record(error, *, context)

Builds the allowlisted diagnostic record of a handled failure.

The record identifies the failure and where it happened. It never contains the exception message, upstream response bodies, prompts, paths or URLs: original_message is always "[details omitted]".

Parameters:

Name	Type	Description	Default
error	BaseError	The normalized error.	required
context	ErrorContext	Where the failure was handled; used when the error carries no context of its own.	required

Returns:

Type	Description
dict[str, Any]	dict[str, Any]: The structured diagnostic record.

code_of(error) staticmethod

Returns the stable code of a normalized error.

Parameters:

Name	Type	Description	Default
error	BaseError	The normalized error.	required

Returns:

Name	Type	Description
str	str	The error's code, or the execution code when it has none.

ensure_reference_id(error) staticmethod

Returns the error's reference ID, setting an opaque one first if none is set.

Never overwrites an ID that is already set, for example by a consuming service.

Parameters:

Name	Type	Description	Default
error	BaseError	The error being recorded or converted into a failed result.	required

Returns:

Name	Type	Description
str	str	The occurrence's reference ID.

is_failed_result(result) staticmethod

Tells whether a metric, evaluator or judge result records a failure.

A result failed when it carries the error key, whatever its value. Valid low scores and success=False results never carry it.

Parameters:

Name	Type	Description	Default
result	object	A MetricResult, EvaluatorOutput (failure keys are undeclared extras) or plain mapping.	required

Returns:

Name	Type	Description
bool	bool	True if the result records a failure.

message_of(error)

Returns a message for an error that is safe to show to a user.

Uses the class-level default_message of a gllm-evals or gllm-inference error, never the instance message or str(error). An error class owned by any other library gets the generic execution message.

Parameters:

Name	Type	Description	Default
error	BaseError	The normalized error.	required

Returns:

Name	Type	Description
str	str	The reviewed message.

record(error, *, context)

Records a handled failure once.

Within the active scope an occurrence is identified by its reference ID. Without a scope, for example when a metric is called directly, it is identified by the error object, so a failure handled at two places is still recorded once.

Sets a missing reference ID first. Emits one WARNING record with a fixed message and the allowlisted diagnostic in extra["gllm_error"], plus a DEBUG record with safe traceback frames when available. Never raises: a reporting failure must not change the primary exception or result.

Parameters:

Name	Type	Description	Default
error	BaseError	The normalized error that was handled.	required
context	ErrorContext	Where the failure was handled.	required

recovery_of(error) staticmethod

Returns a copy of an error's recovery guidance.

Parameters:

Name	Type	Description	Default
error	BaseError	The normalized error.	required

Returns:

Type	Description
dict[str, Any]	dict[str, Any]: The error's RecoveryInfo, or {"retryable": False} when it has none.

scope()

Deduplicates handled-failure records for the duration of one evaluation.

Reuses the active scope when called inside another one, so nested public entry points share one scope. Work submitted to loop.run_in_executor or a ThreadPoolExecutor must run through contextvars.copy_context().run to see the scope.

Yields:

Name	Type	Description
None	None	Control returns to the caller while the scope is active.

to_error_info(error, *, context, _ancestors=frozenset())

Builds the safe record of a normalized error and of every failure it summarizes.

Sets missing reference IDs first, on the error objects themselves, so later handlers reuse them. Children that would form a cycle are skipped.

Parameters:

Name	Type	Description	Default
error	BaseError	The normalized error.	required
context	ErrorContext	Where the failure was handled; used when the error carries no context of its own.	required
_ancestors	frozenset[int]	Identities of the errors already on the path from the root. Callers leave it at its default.	frozenset()

Returns:

Name	Type	Description
ErrorInfo	ErrorInfo	The safe record, keeping the error's code and reference ID.

translate_errors(context)

Makes an async public entry point raise only normalized errors, inside one reporting scope.

The reporter opens the scope; classification is delegated to EvalErrorTranslator.

Parameters:

Name	Type	Description	Default
context	ErrorContext	Where the entry point's unhandled failures happen.	required

Returns:

Type	Description
Callable[[Callable[P, Awaitable[R]]], Callable[P, Awaitable[R]]]	Callable[[Callable[P, Awaitable[R]]], Callable[P, Awaitable[R]]]: The decorator.

EvalAccessError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, PermissionError

A local or remote operation is denied because authentication or authorization fails.

EvalAggregationError(message=None, *, failures=None, **kwargs)

Bases: BaseEvalError, ValueError

Aggregation cannot compute a result from the available outcomes, although its setup is valid.

Initializes the error.

Parameters:

Name	Type	Description	Default
message	str | None	What happened. Defaults to None, which uses the class default.	None
failures	dict[str, Any] | None	The failed MetricResult of each metric, by metric name, when every metric failed on a row. Defaults to None.	None
**kwargs	Any	The other BaseEvalError arguments.	{}

EvalConfigurationError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

Evaluation setup is invalid, incomplete, ambiguous or inconsistent before any content is processed.

EvalConnectionError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ConnectionError

Connection establishment or transport fails, such as DNS, TLS, refusal or reset.

EvalDependencyUnavailableError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

A required integration is missing or verifiably incompatible with the installed runtime.

EvalErrorTranslator(context)

Translates a raw exception caught at an evals boundary into a normalized gllm-evals error.

Bound to the ErrorContext of one boundary. Never raises and never renders the raw error with str() or repr(). An instance is a Callable[[Exception], Exception], so it can be used as a gllm_core.retry.RetryConfig.error_translators entry.

Attributes:

Name	Type	Description
context	ErrorContext	Where the failures this translator handles happen. Its stage and integration drive classification.

Initializes the translator for one boundary.

Parameters:

Name	Type	Description	Default
context	ErrorContext	Where the failures this translator handles happen.	required

__call__(error)

Returns the normalized error for a failure caught at the boundary.

Native control flow (asyncio.CancelledError, KeyboardInterrupt, ...) is not an Exception and must not be passed here.

Parameters:

Name	Type	Description	Default
error	Exception	The caught failure.	required

Returns:

Name	Type	Description
BaseError	BaseError	error itself when it is already normalized, otherwise a new gllm-evals error whose debug_info records only the raw error's type name and whose __cause__ is error.

from_status(status_code, *, raw_error=None, retry_after_seconds=None, message=None)

Classifies a failure from the HTTP-like status reported by a remote integration.

A 403 is an access denial, never a rate limit: only an explicit 429 is throttling.

Parameters:

Name	Type	Description	Default
status_code	int	The reported status code.	required
raw_error	Exception | None	The raw failure, or None when the status was read from a response directly. Defaults to None.	None
retry_after_seconds	int | None	Wait reported by the service for a 429. Defaults to None.	None
message	str | None	A reviewed, complete message replacing the class default: what happened, then what to do next. It must not contain URLs or response bodies. Defaults to None.	None

Returns:

Name	Type	Description
BaseEvalError	BaseEvalError	The normalized error.

reraise(error)

Raises the normalized error for a failure caught at the boundary.

Call it inside the except block that caught error. It is the one-line form of translating and raising, and keeps an already normalized error as it is.

Parameters:

Name	Type	Description	Default
error	Exception	The caught failure.	required

Raises:

Type	Description
BaseError	error itself when it is already normalized, otherwise the new gllm-evals error with error as its cause.

EvalExecutionError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, RuntimeError

An unexpected failure or invalid component state that no more specific class describes.

EvalIOError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, OSError

Local storage read, write, replace or cleanup fails for a reason other than absence or access denial.

EvalInputError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

Supplied evaluation content fails its input contract.

EvalJudgeResponseError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

A judge invocation completed but its answer cannot be used.

EvalJudgesExhaustedError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, RuntimeError

At least two judges in a fallback chain were attempted and none produced a usable result.

Pass every attempt's normalized failure as children and chain the last one as the cause. A single failed attempt is re-raised as its own type instead.

EvalRateLimitError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, RuntimeError

A non-inference integration explicitly rejects an operation because request-rate capacity is exhausted.

EvalResourceNotFoundError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, FileNotFoundError

A referenced file, directory, dataset, worksheet, object or session cannot be found.

EvalResultContractError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

A metric, evaluator or adapter violates the SDK result interface.

EvalSerializationError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, ValueError

Encoding or decoding an SDK-owned persisted or exported artifact fails.

EvalServiceError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, RuntimeError

A non-inference remote service reports a failure or rejection not covered by a more specific class.

EvalServiceUnavailableError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: EvalServiceError

A remote service failed temporarily (HTTP 500, 502, 503 and 504). Retrying the same request can succeed.

EvalTimeoutError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, TimeoutError

An operation exceeds an execution, network, polling or retry deadline. Distinct from cancellation.

EvalUnsupportedOperationError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: BaseEvalError, NotImplementedError

A well-formed operation or input capability is not supported by the selected component.

EvalUnsupportedSyncOperationError(message=None, *, context=None, children=(), debug_info=None, recovery=None)

Bases: EvalUnsupportedOperationError

The component supports only asynchronous evaluation, but a synchronous path was called.