Skip to content

Deepeval Wrapper

DeepEval-compatible LLM wrapper with automatic fallback chain support.

Wraps BaseLMInvoker as a DeepEvalBaseLLM + LMComponent, iterating through primary and ordered fallback invokers on failure with schema state safety.

DeepEvalLLMWrapper(lm_invoker, fallback_lms=None, skip_multimodal_check=False)

Bases: UsageRecordingMixin, DeepEvalBaseLLM, LMComponent

DeepEval-compatible LLM wrapper built on LMComponent.

Provides the DeepEvalBaseLLM interface with automatic fallback chain support. When the primary invoker fails with a gllm-inference error, an unusable answer or a timeout, the next invoker in fallback_lms is tried in order (see utils/judge_chain.py).

Attributes:

Name Type Description
lm_invoker BaseLMInvoker

Primary language model invoker.

fallback_lms list[BaseLMInvoker]

Ordered fallback invokers.

Initializes the DeepEvalLLMWrapper class.

Parameters:

Name Type Description Default
lm_invoker BaseLMInvoker

The LMInvoker instance to wrap.

required
fallback_lms list[BaseLMInvoker] | None

Ordered fallback invokers. Defaults to None.

None
skip_multimodal_check bool

When True, the caller asserts the judge supports image input, so capability resolution is skipped for models not in the blessed list (no probe). Defaults to False.

False

winning_invoker property

The invoker that handled the most recent successful request.

a_generate(prompt, schema=None) async

Runs the model to output LLM response asynchronously.

The response schema is applied to all invokers (primary and fallbacks) before _invoke_lm is called, so whichever invoker handles the request uses the correct structured-output format. All schemas are restored in the finally block.

The lock serialises concurrent calls that share the same primary invoker to prevent schema state races.

Parameters:

Name Type Description Default
prompt str

The complete prompt string built by DeepEval.

required
schema type[BaseModel] | None

Pydantic model for structured output. Defaults to None.

None

Returns:

Type Description
str | BaseModel

str | BaseModel: Text response or validated BaseModel instance.

Raises:

Type Description
EvalJudgeResponseError

If no judge gave a usable reply and the last attempt was unusable.

generate(*args, **kwargs)

Runs the model to output LLM response.

This method is not implemented because our LM invoker does not support synchronous generation.

A gllm_core.concurrency.syncify bridge was tried here (to unblock TurnFaithfulnessMetric, which calls this from inside its own async a_measure; TopicAdherence did too until it was fixed upstream in 4.0.7) and reverted after a real Vertex call proved it unsafe: the underlying google-genai async client's aiohttp session becomes bound to whatever event loop first used it. A wrapper that had already served an a_generate call on the main loop then failed the bridged sync call with RuntimeError: Task <...> got Future <...> attached to a different loop -- exactly the mixed sync/async pattern DeepEval exercises, so this is not a corner case. Faithfulness escapes via include_reason=False, which short-circuits before the sync call; deepeval_patches then rebuilds its reason from the claim verdicts.

Raises:

Type Description
EvalUnsupportedSyncOperationError

Always.

get_model_name(*args, **kwargs)

Return a placeholder model name for DeepEval compatibility.

Returns:

Name Type Description
str str

Placeholder name "dummy-model".

load_model(*args, **kwargs)

No-op — model loading is handled by the invoker.

resolve_multimodal_support() async

Resolve and cache whether the judge chain accepts image input.

Capability spans the whole chain (primary then fallbacks), because multimodal generation falls back through it: the chain is capable if any judge is. Per judge the order is blessed list -> skip_multimodal_check flag -> probe. The probe is authoritative (fail-closed). A capable verdict is cached per model_id and resolved single-flight across wrappers; a failed probe is cached only on this wrapper instance, so a later run with a fresh wrapper re-probes instead of being permanently blocked by a transient failure.

Returns:

Name Type Description
bool bool

True when at least one judge in the chain is image-capable.

supports_multimodal()

Return whether the judge chain accepts image input (synchronous view).

DeepEval calls this synchronously before invoking a multimodal metric, so it cannot probe here. It returns an already-resolved verdict when available, otherwise the deterministic blessed-list/flag verdict. When neither is conclusive (an unknown judge that has not been resolved yet), it returns True so it does not block on its own; the authoritative gate is the async resolve_multimodal_support that callers run before a_measure.

Returns:

Name Type Description
bool bool

True when the judge chain is treated as image-capable.

check_multimodal_capability(metric_name, wrapper, is_multimodal) async

Raise if a multimodal row's judge cannot be confirmed to accept images.

Parameters:

Name Type Description Default
metric_name str

The metric's name, for the error message.

required
wrapper object

The metric's judge wrapper, if any.

required
is_multimodal bool

Whether the row being evaluated carries media.

required

Raises:

Type Description
EvalUnsupportedOperationError

If the row is multimodal and the wrapper cannot confirm image support.

BaseError

The judge's own failure when the capability probe fails for an operational reason, such as authentication, rate limiting, a connection failure or a timeout.

deepeval_score(metric, **fields)

Read the score that a DeepEval metric computed from the judge's answers after a_measure.

A NaN or infinite score comes from the judge's answers, for example a verdict list with no items, so it is an unusable judge answer and not a defect.

Parameters:

Name Type Description Default
metric Any

The DeepEval metric after a_measure.

required
**fields Any

More result fields, such as success.

{}

Returns:

Name Type Description
MetricScore MetricScore

The score, and the metric's reason as the explanation.

Raises:

Type Description
EvalJudgeResponseError

If the score is NaN or infinite.

EvalResultContractError

If the score has another invalid value.

parse_judge_reply(reply, schema)

Turn one judge reply into the text or the validated schema instance that DeepEval expects.

A live judge call parses its reply here inside its attempt, so an unusable reply tries the next judge. A cached batch reply is parsed here too, so both paths accept the same replies.

Parameters:

Name Type Description Default
reply LMOutput | str

The judge reply. An invoker with an output transformer can return text.

required
schema type[BaseModel] | None

The pydantic model DeepEval asked for, or None for a text reply.

required

Returns:

Type Description
str | BaseModel

str | BaseModel: The reply text, or a validated instance of schema.

Raises:

Type Description
EvalJudgeResponseError

If the reply is empty, does not match the schema, or holds NaN or infinity.