Deepeval Wrapper
DeepEval-compatible LLM wrapper with automatic fallback chain support.
Wraps BaseLMInvoker as a DeepEvalBaseLLM + LMComponent, iterating through primary and ordered fallback invokers on failure with schema state safety.
DeepEvalLLMWrapper(lm_invoker, fallback_lms=None, skip_multimodal_check=False)
Bases: UsageRecordingMixin, DeepEvalBaseLLM, LMComponent
DeepEval-compatible LLM wrapper built on LMComponent.
Provides the DeepEvalBaseLLM interface with automatic fallback chain support.
When the primary invoker fails with a gllm-inference error, an unusable answer or a
timeout, the next invoker in fallback_lms is tried in order (see utils/judge_chain.py).
Attributes:
| Name | Type | Description |
|---|---|---|
lm_invoker |
BaseLMInvoker
|
Primary language model invoker. |
fallback_lms |
list[BaseLMInvoker]
|
Ordered fallback invokers. |
Initializes the DeepEvalLLMWrapper class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
lm_invoker
|
BaseLMInvoker
|
The LMInvoker instance to wrap. |
required |
fallback_lms
|
list[BaseLMInvoker] | None
|
Ordered fallback invokers. Defaults to None. |
None
|
skip_multimodal_check
|
bool
|
When True, the caller asserts the judge supports image input, so capability resolution is skipped for models not in the blessed list (no probe). Defaults to False. |
False
|
winning_invoker
property
The invoker that handled the most recent successful request.
a_generate(prompt, schema=None)
async
Runs the model to output LLM response asynchronously.
The response schema is applied to all invokers (primary and fallbacks) before _invoke_lm is called, so whichever invoker handles the request uses the correct structured-output format. All schemas are restored in the finally block.
The lock serialises concurrent calls that share the same primary invoker to prevent schema state races.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompt
|
str
|
The complete prompt string built by DeepEval. |
required |
schema
|
type[BaseModel] | None
|
Pydantic model for structured output. Defaults to None. |
None
|
Returns:
| Type | Description |
|---|---|
str | BaseModel
|
str | BaseModel: Text response or validated BaseModel instance. |
Raises:
| Type | Description |
|---|---|
EvalJudgeResponseError
|
If no judge gave a usable reply and the last attempt was unusable. |
generate(*args, **kwargs)
Runs the model to output LLM response.
This method is not implemented because our LM invoker does not support synchronous generation.
A gllm_core.concurrency.syncify bridge was tried here (to unblock
TurnFaithfulnessMetric, which calls this from inside its own async a_measure;
TopicAdherence did too until it was fixed upstream in 4.0.7) and reverted
after a real Vertex call proved it unsafe: the underlying google-genai async
client's aiohttp session becomes bound to whatever event loop first used it. A
wrapper that had already served an a_generate call on the main loop then failed
the bridged sync call with RuntimeError: Task <...> got Future <...> attached to
a different loop -- exactly the mixed sync/async pattern DeepEval exercises, so
this is not a corner case. Faithfulness escapes via include_reason=False, which
short-circuits before the sync call; deepeval_patches then rebuilds its reason
from the claim verdicts.
Raises:
| Type | Description |
|---|---|
EvalUnsupportedSyncOperationError
|
Always. |
get_model_name(*args, **kwargs)
Return a placeholder model name for DeepEval compatibility.
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
Placeholder name "dummy-model". |
load_model(*args, **kwargs)
No-op — model loading is handled by the invoker.
resolve_multimodal_support()
async
Resolve and cache whether the judge chain accepts image input.
Capability spans the whole chain (primary then fallbacks), because multimodal
generation falls back through it: the chain is capable if any judge is. Per
judge the order is blessed list -> skip_multimodal_check flag -> probe.
The probe is authoritative (fail-closed). A capable verdict is cached per
model_id and resolved single-flight across wrappers; a failed probe is cached
only on this wrapper instance, so a later run with a fresh wrapper re-probes
instead of being permanently blocked by a transient failure.
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True when at least one judge in the chain is image-capable. |
supports_multimodal()
Return whether the judge chain accepts image input (synchronous view).
DeepEval calls this synchronously before invoking a multimodal metric, so it
cannot probe here. It returns an already-resolved verdict when available,
otherwise the deterministic blessed-list/flag verdict. When neither is
conclusive (an unknown judge that has not been resolved yet), it returns True
so it does not block on its own; the authoritative gate is the async
resolve_multimodal_support that callers run before a_measure.
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True when the judge chain is treated as image-capable. |
check_multimodal_capability(metric_name, wrapper, is_multimodal)
async
Raise if a multimodal row's judge cannot be confirmed to accept images.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
metric_name
|
str
|
The metric's name, for the error message. |
required |
wrapper
|
object
|
The metric's judge wrapper, if any. |
required |
is_multimodal
|
bool
|
Whether the row being evaluated carries media. |
required |
Raises:
| Type | Description |
|---|---|
EvalUnsupportedOperationError
|
If the row is multimodal and the wrapper cannot confirm image support. |
BaseError
|
The judge's own failure when the capability probe fails for an operational reason, such as authentication, rate limiting, a connection failure or a timeout. |
deepeval_score(metric, **fields)
Read the score that a DeepEval metric computed from the judge's answers after a_measure.
A NaN or infinite score comes from the judge's answers, for example a verdict list with no items, so it is an unusable judge answer and not a defect.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
metric
|
Any
|
The DeepEval metric after |
required |
**fields
|
Any
|
More result fields, such as |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
MetricScore |
MetricScore
|
The score, and the metric's reason as the explanation. |
Raises:
| Type | Description |
|---|---|
EvalJudgeResponseError
|
If the score is NaN or infinite. |
EvalResultContractError
|
If the score has another invalid value. |
parse_judge_reply(reply, schema)
Turn one judge reply into the text or the validated schema instance that DeepEval expects.
A live judge call parses its reply here inside its attempt, so an unusable reply tries the next judge. A cached batch reply is parsed here too, so both paths accept the same replies.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reply
|
LMOutput | str
|
The judge reply. An invoker with an output transformer can return text. |
required |
schema
|
type[BaseModel] | None
|
The pydantic model DeepEval asked for, or None for a text reply. |
required |
Returns:
| Type | Description |
|---|---|
str | BaseModel
|
str | BaseModel: The reply text, or a validated instance of |
Raises:
| Type | Description |
|---|---|
EvalJudgeResponseError
|
If the reply is empty, does not match the schema, or holds NaN or infinity. |