Deepeval Wrapper

DeepEval-compatible LLM wrapper with automatic fallback chain support.

Wraps BaseLMInvoker as a DeepEvalBaseLLM + LMComponent, iterating through primary and ordered fallback invokers on failure with schema state safety.

DeepEvalLLMWrapper(lm_invoker, fallback_lms=None, skip_multimodal_check=False)

Bases: DeepEvalBaseLLM, LMComponent

DeepEval-compatible LLM wrapper built on LMComponent.

Provides the DeepEvalBaseLLM interface with automatic fallback chain support. When the primary invoker raises BaseInvokerError or TimeoutError, the next invoker in fallback_lms is tried in order.

Attributes:

Name	Type	Description
lm_invoker	BaseLMInvoker	Primary language model invoker.
fallback_lms	list[BaseLMInvoker]	Ordered fallback invokers.

Initializes the DeepEvalLLMWrapper class.

Parameters:

Name	Type	Description	Default
lm_invoker	BaseLMInvoker	The LMInvoker instance to wrap.	required
fallback_lms	list[BaseLMInvoker] | None	Ordered fallback invokers. Defaults to None.	None
skip_multimodal_check	bool	When True, the caller asserts the judge supports image input, so capability resolution is skipped for models not in the blessed list (no probe). Defaults to False.	False

winning_invoker property

The invoker that handled the most recent successful request.

__getstate__()

Exclude non-picklable asyncio lock from serialization.

asyncio.Lock accumulates _contextvars.Context after use, making copy.deepcopy fail. We drop the lock here and recreate it fresh in __setstate__.

__setstate__(state)

Restore state and recreate a fresh lock for this wrapper.

a_generate(prompt, schema=None) async

Runs the model to output LLM response asynchronously.

The response schema is applied to all invokers (primary and fallbacks) before _invoke_lm is called, so whichever invoker handles the request uses the correct structured-output format. All schemas are restored in the finally block.

The lock serialises concurrent calls that share the same primary invoker to prevent schema state races.

Parameters:

Name	Type	Description	Default
prompt	str	The complete prompt string built by DeepEval.	required
schema	Optional[BaseModel]	Pydantic model for structured output. Defaults to None.	None

Returns:

Type	Description
str | BaseModel	str | BaseModel: Text response or validated BaseModel instance.

a_generate_raw_response(prompt, **kwargs) async

Generate raw response for DeepEval GEval metrics.

DeepEval's GEval metric expects this method to return a tuple of (response_object, cost) where response_object has either: 1. choices[0].message.content (OpenAI format) - preferred 2. score and reason attributes (fallback format)

Delegates to a_generate so the fallback chain applies here too.

Parameters:

Name	Type	Description	Default
prompt	str	The prompt to run the model on.	required
**kwargs	Any	Additional arguments (ignored).	{}

Returns:

Name	Type	Description
tuple	tuple	A tuple of (response_object, cost) where cost is always 0.0.

generate(*args, **kwargs)

Runs the model to output LLM response.

This method is not implemented because our LM invoker does not support synchronous generation.

Raises:

Type	Description
NotImplementedError	Always.

get_model_name(*args, **kwargs)

Return a placeholder model name for DeepEval compatibility.

Returns:

Name	Type	Description
str	str	Placeholder name "dummy-model".

load_model(*args, **kwargs)

No-op — model loading is handled by the invoker.

resolve_multimodal_support() async

Resolve and cache whether the judge chain accepts image input.

Capability spans the whole chain (primary then fallbacks), because multimodal generation falls back through it: the chain is capable if any judge is. Per judge the order is blessed list -> skip_multimodal_check flag -> probe. The probe is authoritative (fail-closed). A capable verdict is cached per model_id and resolved single-flight across wrappers; a failed probe is cached only on this wrapper instance, so a later run with a fresh wrapper re-probes instead of being permanently blocked by a transient failure.

Returns:

Name	Type	Description
bool	bool	True when at least one judge in the chain is image-capable.

supports_multimodal()

Return whether the judge chain accepts image input (synchronous view).

DeepEval calls this synchronously before invoking a multimodal metric, so it cannot probe here. It returns an already-resolved verdict when available, otherwise the deterministic blessed-list/flag verdict. When neither is conclusive (an unknown judge that has not been resolved yet), it returns True so it does not block on its own; the authoritative gate is the async resolve_multimodal_support that callers run before a_measure.

Returns:

Name	Type	Description
bool	bool	True when the judge chain is treated as image-capable.