Client Management

Client management helpers for persistent HTTP clients.

ClientManager()

Explicit, caller-owned tracker for clients created within one asyncio run.

Examples:

async with ClientManager() as manager:
    client_config = ClientConfig(http_client=http_client, manager=manager)
    invoker = AnthropicLMInvoker(..., client_config=client_config)
    ...

get_or_create() also works without the async with block -- construct a ClientManager() directly and call release_resources() explicitly when done:

manager = ClientManager()
client_config = ClientConfig(http_client=http_client, manager=manager)
invoker = AnthropicLMInvoker(..., client_config=client_config)
...
await manager.release_resources()

Every client returned by get_or_create() is closed by release_resources() regardless of whether the caller keeps its ClientConfig alive -- the manager tracks each client independent of that config's lifetime.

Initializes a new, empty ClientManager.

is_closed property

Return whether the manager has completed closing tracked clients.

Returns:

Name	Type	Description
bool	bool	True if the manager has completed closing tracked clients.

is_closing property

Return whether the manager is currently closing tracked clients.

Returns:

Name	Type	Description
bool	bool	True if the manager is currently closing tracked clients.

__aenter__() async

Enters the async context manager.

Returns:

Name	Type	Description
ClientManager	ClientManager	This manager instance.

__aexit__(*exc_info) async

Closes every tracked client.

Parameters:

Name	Type	Description	Default
*exc_info	object	Exception info passed by the async with protocol (unused).	()

get_or_create(config, factory, identity, model_id_log, *, provider=None)

Return the tracked client for config's identity if one exists, else build and track one.

Parameters:

Name	Type	Description	Default
config	ClientConfig	The caller's client configuration.	required
factory	Callable[[], ManagedClient]	Builds a fresh client from config.http_client; called only on a cache miss.	required
identity	dict[str, Any]	The calling invoker's own connection parameters (api_key, base_url, organization, headers, timeout, ...).	required
model_id_log	str	Log identifier for the calling invoker.	required
provider	str | None	Provider identifier (e.g. 'cohere', 'twelvelabs'). If not provided, inferred from model_id_log prefix if present. Defaults to None.	None

Returns:

Type	Description
ManagedClient	tuple[ManagedClient, bool]: The live client and whether it is manager-tracked
bool	(False on the closed-manager fallback path).

Raises:

Type	Description
InferenceValueError	If config was already used with a different identity.

release_resources() async

Closes every tracked client and marks this manager closed.

This includes clients built around a caller-supplied config.http_client: opting a ClientConfig into a manager transfers close-ownership of its client to that manager, even for an injected transport.

Concurrent calls are serialized so each tracked client is closed exactly once. If closing any client raises an exception, the failure is surfaced and the manager retains the failed entries so that release_resources() can be retried.

HookedClient(raw_client, teardown=None)

Bases: Generic[T]

Managed-client wrapper for SDK clients with custom async teardown.

The wrapper implements close() and synchronous is_closed() for the existing client manager protocol. It delegates SDK attributes and methods to raw_client and forwards async context management when available. Callers provide the teardown hook that matches their SDK's cleanup API, or let HookedClient resolve it.

Attributes:

Name	Type	Description
raw_client	T	The wrapped provider SDK client.

Wrap a provider client with its teardown hook.

Parameters:

Name	Type	Description	Default
raw_client	T	SDK client to wrap.	required
teardown	TeardownHook | None	Callback that closes raw_client. If None, auto-detected via resolve_default_teardown_hook. Defaults to None.	None

__aenter__() async

Enter the wrapped SDK context when it provides one and return this wrapper.

Returns:

Type	Description
HookedClient[T]	HookedClient[T]: self.

__aexit__(exc_type, exc_val, exc_tb) async

Forward async context exit or use the configured teardown hook.

Parameters:

Name	Type	Description	Default
exc_type	type[BaseException] | None	Exception type from the managed block.	required
exc_val	BaseException | None	Exception raised by the managed block.	required
exc_tb	Any	Traceback for the managed-block exception.	required

Returns:

Type	Description
bool | None	bool | None: The raw client's exception-suppression result, or None when cleanup is performed through the teardown hook.

__getattr__(name)

Delegate public SDK attributes to the wrapped client.

Parameters:

Name	Type	Description	Default
name	str	Attribute or provider operation to retrieve.	required

Returns:

Name	Type	Description
Any	Any	The matching attribute on raw_client.

Raises:

Type	Description
AttributeError	If the attribute does not exist or is private.

close() async

Run the teardown hook once, serializing concurrent close requests.

is_closed()

Check whether the client is closed.

Returns:

Name	Type	Description
bool	bool	True if the client is closed, False otherwise.

ManagedClient

Bases: Protocol

Structural shape a client tracker needs: something it can close and check.

close() async

Closes the client and releases its connections.

is_closed()

Returns whether the client is closed.

PersistentClientMixin

Mixin for invokers that own a closable provider client, ephemeral or persistent.

Provides double-checked locking for safe client reacquisition after release_resources() in persistent mode. Subclasses must call _register_client() in __init__ and implement _create_client().

Attributes:

Name	Type	Description
client_config	Any	The client configuration specifying lifecycle and optional ClientManager.
client_kwargs	dict[str, Any]	Connection and configuration kwargs used to build the client.
_model_id	Any	Identifier for the model/invoker, used for logging and provider resolution.

client property writable

The underlying client instance (persistent mode only).

Returns:

Name	Type	Description
Any	Any	The persistent client instance. If wrapped in a HookedClient, the unwrapped raw_client is returned for direct SDK access.

Raises:

Type	Description
InferenceRuntimeError	If accessed in ephemeral mode.

release_resources() async

Close the underlying persistent client, if any, and release its connections.

A safe no-op in ephemeral mode, since no ambient client is kept. In persistent mode, only closes a client this invoker actually owns: manager-tracked clients are left for the ClientManager to close, and caller-supplied client_config.http_client transports are left for the caller to close.

async_aclose_hook(client) async

Invoke the async SDK client's aclose() method.

Parameters:

Name	Type	Description	Default
client	Any	SDK client to close.	required

async_close_hook(client) async

Invoke the async SDK client's coroutine close() method.

Parameters:

Name	Type	Description	Default
client	Any	SDK client to close.	required

noop_hook(client)

Explicit no-op teardown for caller-owned or stateless clients.

Must be passed explicitly when a client has no standard close method. Using this hook signals an intentional choice rather than an oversight.

Parameters:

Name	Type	Description	Default
client	Any	The client instance (ignored).	required

resolve_default_teardown_hook(client)

Inspect client and return the appropriate standard teardown hook.

Resolution priority: 1. aclose() present and callable -> async_aclose_hook 2. Coroutine close() present -> async_close_hook 3. Synchronous close() present -> sync_close_hook 4. None of the above -> raises InferenceValueError

Use noop_hook explicitly for clients with no close API.

Parameters:

Name	Type	Description	Default
client	Any	The raw provider SDK client to inspect.	required

Returns:

Name	Type	Description
TeardownHook	TeardownHook	The selected teardown hook callable.

Raises:

Type	Description
InferenceValueError	If client exposes no supported close method.

resolve_sdk_httpx_client(client)

Return the underlying httpx client of a Fern-style SDK client, if there is one.

Fern-generated SDK clients (such as Cohere and TwelveLabs) expose no close method of their own. They keep their HTTP transport at client._client_wrapper.httpx_client, which is either the httpx client itself or a wrapper holding it at .httpx_client.

Parameters:

Name	Type	Description	Default
client	Any	The SDK client to inspect.	required

Returns:

Type	Description
Any | None	Any | None: The underlying httpx client, or None if the SDK client has no HTTP client.

sync_close_hook(client) async

Teardown hook for clients exposing a synchronous close().

Executes the blocking call on a thread-pool worker via asyncio.to_thread to avoid stalling the event loop during release_resources(). If close() returns an awaitable (such as a sync wrapper returning a coroutine), it is awaited.

Parameters:

Name	Type	Description	Default
client	Any	The client instance exposing a synchronous close() method.	required