Skip to content

Shared Functionality

Shared functionality for general use.

This module contains shared functionality for general use.

WeakAsyncLockRegistry(lock_factory=None)

Bases: Generic[KeyT]

Store asyncio locks keyed by objects using weak references.

Locks are created lazily to avoid requiring an active event loop at construction time.

Initialize the registry with an optional custom lock factory.

Parameters:

Name Type Description Default
lock_factory Callable[[], Lock] | None

The factory to create locks.

None

get(key)

Return the shared lock for the given key, creating it if absent.

Parameters:

Name Type Description Default
key KeyT

The key to get the lock for.

required

Returns:

Type Description
Lock

asyncio.Lock: The shared lock for the given key.

apply_json_columns(row, json_columns)

Parse JSON-serialized string values for the specified columns of a row dict.

Applied before rows enter the dataset constructor, so the constructor always receives plain dicts with every listed column already decoded. Non-string values and missing columns are left unchanged, which makes the call safe for sources (JSONL, Google Sheets) that may already hand back native lists or dicts.

A blank or whitespace-only cell is treated as absent: the key is removed so the model's own default applies (attachments -> {}, an optional field -> None, a required field -> a clear "Field required" error). Leaving the raw string in place would instead surface as Input should be a valid list [input_value=' ']. This also matches from_csv, which already drops empty cells via pd.notna(v), so a quoted-blank cell and a genuinely empty one behave identically.

The row is modified in place and returned.

Parameters:

Name Type Description Default
row dict[str, Any]

The raw row mapping.

required
json_columns list[str]

Column names whose values are JSON-serialized strings; these are the source column names, applied before any field_mapping rename.

required

Returns:

Type Description
dict[str, Any]

dict[str, Any]: The same mapping, with listed columns decoded.

Raises:

Type Description
EvalInputError

If a listed column contains a non-blank string that is not valid JSON.

count_verdicts(rows, eval_name, target_success)

Count one evaluator's matching, scored and errored rows.

Each row counts once, from the evaluator's first output in that row. Errored rows are not scored rows.

Parameters:

Name Type Description Default
rows list[list[EvaluatorResult]]

Row-grouped evaluation results.

required
eval_name str

Name of the evaluator to count.

required
target_success bool

The aggregate_success value to match.

required

Returns:

Type Description
tuple[int, int, int]

tuple[int, int, int]: Matching rows, scored rows and errored rows.

extract_dataset_name_from_path(path, file_extension)

Extract dataset name from file path by removing directory and extension.

Parameters:

Name Type Description Default
path str

The file path (e.g., "/path/to/data.csv", "data.jsonl")

required
file_extension str

The file extension to remove (e.g., ".csv", ".jsonl")

required

Returns:

Name Type Description
str str

The extracted dataset name (e.g., "data")

Examples:

>>> extract_dataset_name_from_path("/path/to/my_data.csv", ".csv")
"my_data"
>>> extract_dataset_name_from_path("evaluation_set.jsonl", ".jsonl")
"evaluation_set"

extract_metrics_with_fallback(data, gt_prefix='gt_', predicted_prefix='predicted_')

Extract metrics starting with gt_prefix, falling back to predicted_prefix if gt is None.

This function finds all keys starting with gt_prefix, and for each metric: - If the gt_ value is not None, use it - If the gt_ value is None, check for the corresponding predicted_ value - If predicted_ value exists and is not None, use it instead - The returned dictionary uses metric names without the prefix

Parameters:

Name Type Description Default
data dict[str, Any]

The dictionary containing metrics.

required
gt_prefix str

Prefix for ground truth metrics. Defaults to "gt_".

'gt_'
predicted_prefix str

Prefix for predicted metrics. Defaults to "predicted_".

'predicted_'

Returns:

Type Description
dict[str, Any]

dict[str, Any]: Dictionary with metric names (without prefix) as keys and their values.

Examples:

Basic usage with fallback: >>> data = { ... "gt_completeness": None, ... "predicted_completeness": 3.0, ... "gt_redundancy": 2.0, ... "predicted_redundancy": 1.0 ... } >>> extract_metrics_with_fallback(data)

All gt values present: >>> data = { ... "gt_completeness": 3.0, ... "gt_redundancy": 2.0 ... } >>> extract_metrics_with_fallback(data)

All gt values None, using predicted: >>> data = { ... "gt_completeness": None, ... "predicted_completeness": 3.0, ... "gt_generation": None, ... "predicted_generation": 0 ... } >>> extract_metrics_with_fallback(data)

Mixed case: >>> data = { ... "gt_completeness": None, ... "predicted_completeness": 3.0, ... "gt_generation": None, ... "predicted_generation": 0, ... "gt_redundancy": 2.0, ... "predicted_redundancy": 1.0 ... } >>> extract_metrics_with_fallback(data)

first_evaluator_outputs(row_results)

Map each evaluator in a row to its first output, which is the evaluator's verdict for that row.

Row items that are not a dict, and entries that are not an EvaluatorOutput, are ignored.

Parameters:

Name Type Description Default
row_results list[EvaluatorResult]

One row's evaluation results.

required

Returns:

Type Description
dict[str, EvaluatorOutput]

dict[str, EvaluatorOutput]: Each evaluator's first output in the row, keyed by evaluator name.

generate_dataset_name_with_timestamp(base_name='dataset')

Generate a dataset name with timestamp postfix.

The timestamp is in compact ISO 8601 format: YYYYMMDDTHHMMSSZ

Parameters:

Name Type Description Default
base_name str

The base name for the dataset. Defaults to "dataset".

'dataset'

Returns:

Name Type Description
str str

The dataset name with timestamp postfix.

Example

generate_dataset_name_with_timestamp("my_dataset") 'my_dataset_20260420T143000Z'

generate_run_id_if_not_provided(project_name, dataset_name, run_id=None)

Generate a run ID.

Parameters:

Name Type Description Default
project_name str

The name of the project.

required
dataset_name str

The name of the dataset.

required
run_id str | None

Provided run id or None.

None

Returns:

Name Type Description
str str

The run ID.

get_nested_dict_value(data, key, return_all=False)

Recursively get the value of a key from a nested dictionary.

This function searches through nested dictionaries and lists containing dictionaries to find the specified key and return its value(s).

Parameters:

Name Type Description Default
data dict[str, Any]

The dictionary to search in.

required
key str

The key to search for.

required
return_all bool

If True, returns all matching values as a list. If False, returns the first match found. Defaults to False.

False

Returns:

Type Description
Any | list[Any] | None

Any | list[Any] | None: The value(s) found for the key. Returns None if key is not found (when return_all=False), or an empty list if not found (when return_all=True).

Examples:

Basic nested dictionary: >>> data = {"level1": {"level2": {"target": "found"}}} >>> get_nested_dict_value(data, "target") "found"

Key at multiple levels (returns first match): >>> data = {"target": "first", "nested": {"target": "second"}} >>> get_nested_dict_value(data, "target") "first"

Get all matches: >>> data = {"target": "first", "nested": {"target": "second"}} >>> get_nested_dict_value(data, "target", return_all=True) ["first", "second"]

List of dictionaries: >>> data = {"items": [{"id": 1, "name": "A"}, {"id": 2, "name": "B"}]} >>> get_nested_dict_value(data, "name") "A"

Complex nested structure: >>> data = { ... "config": {"settings": {"timeout": 30}}, ... "users": [{"name": "Alice", "role": "admin"}, {"name": "Bob"}], ... "metadata": {"tags": ["important"], "timeout": 60} ... } >>> get_nested_dict_value(data, "timeout") 30 >>> get_nested_dict_value(data, "timeout", return_all=True) [30, 60]

Key not found: >>> get_nested_dict_value({"a": 1}, "missing") None >>> get_nested_dict_value({"a": 1}, "missing", return_all=True) []

is_errored_row(output)

Tell whether a row has no verdict because its evaluator or every attempted metric failed.

A failure inside a metric is returned as a failed MetricResult and leaves no error on the evaluator output, so the output alone cannot tell that failure from a verdict. Skipped metrics were not attempted. A row with at least one scored metric keeps its verdict.

Parameters:

Name Type Description Default
output EvaluatorOutput

One evaluator's output for one row.

required

Returns:

Name Type Description
bool bool

True if the row has no verdict to count.

parse_list_strings_in_dict(data)

Parse string representations of lists in dictionary values.

Recursively processes all values in a dictionary, converting string representations of lists (e.g., '["abc", "def"]') into actual Python lists. Also handles CSV-escaped JSON strings (e.g., '[""abc"", ""def""]').

Parameters:

Name Type Description Default
data dict[str, Any]

Dictionary to process

required

Returns:

Type Description
dict[str, Any]

dict[str, Any]: Dictionary with parsed list strings converted to actual lists

Examples:

Basic string to list conversion: >>> parse_list_strings_in_dict({"key1": "abc", "key2": '["item1", "item2"]'})

Nested dictionary processing: >>> parse_list_strings_in_dict({"nested": {"list": '["a", "b"]'}}) {"nested": {"list": ["a", "b"]}}

CSV-escaped JSON handling: >>> parse_list_strings_in_dict({"csv_escaped": '[""item1"", ""item2""]'})

Mixed data types: >>> parse_list_strings_in_dict({"str": "hello", "list": '["a", "b"]', "num": 42, "bool": True})

List of dictionaries with nested parsing: >>> parse_list_strings_in_dict({"items": [{"name": '["item1", "item2"]'}, {"id": 123}]}) {"items": [{"name": ["item1", "item2"]}, {"id": 123}]}

Complex nested structure: >>> data = { ... "config": {"features": '["auth", "logging"]'}, ... "users": [{"roles": '["admin", "user"]'}, {"status": "active"}], ... "settings": {"enabled": True, "timeout": 30} ... } >>> parse_list_strings_in_dict(data) { "config": {"features": ["auth", "logging"]}, "users": [{"roles": ["admin", "user"]}, {"status": "active"}], "settings": {"enabled": True, "timeout": 30} }

Single-quoted strings (AST parsing): >>> parse_list_strings_in_dict({"single_quotes": "['a', 'b', 'c']"})

Non-list strings (unchanged): >>> parse_list_strings_in_dict({"not_list": "just a string", "empty": ""})