Shared Functionality
Shared functionality for general use.
This module contains shared functionality for general use.
WeakAsyncLockRegistry(lock_factory=None)
Bases: Generic[KeyT]
Store asyncio locks keyed by objects using weak references.
Locks are created lazily to avoid requiring an active event loop at construction time.
Initialize the registry with an optional custom lock factory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
lock_factory
|
Callable[[], Lock] | None
|
The factory to create locks. |
None
|
get(key)
Return the shared lock for the given key, creating it if absent.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
KeyT
|
The key to get the lock for. |
required |
Returns:
| Type | Description |
|---|---|
Lock
|
asyncio.Lock: The shared lock for the given key. |
apply_json_columns(row, json_columns)
Parse JSON-serialized string values for the specified columns of a row dict.
Applied before rows enter the dataset constructor, so the constructor always receives plain dicts with every listed column already decoded. Non-string values and missing columns are left unchanged, which makes the call safe for sources (JSONL, Google Sheets) that may already hand back native lists or dicts.
A blank or whitespace-only cell is treated as absent: the key is removed so the
model's own default applies (attachments -> {}, an optional field -> None,
a required field -> a clear "Field required" error). Leaving the raw string in place
would instead surface as Input should be a valid list [input_value=' ']. This
also matches from_csv, which already drops empty cells via pd.notna(v), so a
quoted-blank cell and a genuinely empty one behave identically.
The row is modified in place and returned.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
row
|
dict[str, Any]
|
The raw row mapping. |
required |
json_columns
|
list[str]
|
Column names whose values are JSON-serialized strings; these are the source column names, applied before any field_mapping rename. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: The same mapping, with listed columns decoded. |
Raises:
| Type | Description |
|---|---|
EvalInputError
|
If a listed column contains a non-blank string that is not valid JSON. |
count_verdicts(rows, eval_name, target_success)
Count one evaluator's matching, scored and errored rows.
Each row counts once, from the evaluator's first output in that row. Errored rows are not scored rows.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
rows
|
list[list[EvaluatorResult]]
|
Row-grouped evaluation results. |
required |
eval_name
|
str
|
Name of the evaluator to count. |
required |
target_success
|
bool
|
The |
required |
Returns:
| Type | Description |
|---|---|
tuple[int, int, int]
|
tuple[int, int, int]: Matching rows, scored rows and errored rows. |
extract_dataset_name_from_path(path, file_extension)
Extract dataset name from file path by removing directory and extension.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
The file path (e.g., "/path/to/data.csv", "data.jsonl") |
required |
file_extension
|
str
|
The file extension to remove (e.g., ".csv", ".jsonl") |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The extracted dataset name (e.g., "data") |
Examples:
>>> extract_dataset_name_from_path("/path/to/my_data.csv", ".csv")
"my_data"
>>> extract_dataset_name_from_path("evaluation_set.jsonl", ".jsonl")
"evaluation_set"
extract_metrics_with_fallback(data, gt_prefix='gt_', predicted_prefix='predicted_')
Extract metrics starting with gt_prefix, falling back to predicted_prefix if gt is None.
This function finds all keys starting with gt_prefix, and for each metric: - If the gt_ value is not None, use it - If the gt_ value is None, check for the corresponding predicted_ value - If predicted_ value exists and is not None, use it instead - The returned dictionary uses metric names without the prefix
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, Any]
|
The dictionary containing metrics. |
required |
gt_prefix
|
str
|
Prefix for ground truth metrics. Defaults to "gt_". |
'gt_'
|
predicted_prefix
|
str
|
Prefix for predicted metrics. Defaults to "predicted_". |
'predicted_'
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Dictionary with metric names (without prefix) as keys and their values. |
Examples:
Basic usage with fallback: >>> data = { ... "gt_completeness": None, ... "predicted_completeness": 3.0, ... "gt_redundancy": 2.0, ... "predicted_redundancy": 1.0 ... } >>> extract_metrics_with_fallback(data)
All gt values present: >>> data = { ... "gt_completeness": 3.0, ... "gt_redundancy": 2.0 ... } >>> extract_metrics_with_fallback(data)
All gt values None, using predicted: >>> data = { ... "gt_completeness": None, ... "predicted_completeness": 3.0, ... "gt_generation": None, ... "predicted_generation": 0 ... } >>> extract_metrics_with_fallback(data)
Mixed case: >>> data = { ... "gt_completeness": None, ... "predicted_completeness": 3.0, ... "gt_generation": None, ... "predicted_generation": 0, ... "gt_redundancy": 2.0, ... "predicted_redundancy": 1.0 ... } >>> extract_metrics_with_fallback(data)
first_evaluator_outputs(row_results)
Map each evaluator in a row to its first output, which is the evaluator's verdict for that row.
Row items that are not a dict, and entries that are not an EvaluatorOutput, are ignored.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
row_results
|
list[EvaluatorResult]
|
One row's evaluation results. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, EvaluatorOutput]
|
dict[str, EvaluatorOutput]: Each evaluator's first output in the row, keyed by evaluator name. |
generate_dataset_name_with_timestamp(base_name='dataset')
Generate a dataset name with timestamp postfix.
The timestamp is in compact ISO 8601 format: YYYYMMDDTHHMMSSZ
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base_name
|
str
|
The base name for the dataset. Defaults to "dataset". |
'dataset'
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The dataset name with timestamp postfix. |
Example
generate_dataset_name_with_timestamp("my_dataset") 'my_dataset_20260420T143000Z'
generate_run_id_if_not_provided(project_name, dataset_name, run_id=None)
Generate a run ID.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
project_name
|
str
|
The name of the project. |
required |
dataset_name
|
str
|
The name of the dataset. |
required |
run_id
|
str | None
|
Provided run id or None. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The run ID. |
get_nested_dict_value(data, key, return_all=False)
Recursively get the value of a key from a nested dictionary.
This function searches through nested dictionaries and lists containing dictionaries to find the specified key and return its value(s).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, Any]
|
The dictionary to search in. |
required |
key
|
str
|
The key to search for. |
required |
return_all
|
bool
|
If True, returns all matching values as a list. If False, returns the first match found. Defaults to False. |
False
|
Returns:
| Type | Description |
|---|---|
Any | list[Any] | None
|
Any | list[Any] | None: The value(s) found for the key. Returns None if key is not found (when return_all=False), or an empty list if not found (when return_all=True). |
Examples:
Basic nested dictionary: >>> data = {"level1": {"level2": {"target": "found"}}} >>> get_nested_dict_value(data, "target") "found"
Key at multiple levels (returns first match): >>> data = {"target": "first", "nested": {"target": "second"}} >>> get_nested_dict_value(data, "target") "first"
Get all matches: >>> data = {"target": "first", "nested": {"target": "second"}} >>> get_nested_dict_value(data, "target", return_all=True) ["first", "second"]
List of dictionaries: >>> data = {"items": [{"id": 1, "name": "A"}, {"id": 2, "name": "B"}]} >>> get_nested_dict_value(data, "name") "A"
Complex nested structure: >>> data = { ... "config": {"settings": {"timeout": 30}}, ... "users": [{"name": "Alice", "role": "admin"}, {"name": "Bob"}], ... "metadata": {"tags": ["important"], "timeout": 60} ... } >>> get_nested_dict_value(data, "timeout") 30 >>> get_nested_dict_value(data, "timeout", return_all=True) [30, 60]
Key not found: >>> get_nested_dict_value({"a": 1}, "missing") None >>> get_nested_dict_value({"a": 1}, "missing", return_all=True) []
is_errored_row(output)
Tell whether a row has no verdict because its evaluator or every attempted metric failed.
A failure inside a metric is returned as a failed MetricResult and leaves no error on the evaluator output,
so the output alone cannot tell that failure from a verdict. Skipped metrics were not attempted. A row with at
least one scored metric keeps its verdict.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output
|
EvaluatorOutput
|
One evaluator's output for one row. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the row has no verdict to count. |
parse_list_strings_in_dict(data)
Parse string representations of lists in dictionary values.
Recursively processes all values in a dictionary, converting string representations of lists (e.g., '["abc", "def"]') into actual Python lists. Also handles CSV-escaped JSON strings (e.g., '[""abc"", ""def""]').
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, Any]
|
Dictionary to process |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: Dictionary with parsed list strings converted to actual lists |
Examples:
Basic string to list conversion: >>> parse_list_strings_in_dict({"key1": "abc", "key2": '["item1", "item2"]'})
Nested dictionary processing: >>> parse_list_strings_in_dict({"nested": {"list": '["a", "b"]'}}) {"nested": {"list": ["a", "b"]}}
CSV-escaped JSON handling: >>> parse_list_strings_in_dict({"csv_escaped": '[""item1"", ""item2""]'})
Mixed data types: >>> parse_list_strings_in_dict({"str": "hello", "list": '["a", "b"]', "num": 42, "bool": True})
List of dictionaries with nested parsing: >>> parse_list_strings_in_dict({"items": [{"name": '["item1", "item2"]'}, {"id": 123}]}) {"items": [{"name": ["item1", "item2"]}, {"id": 123}]}
Complex nested structure: >>> data = { ... "config": {"features": '["auth", "logging"]'}, ... "users": [{"roles": '["admin", "user"]'}, {"status": "active"}], ... "settings": {"enabled": True, "timeout": 30} ... } >>> parse_list_strings_in_dict(data) { "config": {"features": ["auth", "logging"]}, "users": [{"roles": ["admin", "user"]}, {"status": "active"}], "settings": {"enabled": True, "timeout": 30} }
Single-quoted strings (AST parsing): >>> parse_list_strings_in_dict({"single_quotes": "['a', 'b', 'c']"})
Non-list strings (unchanged): >>> parse_list_strings_in_dict({"not_list": "just a string", "empty": ""})