Evaluate Suites

Evaluate Suites Module.

This module provides a helper function for evaluating different data partitions (suites) with different evaluator sets under a shared run_id and experiment tracker.

EvalSuite

Bases: BaseModel

Evaluation suite defining a data partition and its evaluators.

Attributes:

Name	Type	Description
data	str | BaseDataset | list[LLMTestCase]	Input data as a string path, BaseDataset, or list of evaluation inputs.
evaluators	list[BaseEvaluator]	List of evaluators to apply to this suite's data.
name	str | None	Optional name for the suite. If not provided, auto-generated as suite_0, suite_1, etc.

from_yaml(source, extra_dataset_kwargs=None) staticmethod

Build an EvalSuite from a single YAML suite document.

See gllm_evals/utils/yaml_suite_loader.py for the full YAML format, evaluator/metric resolution rules, ${VAR} substitution, and dataset resolution behavior.

Parameters:

Name	Type	Description	Default
source	str | Path	Path to the YAML suite document.	required
extra_dataset_kwargs	dict[str, Any] | None	Extra kwargs merged into dataset resolution, for values that can't be expressed in YAML (e.g. a Langfuse client). Defaults to None.	None

Returns:

Name	Type	Description
EvalSuite	EvalSuite	The constructed suite.

Raises:

Type	Description
ValueError	If the document is malformed, or an evaluator/metric/dataset reference is invalid.

from_yaml_dir(directory, pattern='*.y*ml', extra_dataset_kwargs=None) staticmethod

Build one EvalSuite per matching YAML file in a directory.

Parameters:

Name	Type	Description	Default
directory	str | Path	Directory containing one YAML file per suite.	required
pattern	str	Glob pattern for matching files. Defaults to ".yml".	'*.y*ml'
extra_dataset_kwargs	dict[str, Any] | None	Forwarded identically to every file. Defaults to None.	None

Returns:

Type	Description
list[EvalSuite]	list[EvalSuite]: One suite per matching file, sorted by path relative to directory.

Raises:

Type	Description
ValueError	If no files match, or any single file fails to load.

evaluate_suites(suites, experiment_tracker=None, batch_size=10, allow_batch_evaluation=False, run_aggregators=None, dataset_name=None, run_id=None, **kwargs) async

Evaluate multiple suites with different evaluators under a shared run_id.

Allows different data partitions (suites) to use different evaluator sets while sharing one tracker and one run_id. Dataset names are namespaced per suite. Warns on duplicate test case content.

Parameters:

Name	Type	Description	Default
suites	list[EvalSuite]	List of evaluation suites, each with data and evaluators.	required
experiment_tracker	type[BaseExperimentTracker] | BaseExperimentTracker | None	Tracker class, instance, or None for CSV default.	None
batch_size	int	Batch size for evaluation (runner-level chunking). Defaults to 10.	10
allow_batch_evaluation	bool	Enable batch processing for LLM API calls. Defaults to False.	False
run_aggregators	list[RunAggregatorCallable] | None	Custom run aggregators. Defaults to None.	None
dataset_name	str | None	Base dataset name. If None, auto-generated with timestamp.	None
run_id	str | None	Shared run ID. If None, auto-generated.	None
**kwargs	Any	Additional configuration parameters. project_name (str), if passed here, is only used when experiment_tracker is a class or None -- an already-built tracker instance's own project_name always takes precedence (see _resolve_project_name_and_tracker()).	{}

Returns:

Name	Type	Description
SuiteExperimentResult	SuiteExperimentResult	Top-level result with per-suite results and pooled aggregators.

Raises:

Type	Description
ValueError	If the resolved run_id contains a path separator -- raised here, before any suite is evaluated, so a dataset_name like "team/orders" fails before evaluators run rather than after.