Yaml Suite Loader

Build EvalSuite instances from a YAML suite document.

build_eval_suite_from_yaml(source, extra_dataset_kwargs=None, _default_name=None)

Parse a single YAML file into a fully-constructed EvalSuite.

Parameters:

Name	Type	Description	Default
source	str | Path	Path to the YAML suite document.	required
extra_dataset_kwargs	dict[str, Any] | None	Extra kwargs merged into dataset resolution, for values that can't be expressed in YAML (e.g. a Langfuse client).	None
_default_name	str | None	Internal-only fallback name, set exclusively by build_eval_suites_from_dir to a path-relative-to-directory name (FR-009). Not part of EvalSuite.from_yaml's public signature.	None

Returns:

Name	Type	Description
EvalSuite	EvalSuite	The constructed suite.

Raises:

Type	Description
ValueError	If the file can't be read/parsed, the document is malformed, or an evaluator/metric/dataset reference is invalid. Every failure mode is normalized to ValueError and prefixed with source here (not left to each caller), so both EvalSuite.from_yaml and build_eval_suites_from_dir inherit the same guarantee.

build_eval_suites_from_dir(directory, pattern='*.y*ml', extra_dataset_kwargs=None)

Load every matching YAML file in a directory into a list of EvalSuites.

Parameters:

Name	Type	Description	Default
directory	str | Path	Directory containing one YAML file per suite.	required
pattern	str	Glob pattern for matching files. Defaults to ".yml".	'*.y*ml'
extra_dataset_kwargs	dict[str, Any] | None	Forwarded identically to every file.	None

Returns:

Type	Description
list[EvalSuite]	list[EvalSuite]: One suite per matching file, sorted by path relative to directory.

Raises:

Type	Description
ValueError	If no files match, or any single file fails to load.

register_evaluator_class(name, cls)

Register a custom evaluator class under a short name for use in YAML suites.

Parameters:

Name	Type	Description	Default
name	str	The class_name value YAML entries will reference.	required
cls	type[BaseEvaluator]	The evaluator class to register.	required

register_metric_class(name, cls)

Register a custom metric class under a short name for use in YAML suites.

Parameters:

Name	Type	Description	Default
name	str	The class_name value YAML entries will reference.	required
cls	type[BaseMetric]	The metric class to register.	required