Skip to content

Yaml Suite Loader

Build EvalSuite instances from a YAML suite document.

build_eval_suite_from_yaml(source, extra_dataset_kwargs=None, _default_name=None)

Parse a single YAML file into a fully-constructed EvalSuite.

Parameters:

Name Type Description Default
source str | Path

Path to the YAML suite document.

required
extra_dataset_kwargs dict[str, Any] | None

Extra kwargs merged into dataset resolution, for values that can't be expressed in YAML.

None
_default_name str | None

Internal-only fallback name, set exclusively by build_eval_suites_from_dir to a path-relative-to-directory name (FR-009). Not part of EvalSuite.from_yaml's public signature.

None

Returns:

Name Type Description
EvalSuite EvalSuite

The constructed suite.

Raises:

Type Description
EvalConfigurationError

If the document is malformed or an evaluator/metric/dataset reference is invalid.

EvalResourceNotFoundError

If the file does not exist.

EvalInputError

If an inline data row is invalid.

BaseError

Any other normalized failure. Every failure mode is normalized here (not left to each caller) and carries a note naming the suite file, so both EvalSuite.from_yaml and build_eval_suites_from_dir inherit the same guarantee. A normalized error raised while building an evaluator or metric is re-raised unchanged.

build_eval_suites_from_dir(directory, pattern='*.y*ml', extra_dataset_kwargs=None)

Load every matching YAML file in a directory into a list of EvalSuites.

Parameters:

Name Type Description Default
directory str | Path

Directory containing one YAML file per suite.

required
pattern str

Glob pattern for matching files. Defaults to ".yml".

'*.y*ml'
extra_dataset_kwargs dict[str, Any] | None

Forwarded identically to every file.

None

Returns:

Type Description
list[EvalSuite]

list[EvalSuite]: One suite per matching file, sorted by path relative to directory.

Raises:

Type Description
EvalResourceNotFoundError

If no files match.

BaseError

The normalized failure of the first file that fails to load.

register_evaluator_class(name, cls)

Register a custom evaluator class under a short name for use in YAML suites.

Parameters:

Name Type Description Default
name str

The class_name value YAML entries will reference.

required
cls type[BaseEvaluator]

The evaluator class to register.

required

register_metric_class(name, cls)

Register a custom metric class under a short name for use in YAML suites.

Parameters:

Name Type Description Default
name str

The class_name value YAML entries will reference.

required
cls type[BaseMetric]

The metric class to register.

required