Base runner

Base class for runner.

Authors

Christina Alexandra (christina.alexandra@gdplabs.id)

References

NONE

BaseRunner(data, inference_fn, evaluators, experiment_tracker=None, batch_size=10, **kwargs)

Bases: ABC

Abstract class for runner.

This class defines the interface for all runner.

Attributes:

Name	Type	Description
data	str | BaseDataset	The data to evaluate.
inference_fn	Callable	The inference function to use.
evaluators	list[BaseEvaluator]	The evaluators to use.
experiment_tracker	BaseExperimentTracker | None	The experiment tracker.
batch_size	int	The batch size to use for evaluation.
**kwargs	int	Additional configuration parameters.

Initialize the runner.

Parameters:

Name	Type	Description	Default
data	str | BaseDataset	The data to evaluate.	required
inference_fn	Callable	The inference function to use.	required
evaluators	list[BaseEvaluator]	The evaluators to use.	required
experiment_tracker	BaseExperimentTracker | None	The experiment tracker.	None
batch_size	int	The batch size to use for evaluation.	10
**kwargs	Any	Additional configuration parameters.	{}

evaluate() abstractmethod async

Run the evaluator on the dataset.

The dataset is evaluated in batches of the given batch size.

Returns:

Name	Type	Description
EvaluationResult	EvaluationResult	Structured result containing evaluation results and experiment URLs/paths.