Geval generation evaluator

GEval Generation Evaluator.

GEvalGenerationEvaluator(metrics=None, enabled_metrics=None, model=DefaultValues.MODEL, model_credentials=None, model_config=None, num_judges=DefaultValues.NUM_JUDGES, aggregation_method=None, max_concurrent_judges=None, run_parallel=True, judge=None, refusal_metric=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None)

Bases: BaseGenerationEvaluator

GEval Generation Evaluator.

This evaluator is used to evaluate the generation of the model.

Default expected input

• input (str): The input provided to the AI system or component (e.g., a query, prompt, or instruction).
• retrieved_context (str): Supporting context used during generation (e.g., retrieved documents).
• expected_output (str): The reference output used for comparison.
• actual_output (str): The output generated by the AI system or component to evaluate.

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
metrics	List[BaseMetric]	The list of metrics to evaluate.
run_parallel	bool	Whether to run the metrics in parallel.
metrics_aggregator	MetricsAggregator	The aggregator for polarity-aware binary scoring.

Initialize the GEval Generation Evaluator.

Parameters:

Name	Type	Description	Default
metrics	List[BaseMetric] | None	The list of metrics to evaluate.	None
enabled_metrics	List[type[BaseMetric] | str] | None	The list of enabled metrics.	None
model	str | ModelId | BaseLMInvoker	The model to use for the metrics.	MODEL
model_credentials	str | None	The model credentials to use for the metrics.	None
model_config	dict[str, Any] | None	The model config to use for the metrics.	None
num_judges	int	The number of judges to use for the metric. Defaults to 1.	NUM_JUDGES
aggregation_method	AggregationSelector | None	The aggregation method to use for each metric. If None, each metric uses its own default (MAJORITY_VOTE for GEval metrics).	None
max_concurrent_judges	int | None	The maximum number of concurrent judges per metric. If None, each metric uses its own default.	None
run_parallel	bool	Whether to run the metrics in parallel.	True
judge	List[Dict[str, Any]] | None	Judge configuration for metric-level aggregation. List of judge model configs with different models for heterogeneous judges.	None
refusal_metric	GEvalRefusalMetric | None	The refusal metric to use. If None, the default refusal metric will be used. Defaults to GEvalRefusalMetric.	None
batch_status_check_interval	float	Time between batch status checks in seconds. Defaults to 30.0.	BATCH_STATUS_CHECK_INTERVAL
batch_max_iterations	int	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	BATCH_MAX_ITERATIONS
metrics_aggregator	MetricsAggregator | None	The aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	None