Refusal

Refusal metric.

This metric is used to evaluate the refusal of the model's output.

Authors

Christina Alexandra (christina.alexandra@gdplabs.id)

References

None

RefusalMetric(model=DefaultValues.MODEL, model_credentials=None, model_config=None, prompt_builder=None, response_schema=None)

Bases: LMBasedMetric

Refusal metric.

Attributes:

Name	Type	Description
name	str	The name of the metric.
response_schema	ResponseSchema	The response schema to use for the metric.
prompt_builder	PromptBuilder	The prompt builder to use for the metric.
model	Union[str, ModelId, BaseLMInvoker]	The model to use for the metric.
model_config	dict[str, Any] | None	The model config to use for the metric. Defaults to an empty dictionary.
model_credentials	str	The model credentials to use for the metric.

Initialize the RefusalMetric class.

Default expected input: - query (str): The query to evaluate the refusal of the model's output. - expected_response (str): The expected response to evaluate the refusal of the model's output.

Parameters:

Name	Type	Description	Default
model	Union[str, ModelId, BaseLMInvoker]	The model to use for the metric.	MODEL
model_credentials	str | None	The model credentials to use for the metric. Defaults to None.	None
model_config	dict[str, Any] | None	The model config to use for the metric. Defaults to an empty dictionary.	None
prompt_builder	PromptBuilder | None	The prompt builder to use for the metric. Defaults to default prompt builder.	None
response_schema	ResponseSchema | None	The response schema to use for the metric. Defaults to CompletenessResponseSchema.	None

RefusalResponseSchema

Bases: BaseModel

Response schema for the refusal metric.

Attributes:

Name	Type	Description
question	str	The question that was asked.
expected_output_statements	list[str]	The expected output statements.
count	str	The count of the generated output statements.
score	int	The score of the generated output statements.
explanation	str	The explanation of the generated output statements.