Custom Prompt Utils

Shared utility functions for custom prompt handling across metric classes.

This module provides reusable functions for extracting and applying custom prompts, reducing code duplication across LMBasedMetric, DeepEval, RAGAS, and LangChain metrics.

append_evaluation_steps_to_text(steps, base_text)

Append formatted evaluation steps to base text.

Parameters:

Name	Type	Description	Default
steps	list | str	List of evaluation steps or string.	required
base_text	str	Base text to append to.	required

Returns:

Name	Type	Description
str	str	Base text with evaluation steps appended.

apply_fewshot_with_tags(fewshot_text, mode, base_prompt, metric_name)

Apply fewshot examples to a prompt with tag support.

Supports two modes: - replace: Replaces content between tags - append: Appends to existing content or adds to end of prompt

Parameters:

Name	Type	Description	Default
fewshot_text	str	The fewshot examples text.	required
mode	str	Mode ("replace" or "append").	required
base_prompt	str	The base prompt template.	required
metric_name	str	Name of the metric (for error messages).	required

Returns:

Name	Type	Description
str	str	Modified prompt with fewshot examples applied.

Raises:

Type	Description
ValueError	If replace mode is used without tags.

apply_runtime_fewshot_and_info(fewshot, info, mode, base_context, metric_name)

Apply runtime fewshot and temp_info to base context.

Wraps content in XML tags and combines into a single context string. Used by DeepEvalGEvalMetric for runtime parameter application.

Parameters:

Name	Type	Description	Default
fewshot	str | None	Runtime fewshot examples (or None).	required
info	str | None	Runtime additional info (or None).	required
mode	str	How to merge fewshot ("append" or "replace").	required
base_context	str | None	Existing context to build upon.	required
metric_name	str	Name of the metric (for validation).	required

Returns:

Name	Type	Description
str	str	Combined context with fewshot and info sections.

evaluate_with_custom_prompt_lifecycle(metric, data) async

Evaluate a single item with custom prompt lifecycle.

This utility function encapsulates the complete custom prompt lifecycle for metrics that support custom prompts:

1. Save current prompt state
2. Apply custom prompts from data
3. Evaluate the item
4. Restore original prompt state (in finally block)

This pattern prevents custom prompts from accumulating across evaluations. Only use this for metrics that implement custom prompt support.

Parameters:

Name	Type	Description	Default
metric		The metric instance (must have _save_prompt_state, _apply_custom_prompts, _evaluate, _namespace_scores, and _restore_prompt_state methods).	required
data		Single data item to evaluate.	required

Returns:

Name	Type	Description
MetricOutput		Evaluation result with scores namespaced by metric name.

Example

async def evaluate(self, data):
    if not isinstance(data, list):
        return await evaluate_with_custom_prompt_lifecycle(self, data)
    # ... handle batch

extract_custom_prompt(data, metric_name)

Extract custom prompts for a metric from data row.

Looks for CSV columns following the pattern: - temp_fewshot_{metric_name}: Few-shot examples (primary) - fewshot_{metric_name}: Few-shot examples (deprecated, emits warning) - temp_fewshot_{metric_name}mode: Mode ("append" or "replace") - temp_info: Per-row domain info - evaluation_step_{metric_name}: Evaluation steps

Parameters:

Name	Type	Description	Default
data	LLMTestCase	The input data row.	required
metric_name	str	Name of the metric.	required

Returns:

Type	Description
dict	Dictionary containing extracted custom prompts with keys:
dict	CustomPromptKeys.FEWSHOT: {EXAMPLES: str, MODE: str}
dict	CustomPromptKeys.EVALUATION_STEPS: list[str]
dict	CustomPromptKeys.TEMP_INFO: str

format_evaluation_steps(steps)

Format evaluation steps as numbered text.

Parameters:

Name	Type	Description	Default
steps	list | str	List of evaluation steps or string.	required

Returns:

Name	Type	Description
str	str	Formatted evaluation steps text with numbering.

Examples:

>>> format_evaluation_steps(["Check grammar", "Check clarity"])
"1. Check grammar\\n2. Check clarity"
>>> format_evaluation_steps("Single step")
"Single step"