Skip to content

Custom Prompt Utils

Shared utility functions for custom prompt handling across metric classes.

This module provides reusable functions for extracting and applying custom prompts, reducing code duplication across LMBasedMetric and DeepEval metrics.

append_evaluation_steps_to_text(steps, base_text)

Append formatted evaluation steps to base text.

Parameters:

Name Type Description Default
steps list | str

List of evaluation steps or string.

required
base_text str

Base text to append to.

required

Returns:

Name Type Description
str str

Base text with evaluation steps appended.

apply_fewshot_with_tags(fewshot_text, mode, base_prompt, metric_name)

Apply fewshot examples to a prompt with tag support.

Supports two modes: - replace: Replaces content between tags - append: Appends to existing content or adds to end of prompt

Parameters:

Name Type Description Default
fewshot_text str

The fewshot examples text.

required
mode str

Mode ("replace" or "append").

required
base_prompt str

The base prompt template.

required
metric_name str

Name of the metric (for error messages).

required

Returns:

Name Type Description
str str

Modified prompt with fewshot examples applied.

Raises:

Type Description
ValueError

If replace mode is used without tags.

apply_runtime_fewshot_and_info(fewshot, info, mode, base_context, metric_name)

Apply runtime fewshot and temp_info to base context.

Wraps content in XML tags and combines into a single context string. Used by DeepEvalGEvalMetric for runtime parameter application.

Parameters:

Name Type Description Default
fewshot str | None

Runtime fewshot examples (or None).

required
info str | None

Runtime additional info (or None).

required
mode str

How to merge fewshot ("append" or "replace").

required
base_context str | None

Existing context to build upon.

required
metric_name str

Name of the metric (for validation).

required

Returns:

Name Type Description
str str

Combined context with fewshot and info sections.

extract_custom_prompt(data, metric_name)

Extract custom prompts for a metric from data row.

Looks for CSV columns following the pattern: - temp_fewshot_{metric_name}: Few-shot examples (primary) - fewshot_{metric_name}: Few-shot examples (deprecated, emits warning) - temp_fewshot_{metric_name}mode: Mode ("append" or "replace") - temp_info: Per-row domain info - evaluation_step_{metric_name}: Evaluation steps

Parameters:

Name Type Description Default
data LLMTestCase

The input data row.

required
metric_name str

Name of the metric.

required

Returns:

Type Description
dict

Dictionary containing extracted custom prompts with keys:

dict
  • CustomPromptKeys.FEWSHOT: {EXAMPLES: str, MODE: str}
dict
  • CustomPromptKeys.EVALUATION_STEPS: list[str]
dict
  • CustomPromptKeys.TEMP_INFO: str

format_evaluation_steps(steps)

Format evaluation steps as numbered text.

Parameters:

Name Type Description Default
steps list | str

List of evaluation steps or string.

required

Returns:

Name Type Description
str str

Formatted evaluation steps text with numbering.

Examples:

>>> format_evaluation_steps(["Check grammar", "Check clarity"])
"1. Check grammar\\n2. Check clarity"
>>> format_evaluation_steps("Single step")
"Single step"