Custom Prompt Utils
Shared utility functions for custom prompt handling across metric classes.
This module provides reusable functions for extracting and applying custom prompts, reducing code duplication across LMBasedMetric and DeepEval metrics.
append_evaluation_steps_to_text(steps, base_text)
Append formatted evaluation steps to base text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
steps
|
list | str
|
List of evaluation steps or string. |
required |
base_text
|
str
|
Base text to append to. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
Base text with evaluation steps appended. |
apply_fewshot_with_tags(fewshot_text, mode, base_prompt, metric_name)
Apply fewshot examples to a prompt with tag support.
Supports two modes:
- replace: Replaces content between
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fewshot_text
|
str
|
The fewshot examples text. |
required |
mode
|
str
|
Mode ("replace" or "append"). |
required |
base_prompt
|
str
|
The base prompt template. |
required |
metric_name
|
str
|
Name of the metric (for error messages). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
Modified prompt with fewshot examples applied. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If replace mode is used without |
apply_runtime_fewshot_and_info(fewshot, info, mode, base_context, metric_name)
Apply runtime fewshot and temp_info to base context.
Wraps content in XML tags and combines into a single context string. Used by DeepEvalGEvalMetric for runtime parameter application.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fewshot
|
str | None
|
Runtime fewshot examples (or None). |
required |
info
|
str | None
|
Runtime additional info (or None). |
required |
mode
|
str
|
How to merge fewshot ("append" or "replace"). |
required |
base_context
|
str | None
|
Existing context to build upon. |
required |
metric_name
|
str
|
Name of the metric (for validation). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
Combined context with fewshot and info sections. |
extract_custom_prompt(data, metric_name)
Extract custom prompts for a metric from data row.
Looks for CSV columns following the pattern: - temp_fewshot_{metric_name}: Few-shot examples (primary) - fewshot_{metric_name}: Few-shot examples (deprecated, emits warning) - temp_fewshot_{metric_name}mode: Mode ("append" or "replace") - temp_info: Per-row domain info - evaluation_step_{metric_name}: Evaluation steps
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
LLMTestCase
|
The input data row. |
required |
metric_name
|
str
|
Name of the metric. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Dictionary containing extracted custom prompts with keys: |
dict
|
|
dict
|
|
dict
|
|
format_evaluation_steps(steps)
Format evaluation steps as numbered text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
steps
|
list | str
|
List of evaluation steps or string. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
Formatted evaluation steps text with numbering. |
Examples:
>>> format_evaluation_steps(["Check grammar", "Check clarity"])
"1. Check grammar\\n2. Check clarity"
>>> format_evaluation_steps("Single step")
"Single step"