Dataset

Dataset module for loading and managing evaluation datasets.

This module provides dataset classes and utilities for loading evaluation data from various sources including HuggingFace datasets, Google Sheets, CSV files, and JSONL files. It includes a factory pattern for automatic dataset type detection and creation.

Supported dataset types: - BaseDataset: Abstract base class for all datasets - DictDataset: Dictionary-based dataset implementation - HuggingFaceDataset: Load datasets from HuggingFace Hub - SpreadsheetDataset: Load datasets from Google Sheets or CSV files

BaseDataset(version=None, hash=None, name=None, description=None, schema=None, additional_metadata=None, attachments_config=None)

Bases: ABC, Iterable

Base class for all datasets.

Attributes:

Name	Type	Description
dataset	list[LLMTestCase]	The dataset to evaluate.
version	str | None	The version of the dataset.
hash	str | None	The hash of the dataset.
name	str | None	The name of the dataset.
description	str | None	The description of the dataset.
schema	type[BaseModel] | None	The schema of the dataset.
additional_metadata	dict[str, Any] | None	Additional metadata of the dataset.

Initialize the dataset.

Parameters:

Name	Type	Description	Default
version	str | None	The version of the dataset. Defaults to None.	None
hash	str | None	The hash of the dataset. Defaults to None.	None
name	str | None	The name of the dataset. Defaults to None.	None
description	str | None	The description of the dataset. Defaults to None.	None
schema	type[BaseModel] | None	The schema of the dataset. Defaults to None.	None
additional_metadata	dict[str, Any] | None	Additional metadata of the dataset. Defaults to None.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None

__getitem__(index)

Get the item at the given index.

Parameters:

Name	Type	Description	Default
index	int	The index of the item to get.	required

Returns:

Type	Description
LLMTestCase | list[LLMTestCase]	LLMTestCase | list[LLMTestCase]: The item at the given index or a list of items if the index is a list.

__iter__()

Iterate over the dataset.

Returns:

Type	Description
Iterator[LLMTestCase]	Iterator[LLMTestCase]: An iterator over the dataset.

__len__()

Get the length of the dataset.

Returns:

Name	Type	Description
int	int	The length of the dataset.

filter(filter_fn)

Filter the dataset.

Parameters:

Name	Type	Description	Default
filter_fn	Callable[[LLMTestCase], bool]	The filter function.	required

Returns:

Type	Description
list[LLMTestCase]	list[LLMTestCase]: The filtered items.

load() abstractmethod

Load the dataset.

Returns:

Type	Description
list[LLMTestCase]	list[LLMTestCase]: The loaded dataset.

Raises:

Type	Description
NotImplementedError	If the load method is not implemented.

map(map_fn)

Map the dataset.

Parameters:

Name	Type	Description	Default
map_fn	Callable[[LLMTestCase], LLMTestCase]	The map function.	required

Returns:

Type	Description
list[LLMTestCase]	list[LLMTestCase]: The mapped items.

sample(n=3)

Sample n items from the dataset.

Parameters:

Name	Type	Description	Default
n	int	The number of items to sample.	3

Returns:

Type	Description
list[LLMTestCase]	list[LLMTestCase]: The sampled items.

shuffle()

Shuffle the dataset.

to_standard_format()

Convert dataset to standard format for inference.

Returns LLMTestCase instances directly — self.dataset already holds list[LLMTestCase] after Task 1.

Returns:

Type	Description
list[LLMTestCase]	list[LLMTestCase]: Dataset in standard format.

validate() abstractmethod

Validate the dataset.

Raises:

Type	Description
NotImplementedError	If the validate method is not implemented.

DictConversationalDataset(dataset, dataset_name=None, test_case_type=None, attachments_config=None, field_mapping=None)

Bases: DictDataset

A DictDataset whose factories default to ConversationalTestCase instead of LLMTestCase.

DictDataset(dataset, dataset_name=None, test_case_type=None, attachments_config=None, field_mapping=None)

Bases: BaseDataset

Dict-Based Dataset.

This class is a subclass of the BaseDataset class. It is used to store a dataset in a dictionary format.

Rows are stored raw and constructed in load() using the test_case_type fixed at construction (default LLMTestCase -- backwards compatible). The type is on the constructor rather than on load() because BaseDataset.__init__ ends with self.dataset = self.load(): a load()-only type would leave ds.dataset, len(ds) and iteration showing a junk LLMTestCase view of every conversation.

Attributes:

Name	Type	Description
dataset	list[LLMTestCase] | list[ConversationalTestCase]	The dataset to evaluate.

Initialize the DictDataset class.

Parameters:

Name	Type	Description	Default
dataset	list[dict] | list[LLMTestCase] | list[ConversationalTestCase]	The dataset to use for the evaluation. Raw mappings are constructed as test_case_type in load(); already-built model instances pass through untouched.	required
dataset_name	str | None	The name of the dataset.	None
test_case_type	TestCaseType | None	The model each raw row is constructed as. Defaults to _default_test_case_type.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None
field_mapping	dict[str, str] | None	Mapping of source column names to test-case field names, applied in load(). Unmapped keys pass through unchanged. Defaults to None.	None

from_csv(path, dataset_name=None, test_case_type=None, attachments_config=None, json_columns=None, field_mapping=None, **kwargs) classmethod

Load a dataset from a CSV file.

CSV cannot represent nested structures natively, so a ConversationalTestCase transcript is stored as a JSON-serialized string in a turns column and listed in json_columns (FR-017)::

DictDataset.from_csv(
    "conversations.csv",
    test_case_type=ConversationalTestCase,
    json_columns=["turns"],
)

Parameters:

Name	Type	Description	Default
path	str	The path to the CSV file.	required
dataset_name	str | None	The name of the dataset. If None, defaults to filename. Defaults to None.	None
test_case_type	TestCaseType | None	The model each row is constructed as. Defaults to cls._default_test_case_type.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None
json_columns	list[str] | None	Source column names (before any field_mapping rename) whose values are JSON-serialized strings to decode before construction. Defaults to None.	None
field_mapping	dict[str, str] | None	Mapping of source column names to test-case field names. Unmapped keys pass through unchanged. Defaults to None.	None
**kwargs	Any	Additional arguments to pass to pandas read_csv.	{}

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded dataset.

Raises:

Type	Description
EvalResourceNotFoundError	If the file does not exist.
EvalInputError	If the file is empty or malformed, or a row is incompatible with test_case_type.
BaseError	A normalized error for any other read failure, such as a decoding or I/O error.

from_gsheets(sheet_id, worksheet_name, client_email, private_key, dataset_name=None, test_case_type=None, attachments_config=None, json_columns=None, field_mapping=None) async classmethod

Load a dataset from Google Sheets.

Parameters:

Name	Type	Description	Default
sheet_id	str	The ID of the Google Sheet.	required
worksheet_name	str	The name of the worksheet within the Google Sheet.	required
client_email	str	The client email for Google Sheets API.	required
private_key	str	Base64-encoded private key for Google Sheets API.	required
dataset_name	str | None	The name of the dataset. If None, defaults to worksheet_name. Defaults to None.	None
test_case_type	TestCaseType | None	The model each row is constructed as. Defaults to cls._default_test_case_type.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None
json_columns	list[str] | None	Source column names (before any field_mapping rename) whose values are JSON-serialized strings to decode before construction. Defaults to None.	None
field_mapping	dict[str, str] | None	Mapping of source column names to test-case field names. Unmapped keys pass through unchanged. Defaults to None.	None

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded dataset.

Raises:

Type	Description
EvalInputError	If the worksheet is empty.

from_huggingface_hub(path_or_name, split, dataset_name=None, test_case_type=LLMTestCase, attachments_config=None, field_mapping=None, **kwargs) staticmethod

Load a dataset from HuggingFace Hub.

Parameters:

Name	Type	Description	Default
path_or_name	str	The path or name of the dataset on HuggingFace Hub.	required
split	str	The split of the dataset (e.g. "train", "test").	required
dataset_name	str | None	The name of the dataset. If None, defaults to path_or_name. Defaults to None.	None
test_case_type	TestCaseType	Only LLMTestCase is supported here. Defaults to LLMTestCase.	LLMTestCase
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None
field_mapping	dict[str, str] | None	Mapping of source column names to LLMTestCase field names. Unmapped keys pass through unchanged. Defaults to None.	None
**kwargs	Any	Additional arguments to pass to datasets.load_dataset.	{}

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded dataset.

Raises:

Type	Description
EvalUnsupportedOperationError	If test_case_type is ConversationalTestCase.
EvalInputError	If a row is incompatible with LLMTestCase.
BaseError	The normalized Hub failure, such as EvalResourceNotFoundError or EvalAccessError.

from_jsonl(path, dataset_name=None, test_case_type=None, attachments_config=None, json_columns=None, field_mapping=None, **kwargs) classmethod

Load a dataset from a JSONL file.

Parameters:

Name	Type	Description	Default
path	str	The path to the JSONL file.	required
dataset_name	str | None	The name of the dataset. If None, defaults to filename. Defaults to None.	None
test_case_type	TestCaseType | None	The model each row is constructed as. Defaults to cls._default_test_case_type.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None
json_columns	list[str] | None	Source column names (before any field_mapping rename) whose values are JSON-serialized strings to decode before construction. JSONL stores native JSON, so this is rarely needed here -- it is accepted for parity with from_csv. Defaults to None.	None
field_mapping	dict[str, str] | None	Mapping of source column names to test-case field names. Unmapped keys pass through unchanged. Defaults to None.	None
**kwargs	Any	Additional arguments to pass to the constructor (deprecated, use attachments_config instead).	{}

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded dataset.

Raises:

Type	Description
EvalResourceNotFoundError	If the file does not exist.
EvalInputError	If a line is not valid JSON, the file is empty, or a row is incompatible with test_case_type.

load()

Load the dataset.

Already-built LLMTestCase / ConversationalTestCase instances are returned unchanged. This pass-through is what lets evaluate_suites re-wrap a typed list[ConversationalTestCase] in a default DictDataset without flattening it back into LLMTestCase rows, and keeps the existing direct-construction call sites (evaluate.py, from_huggingface_hub, the simple agent datasets) working with no change.

Returns:

Type	Description
list[LLMTestCase] | list[ConversationalTestCase]	list[LLMTestCase] | list[ConversationalTestCase]: The loaded dataset.

Raises:

Type	Description
EvalInputError	If an item is neither a test-case model nor a mapping, or an already-built instance doesn't match test_case_type.

validate()

Validate the dataset.

Raises:

Type	Description
EvalInputError	If the dataset is not a list of test cases.

HuggingFaceDataset(dataset, dataset_name=None, attachments_config=None, field_mapping=None)

Bases: BaseDataset

Hugging Face dataset class for the evaluator.

Attributes:

Name	Type	Description
dataset	list[LLMTestCase]	The dataset to use for the evaluation.

Initialize the HuggingFaceDataset class.

Parameters:

Name	Type	Description	Default
dataset	Dataset	The dataset to use for the evaluation.	required
dataset_name	str | None	The name of the dataset. Defaults to None.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None
field_mapping	dict[str, str] | None	Mapping of source column names to LLMTestCase field names applied during load(). Unmapped keys pass through unchanged. Defaults to None.	None

from_hub(path_or_name, split, dataset_name=None, attachments_config=None, field_mapping=None, **kwargs) staticmethod

Create a HuggingFaceDataset from a Hugging Face dataset.

Parameters:

Name	Type	Description	Default
path_or_name	str	The path or name of the dataset.	required
split	str	The split of the dataset.	required
dataset_name	str | None	The name of the dataset. If None, defaults to path_or_name. Defaults to None.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None
field_mapping	dict[str, str] | None	Mapping of source column names to LLMTestCase field names applied during load(). Unmapped keys pass through unchanged. Defaults to None.	None
**kwargs	Any	Additional arguments to pass to the load function.	{}

Returns:

Name	Type	Description
HuggingFaceDataset	HuggingFaceDataset	The created dataset.

Raises:

Type	Description
EvalInputError	If a row is incompatible with LLMTestCase.
BaseError	The normalized Hub failure, such as EvalResourceNotFoundError or EvalAccessError.

from_list(dataset, dataset_name=None, attachments_config=None) staticmethod

Create a HuggingFaceDataset from a list of LLMTestCase.

Parameters:

Name	Type	Description	Default
dataset	list[LLMTestCase]	The dataset to create.	required
dataset_name	str | None	The name of the dataset. Defaults to None.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None

Returns:

Name	Type	Description
HuggingFaceDataset	HuggingFaceDataset	The created dataset.

load()

Load the dataset.

Returns:

Type	Description
list[LLMTestCase]	list[LLMTestCase]: The loaded dataset.

validate()

Validate the dataset.

Raises:

Type	Description
EvalInputError	If the dataset is not a list of LLMTestCase.

SpreadsheetDataset(dataset, dataset_name=None, attachments_config=None, field_mapping=None)

Bases: BaseDataset

Spreadsheet dataset class for the evaluator.

Attributes:

Name	Type	Description
dataset	list[LLMTestCase]	The dataset to use for the evaluation.

Initialize the SpreadsheetDataset class.

Parameters:

Name	Type	Description	Default
dataset	Dataset	The dataset to use for the evaluation.	required
dataset_name	str | None	The name of the dataset. Defaults to None.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None
field_mapping	dict[str, str] | None	Mapping of source column names to LLMTestCase field names applied during load(). Unmapped keys pass through unchanged. Defaults to None.	None

from_gsheets(sheet_id, worksheet_name, client_email, private_key, dataset_name=None, attachments_config=None, field_mapping=None) async staticmethod

Load the dataset from Google Sheets.

Parameters:

Name	Type	Description	Default
sheet_id	str	The ID of the Google Sheet.	required
worksheet_name	str	The name of the worksheet within the Google Sheet.	required
client_email	str	The client email for Google Sheets API.	required
private_key	str	Base64-encoded private key for Google Sheets API.	required
dataset_name	str | None	The name of the dataset. If None, defaults to worksheet_name. Defaults to None.	None
attachments_config	AttachmentConfig | dict[str, Any] | None	Configuration for loading attachments. Defaults to None.	None
field_mapping	dict[str, str] | None	Mapping of source column names to LLMTestCase field names applied during load(). Unmapped keys pass through unchanged. Defaults to None.	None

Returns:

Name	Type	Description
SpreadsheetDataset	SpreadsheetDataset	The loaded dataset.

Raises:

Type	Description
EvalInputError	If the worksheet is empty or a row is incompatible with LLMTestCase.

load()

Load the dataset.

Returns:

Type	Description
list[LLMTestCase]	list[LLMTestCase]: The loaded dataset.

validate()

Validate the dataset.

Raises:

Type	Description
EvalInputError	If the dataset is not a list of LLMTestCase.

get_dataset(dataset, **kwargs)

Detect the dataset type and create an instance.

Supported dataset string format

• hf/ (Hugging Face dataset)
• gs/ (Google Sheets dataset) Required parameters: sheet_id, client_email, private_key
• (JSONL file)
• (CSV file)

Parameters:

Name	Type	Description	Default
dataset	str | BaseDataset	The dataset to detect.	required
**kwargs	Any	Additional arguments to pass to the dataset constructor.	{}

Returns:

Type	Description
tuple[BaseDataset, str]	tuple[BaseDataset, str]: The detected dataset and its name.

Raises:

Type	Description
EvalConfigurationError	If the dataset format is not supported.

load_simple_agent_dataset(current_dir=None, filename='simple_agent_data.csv')

Load the simple agent dataset from the local CSV file.

The dataset contains agent interaction data with trajectories, questions, and responses, suitable for both agent trajectory evaluation and generation quality evaluation.

Reads the CSV directly so that JSON-string columns (e.g. agent_trajectory) are parsed by parse_agent_row before LLMTestCase construction.

Parameters:

Name	Type	Description	Default
current_dir	Path | str | None	The directory containing the CSV file. If None, defaults to the examples/sample_data directory.	None
filename	str	The name of the CSV file. Defaults to "simple_agent_data.csv".	'simple_agent_data.csv'

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded simple agent dataset as list[LLMTestCase].

Raises:

Type	Description
EvalResourceNotFoundError	If the CSV file doesn't exist.
EvalInputError	If the CSV file is empty or malformed, or a row is incompatible with LLMTestCase.
BaseError	A normalized error for any other read failure.

load_simple_agent_tool_call_dataset(current_dir=None, filename='simple_agent_tool_call_data.json')

Load the simple agent tool call dataset from the local JSON file.

The dataset contains agent interaction data with tool calls, questions, and responses, suitable for tool correctness evaluation.

Parameters:

Name	Type	Description	Default
current_dir	Path | str | None	The directory containing the JSON file. If None, defaults to the examples/sample_data directory.	None
filename	str	The name of the JSON file. Defaults to "simple_agent_tool_call_data.json".	'simple_agent_tool_call_data.json'

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded simple agent tool call dataset as list[LLMTestCase] with keys: 'question_id', 'query', 'generated_response', 'expected_response', 'tools_called', 'expected_tools', 'agent_trajectory', 'expected_agent_trajectory'.

Raises:

Type	Description
EvalResourceNotFoundError	If the JSON file doesn't exist.
EvalInputError	If the JSON file is empty or malformed.
BaseError	A normalized error for any other read failure.

load_simple_qa_dataset(current_dir=None, filename='simple_qa_data.csv')

Load the simple QA dataset from the local CSV file.

The dataset contains question-answer pairs with generated responses and contexts, suitable for RAG evaluation and testing.

Parameters:

Name	Type	Description	Default
current_dir	Path | str | None	The directory containing the CSV file. If None, defaults to the examples/sample_data directory.	None
filename	str	The name of the CSV file. Defaults to "simple_qa_data.csv".	'simple_qa_data.csv'

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded simple QA dataset as LLMTestCase instances with fields: 'input', 'actual_output', 'expected_output', 'retrieved_context'.

Raises:

Type	Description
EvalResourceNotFoundError	If the CSV file doesn't exist.
EvalInputError	If the CSV file is empty or malformed.
BaseError	A normalized error for any other read failure.

load_simple_rag_dataset(current_dir=None, filename='simple_rag_data.csv')

Load the RAG dataset from the local CSV file.

The dataset contains query-response pairs with generated responses, expected responses, and retrieved contexts, suitable for RAG evaluation and testing.

Parameters:

Name	Type	Description	Default
current_dir	Path | str | None	The directory containing the CSV file. If None, defaults to the examples/sample_data directory.	None
filename	str	The name of the CSV file. Defaults to "simple_rag_data.csv".	'simple_rag_data.csv'

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded RAG dataset as LLMTestCase instances with fields: 'input', 'actual_output', 'expected_output', 'retrieved_context'.

Raises:

Type	Description
EvalResourceNotFoundError	If the CSV file doesn't exist.
EvalInputError	If the CSV file is empty or malformed.
BaseError	A normalized error for any other read failure.

load_simple_summarization_dataset(current_dir=None, filename='simple_summarization_data.csv')

Load the simple summarization dataset from the local CSV file.

Parameters:

Name	Type	Description	Default
current_dir	Path | str | None	The directory containing the CSV file. If None, defaults to the examples/sample_data directory.	None
filename	str	The name of the CSV file. Defaults to "simple_summarization_data.csv".	'simple_summarization_data.csv'

Returns:

Name	Type	Description
DictDataset	DictDataset	The loaded simple summarization dataset containing LLMTestCase dictionaries with keys: 'input' and 'summary'.

Raises:

Type	Description
EvalResourceNotFoundError	If the CSV file doesn't exist.
EvalInputError	If the CSV file is empty or malformed.
BaseError	A normalized error for any other read failure.

load_tool_schema(current_dir=None, filename='tool_schema.json')

Load the tool schema from the local JSON file.

The tool schema contains definitions of available tools that agents can use, including their names, descriptions, and parameter specifications.

Parameters:

Name	Type	Description	Default
current_dir	Path | str | None	The directory containing the JSON file. If None, defaults to the examples/sample_data directory.	None
filename	str	The name of the JSON file. Defaults to "tool_schema.json".	'tool_schema.json'

Returns:

Type	Description
list[dict]	list[dict]: The loaded tool schema containing tool definitions with keys: 'name', 'description', and 'parameters' (following JSON Schema format).

Raises:

Type	Description
EvalResourceNotFoundError	If the JSON file doesn't exist.
EvalInputError	If the JSON file is empty or malformed.
BaseError	A normalized error for any other read failure.