Experiment Tracker

Experiment tracker module for logging and tracking evaluation experiments.

This module provides experiment tracking functionality to log evaluation runs, metrics, and results to various tracking backends. It provides CSV and JSON file-based trackers for local development and a Google Sheets tracker for shared results.

Available trackers: - BaseExperimentTracker: Abstract base class for experiment trackers - CSVExperimentTracker: CSV file-based experiment tracking for local use - JSONExperimentTracker: JSON file-based experiment tracking that returns accumulated rows - get_experiment_tracker: Factory function for creating tracker instances - GoogleSheetsExperimentTracker: Integration with Google Sheets for experiment tracking

BaseExperimentTracker(project_name, **kwargs)

Bases: ABC

Base class for all experiment trackers.

This class defines the core interface for experiment tracking across different backends. It provides methods for logging individual results and batch results using the observability pattern.

Attributes:

Name	Type	Description
project_name	str	The name of the project.
supported_test_case_types	tuple[type, ...]	The row types this tracker can log.

Initialize the experiment tracker.

Parameters:

Name	Type	Description	Default
project_name	str	The name of the project.	required
**kwargs	Any	Additional configuration parameters.	{}

alog(evaluation_result, dataset_name, data, run_id=None, metadata=None, **kwargs) abstractmethod async

Log a single evaluation result (asynchronous).

Parameters:

Name	Type	Description	Default
evaluation_result	EvaluatorResult	The evaluation result to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	LLMTestCase | ConversationalTestCase	The input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
**kwargs	Any	Additional configuration parameters.	{}

get_batch_summaries(run_id)

Get the latest cumulative summary for a run.

Parameters:

Name	Type	Description	Default
run_id	str	Identifier of the experiment run to look up.	required

Returns:

Type	Description
dict[str, Any]	dict[str, Any]: The cumulative summary computed by run aggregators across all
dict[str, Any]	batches seen so far for run_id. Returns an empty dict if no aggregators
dict[str, Any]	have run for that run id.

get_experiment_history(**kwargs)

Get all experiment runs.

Parameters:

Name	Type	Description	Default
**kwargs	Any	Additional configuration parameters.	{}

Returns:

Type	Description
list[dict[str, Any]]	list[dict[str, Any]]: List of experiment runs.

get_experiment_urls(run_id, dataset_name, **kwargs) abstractmethod

Get experiment URIs and paths for accessing experiment data.

This method returns URIs or local paths that can be used to access experiment results and leaderboards. Different experiment trackers will return different types of URIs/paths based on their capabilities.

Parameters:

Name	Type	Description	Default
run_id	str	ID of the experiment run.	required
dataset_name	str	Name of the dataset that was evaluated.	required
**kwargs	Any	Additional configuration parameters.	{}

Returns:

Name	Type	Description
ExperimentUris	ExperimentUris	Dictionary containing URIs and paths for accessing experiment data. Different trackers will populate different fields: - Simple: run_uri (local results path), leaderboard_uri (local leaderboard path)

get_run_results(run_id, **kwargs)

Get detailed results for a specific run.

Parameters:

Name	Type	Description	Default
run_id	str	ID of the experiment run.	required
**kwargs	Any	Additional configuration parameters.	{}

Returns:

Type	Description
list[dict[str, Any]]	list[dict[str, Any]]: Detailed run results.

log(evaluation_result, dataset_name, data, run_id=None, metadata=None, **kwargs) abstractmethod

Log a single evaluation result (synchronous).

Parameters:

Name	Type	Description	Default
evaluation_result	EvaluatorResult	The evaluation result to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	LLMTestCase | ConversationalTestCase	The input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
**kwargs	Any	Additional configuration parameters.	{}

log_batch(evaluation_results, dataset_name, data, run_id=None, metadata=None, **kwargs) abstractmethod async

Log a batch of evaluation results (asynchronous).

Parameters:

Name	Type	Description	Default
evaluation_results	list[list[EvaluatorResult]]	Row-grouped evaluation results to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	list[LLMTestCase | ConversationalTestCase]	List of input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
**kwargs	Any	Additional configuration parameters.	{}

set_run_aggregators(aggregators)

Set run aggregators on the tracker.

Default is a no-op. Trackers that support run-level aggregation (e.g. CSV) should override this to wire the aggregators into their batch-summary logic.

Parameters:

Name	Type	Description	Default
aggregators	list[Any]	Run aggregator callables to apply after each batch.	required

CSVExperimentTracker(project_name, output_dir='./gllm_evals/experiments', score_key='score', extra_score_keys=None, leaderboard_score_key='aggregate_success', run_aggregators=None, companion_fields=None, include_eval_result=True)

Bases: BaseExperimentTracker

CSV file-based experiment tracker for development and testing.

This class provides a CSV local storage implementation for experiment tracking. It stores experiment data in CSV format with two files: experiment_results.csv and leaderboard.csv.

Attributes:

Name	Type	Description
project_name	str	The name of the project.
output_dir	Path	Directory to store experiment results.
experiment_results_file	Path	CSV file for experiment results.
leaderboard_file	Path	CSV file for leaderboard data.
logger	Logger	Logger for tracking errors and warnings.

Constants

MAX_OUTER_DICT_PARTS (int): Maximum number of parts in an outer dict score path. Used to distinguish main evaluator scores from nested sub-metrics.

Initialize CSV tracker with project name and output directory.

Parameters:

Name	Type	Description	Default
project_name	str	The name of the project.	required
output_dir	str	Directory to store experiment results.	'./gllm_evals/experiments'
score_key	str | list[str]	The key to extract scores from evaluation results. - If str: Direct key access (e.g., "score") or dot notation (e.g., "metrics.accuracy") - If list[str]: Nested key path (e.g., ["generation", "score"]) Defaults to "score".	'score'
extra_score_keys	list[str] | None	Additional score keys to extract for CSV columns only (not used for leaderboard primary selection). Defaults to DEFAULT_EXTRA_SCORE_KEYS.	None
leaderboard_score_key	str	The score key to use for leaderboard primary selection. Overrides score_key for leaderboard purposes. Set to "aggregate_success" to use the new binary metrics pass rate (default). Set to "score" for backward compatibility with per-metric scores. Defaults to "aggregate_success".	'aggregate_success'
run_aggregators	list[Any] | None	Custom run aggregators to compute batch-level statistics. Each callable receives (evaluation_results, data) and returns a dict of summary metrics. Defaults to None (empty list).	None
companion_fields	list[str] | None	Per-metric fields to extract alongside the primary score_key, preserving their native type (e.g. bool for "success"). Results in columns like generation.completeness.success. Defaults to DEFAULT_COMPANION_FIELDS.	None
include_eval_result	bool	When True, adds an eval_result column storing the full EvaluatorResult JSON per row (includes explanations, rubric scores, thresholds). Defaults to True.	True

alog(evaluation_result, dataset_name, data, run_id=None, metadata=None, **kwargs) async

Log a single evaluation result (asynchronous).

Parameters:

Name	Type	Description	Default
evaluation_result	EvaluatorResult	The evaluation result to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	LLMTestCase | ConversationalTestCase	The input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
**kwargs	Any	Additional configuration parameters (currently unused but accepted for interface consistency).	{}

get_experiment_history()

Get all experiment runs from leaderboard.

Returns:

Type	Description
list[dict[str, Any]]	list[dict[str, Any]]: List of experiment runs.

get_experiment_urls(run_id, dataset_name, **kwargs)

Get experiment local file paths for CSV experiment tracker.

Parameters:

Name	Type	Description	Default
run_id	str	ID of the experiment run.	required
dataset_name	str	Name of the dataset that was evaluated.	required
**kwargs	Any	Additional configuration parameters.	{}

Returns:

Name	Type	Description
ExperimentUris	ExperimentUris	Dictionary containing local file paths for accessing experiment data.

get_run_results(run_id)

Get detailed results for a specific run.

Parameters:

Name	Type	Description	Default
run_id	str	ID of the experiment run.	required

Returns:

Type	Description
list[dict[str, Any]]	list[dict[str, Any]]: Detailed run results.

log(evaluation_result, dataset_name, data, run_id=None, metadata=None, **kwargs)

Log a single evaluation result.

Parameters:

Name	Type	Description	Default
evaluation_result	EvaluatorResult	The evaluation result to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	LLMTestCase | ConversationalTestCase	The input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
**kwargs	Any	Additional configuration parameters (currently unused but accepted for interface consistency).	{}

log_batch(evaluation_results, dataset_name, data, run_id=None, metadata=None, **kwargs) async

Log a batch of evaluation results (asynchronous).

Parameters:

Name	Type	Description	Default
evaluation_results	list[list[EvaluatorResult]]	Row-grouped evaluation results to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	list[LLMTestCase | ConversationalTestCase]	List of input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
**kwargs	Any	Additional configuration parameters.	{}

set_run_aggregators(aggregators)

Set run aggregators on the tracker.

Parameters:

Name	Type	Description	Default
aggregators	list[Any]	Run aggregator callables to apply after each batch.	required

GoogleSheetsExperimentTracker(project_name, config, score_key='score', extra_score_keys=None, leaderboard_score_key='aggregate_success', run_aggregators=None, companion_fields=None, include_eval_result=True, batch_size=GSheetDefaultValues.DEFAULT_BATCH_SIZE, max_retries=GSheetDefaultValues.MAX_RETRIES, keep_fields=None, field_mapping=None)

Bases: BaseExperimentTracker

Google Sheets experiment tracker for evaluation results.

Initialize a Google Sheets–backed experiment tracker instance.

This constructor wires the tracker into the Google Sheets–based experiment tracking workflow defined by :class:BaseExperimentTracker.

Parameters:

Name	Type	Description	Default
project_name	str	Logical name of the project or benchmark suite.	required
config	GoogleSheetsTrackerConfig	GoogleSheetsTrackerConfig containing Google Sheets resource/auth settings.	required
score_key	str | list[str]	Metric key(s) used to extract scores from evaluation outputs.	'score'
extra_score_keys	list[str] | None	Additional score keys to extract for result columns. Defaults to DEFAULT_EXTRA_SCORE_KEYS.	None
leaderboard_score_key	str	Score key used for leaderboard aggregation.	'aggregate_success'
run_aggregators	list[Any] | None	Custom run aggregators for batch-level metrics.	None
companion_fields	list[str] | None	Per-metric fields to extract alongside scores. Defaults to DEFAULT_COMPANION_FIELDS.	None
include_eval_result	bool	Whether to include the full evaluator result JSON. Defaults to True.	True
batch_size	int	Maximum number of rows per batch write to Google Sheets.	DEFAULT_BATCH_SIZE
max_retries	int	Maximum retry attempts for transient API failures.	MAX_RETRIES
keep_fields	list[str] | None	Optional list of fields to keep when logging.	None
field_mapping	dict[str, str] | None	Optional mapping of output field name -> source field name.	None

Raises:

Type	Description
ValueError	If config validation fails.

alog(evaluation_result, dataset_name, data, run_id=None, metadata=None, update_leaderboard=True, **kwargs) async

Log a single evaluation result (asynchronous).

Note

gspread is synchronous, so this offloads the blocking write to the running loop's default ThreadPoolExecutor via run_in_executor(None, ...). Concurrent alog calls share that pool; with the CPython default (min(32, os.cpu_count() + 4) workers) sustained high concurrency can saturate it and block unrelated tasks. For high-throughput cases prefer log_batch (one executor call per batch) or configure a dedicated executor on the loop via loop.set_default_executor.

Parameters:

Name	Type	Description	Default
evaluation_result	EvaluatorResult	The evaluation result to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	LLMTestCase | ConversationalTestCase	The input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
update_leaderboard	bool	Whether to update the leaderboard after writing this row.	True
**kwargs	Any	Additional configuration parameters.	{}

get_experiment_urls(run_id, dataset_name, **kwargs)

Get experiment URLs and paths for accessing experiment data.

Parameters:

Name	Type	Description	Default
run_id	str	ID of the experiment run.	required
dataset_name	str	Name of the dataset that was evaluated.	required
**kwargs	Any	Additional configuration parameters.	{}

Returns:

Name	Type	Description
ExperimentUris	ExperimentUris	Dictionary containing URIs for accessing experiment data.

log(evaluation_result, dataset_name, data, run_id=None, metadata=None, update_leaderboard=True, **kwargs)

Log a single evaluation result (synchronous).

Parameters:

Name	Type	Description	Default
evaluation_result	EvaluatorResult	The evaluation result to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	LLMTestCase | ConversationalTestCase	The input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
update_leaderboard	bool	Whether to update the leaderboard after writing this row. Set to False when calling log repeatedly and update the leaderboard once after the batch.	True
**kwargs	Any	Additional configuration parameters.	{}

log_batch(evaluation_results, dataset_name, data, run_id=None, metadata=None, update_leaderboard=True, **kwargs) async

Log a batch of evaluation results (asynchronous).

Parameters:

Name	Type	Description	Default
evaluation_results	list[list[EvaluatorResult]]	Row-grouped evaluation results to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	list[LLMTestCase | ConversationalTestCase]	List of input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
update_leaderboard	bool	Whether to update the leaderboard once after writing the batch.	True
**kwargs	Any	Additional configuration parameters.	{}

set_run_aggregators(aggregators)

Set run aggregators on the tracker.

Parameters:

Name	Type	Description	Default
aggregators	list[Any]	Run aggregator callables to apply after each batch.	required

JSONExperimentTracker(project_name, *, output_dir='./gllm_evals/experiments', leaderboard_score_key='score', **kwargs)

Bases: BaseExperimentTracker

JSON file-based experiment tracker that returns accumulated rows to the caller.

This class provides a JSON local storage implementation for experiment tracking. It stores experiment data in two files: one .json per run under experiment_results/, and a cross-run leaderboard.json.

Attributes:

Name	Type	Description
project_name	str	The name of the project.
output_dir	Path	Directory leaderboard.json and experiment_results/ live under.
experiment_results_dir	Path	output_dir / "experiment_results" -- where .json files are written.
leaderboard_file	Path	output_dir / "leaderboard.json" -- the cross-run leaderboard file.
leaderboard_score_key	str | list[str]	Score extraction key, read internally by self._get_evaluator_score() (aggregate_success is used directly at the default "score", matching CSVExperimentTracker's own leaderboard_score_key default; self._get_score() is used only when this is customized).

Initialize the JSON tracker with project name and output directory.

Parameters:

Name	Type	Description	Default
project_name	str	The name of the project.	required
output_dir	str | None	Directory leaderboard.json and experiment_results/ live under. Defaults to "./gllm_evals/experiments". An explicit None resolves to the same default.	'./gllm_evals/experiments'
leaderboard_score_key	str | list[str]	The key to extract scores from evaluation results. Defaults to "score".	'score'
**kwargs	Any	Additional arguments passed to the base tracker.	{}

alog(evaluation_result, dataset_name, data, run_id=None, metadata=None, **kwargs) async

Log a single evaluation result (asynchronous).

Parameters:

Name	Type	Description	Default
evaluation_result	EvaluatorResult	The evaluation result to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	LLMTestCase | ConversationalTestCase	The input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
**kwargs	Any	Additional configuration parameters (currently unused but accepted for interface consistency).	{}

get_experiment_history(**kwargs)

Get all experiment runs from leaderboard.json.

Parameters:

Name	Type	Description	Default
**kwargs	Any	Unused.	{}

Returns:

Type	Description
list[dict[str, Any]]	list[dict[str, Any]]: List of experiment runs, most recently updated first. Rows written before this tracker recorded timestamps sort last, not first.

get_experiment_urls(run_id, dataset_name, **kwargs)

Get experiment local file paths for JSON experiment tracker.

Parameters:

Name	Type	Description	Default
run_id	str	ID of the experiment run.	required
dataset_name	str	Name of the dataset that was evaluated.	required
**kwargs	Any	Additional configuration parameters.	{}

Returns:

Name	Type	Description
ExperimentUris	ExperimentUris	Dictionary containing local file paths for accessing experiment data.

get_run_results(run_id, **kwargs)

Get detailed results for a specific run.

average_score blends every evaluator's own score across every row into one number -- CSVExperimentTracker has no equivalent, since its leaderboard never blends across evaluators (see _calculate_scores_by_evaluator() for the per-evaluator numbers that do match CSV's). It's None, not a false 0.0, if entries exist but none could be scored.

Parameters:

Name	Type	Description	Default
run_id	str	ID of the experiment run.	required
**kwargs	Any	Unused.	{}

Returns:

Type	Description
list[dict[str, Any]]	list[dict[str, Any]]: Detailed run results, in the order they were logged.

log(evaluation_result, dataset_name, data, run_id=None, metadata=None, **kwargs)

Log a single evaluation result.

Parameters:

Name	Type	Description	Default
evaluation_result	EvaluatorResult	The evaluation result to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	LLMTestCase | ConversationalTestCase	The input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
**kwargs	Any	Additional configuration parameters (currently unused but accepted for interface consistency).	{}

Raises:

Type	Description
NotImplementedError	If data is a ConversationalTestCase.

log_batch(evaluation_results, dataset_name, data, run_id=None, metadata=None, **kwargs) async

Log a batch of evaluation results (asynchronous).

Parameters:

Name	Type	Description	Default
evaluation_results	list[list[EvaluatorResult]]	Row-grouped evaluation results to log.	required
dataset_name	str	Name of the dataset being evaluated.	required
data	list[LLMTestCase | ConversationalTestCase]	List of input data that was evaluated.	required
run_id	str | None	ID of the experiment run. Can be auto-generated if None.	None
metadata	dict[str, Any] | None	Additional metadata to log.	None
**kwargs	Any	Additional configuration parameters.	{}

Raises:

Type	Description
EvalResultContractError	If evaluation_results and data have different lengths.
EvalUnsupportedOperationError	If any row in data is a ConversationalTestCase.

set_run_aggregators(aggregators)

Set run aggregators on the tracker.

Parameters:

Name	Type	Description	Default
aggregators	list[Any]	Run aggregator callables to apply after each batch.	required

get_experiment_tracker(experiment_tracker, project_name, **kwargs)

Get an experiment tracker instance (create or return existing).

Supported experiment tracker types

• CSVExperimentTracker (Default - Local CSV)
• JSONExperimentTracker (Local JSON, returns accumulated rows to the caller)

Parameters:

Name	Type	Description	Default
experiment_tracker	BaseExperimentTracker | type[BaseExperimentTracker]	The experiment tracker to get.	required
project_name	str	The name of the project.	required
**kwargs	Any	Additional arguments to pass to the constructor.	{}

Returns:

Name	Type	Description
BaseExperimentTracker	BaseExperimentTracker	The experiment tracker.

Raises:

Type	Description
EvalConfigurationError	If the experiment tracker is not supported.