Pytrec Utils

Utility functions for pytrec_eval.

create_trec_inputs(*, ground_truth_chunk_ids, retrieved_chunks)

Creates the inputs for pytrec_eval.

This function is used to create the inputs for pytrec_eval. there are two inputs: - qrel (dict[str, dict[str, int]]): the ground truth of the retrieved chunks. There are two possible values: - 1: the chunk is relevant to the query. - 0: the chunk is not relevant to the query. - results (dict[str, dict[str, float]]): the retrieved chunks with their similarity score.

Example:

qrel = {
    "q1": {"chunk1": 1, "chunk2": 1},
}
results = {"q1": {"chunk1": 0.9, "chunk2": 0.8, "chunk3": 0.7}}

Parameters:

Name	Type	Description	Default
ground_truth_chunk_ids	list[str]	The ground truth chunk IDs.	required
retrieved_chunks	dict[str, float]	The retrieved chunk ids with their similarity score.	required

Returns:

Type	Description
tuple[dict[str, dict[str, int]], dict[str, dict[str, float]]]	tuple[dict[str, dict[str, int]], dict[str, dict[str, float]]]: The inputs for pytrec_eval.