Evaluator

class Evaluator(groundtruth: Dataset, name: str | None = None, **predictions: Dataset)

Bases: object

Abstract class of Evaluator, made to measure prediction quality with respect to a Dataset of groundtruth annotations. Depending on data type, the method used for evaluation might differ, refer to the specialized classes for information. The fundamental building block is the Dataset object representing the groundtruth. additional kwargs given to the constructor are also Dataset objects that must match the groundtruth, in terms of image and label maps (if any)

Constructor of the Evaluator object.

Parameters:

• groundtruth – Dataset object representing the ground truth with annotations, image data and label_map

• name – Name of Evaluator. If set to None, will be deduced from groundtruth’s dataset name

• **predictions – keyword arguments for additional datasets to compare the groundtruth to. Its images must match the groundtruth dataset (see add_prediction_dataset method below).

Attributes

name: str | None

Name of Evaluator. Can be deduced from groundtruth’s dataset name and will be used in export functions like DetectionEvaluator.to_fiftyone()

groundtruth: DataFrame

DataFrame comprising annotation data. Must have at least image_id column

predictions_dictionary: dict[str, DataFrame]

dictionary of DataFrames comprising prediction data. Must have at least image_id and confidence columns

images: DataFrame

DataFrame comprising image data. This dataframe should be referred to by both gt and predictions with the image_id column

images_root: Path

Root folder where to grab images. Image filepath will be concatenation of images_root and their relative path

label_map: dict[int, str]

Mapping from category_id to category_str. If used, is generally taken from the groundtruth Dataset. The prediction must be compatible with it

Methods

add_predictions(predictions_name, predictions)	Method to add predictions to the Evaluator from a dataframe.
add_predictions_dataset(predictions_name, ...)	Method to add predictions to the Evaluator from a Dataset object.
from_parquet(input_dir)	Class method to construct an instance of this class or a subclass.
get_annotations_attributes([predictions_name])	Get the name of columns related to annotations attributes.
get_image_attributes()	Get the name of columns related to image attributes.
to_parquet(output_dir[, overwrite])	Save the current object to a folder containing parquet files for dataframes inside this object, and a metadata.yaml file for other attributes.