compute_count_error#

CrowdDetectionEvaluator.compute_count_error(groups: str | ContinuousGroup | Sequence[str | ContinuousGroup] = 'category_id', quantiles: Iterable[float] = (0.25, 0.5, 0.75), confidence_index: Iterable[float] | None = None) → tuple[DataFrame, DataFrame][source]#

Compute Count error metrics, both absolute (in number of objects found) and relative (with respect to groundtruth number of objects) with respect to confidence threshold.

Along with these metrics, it computes standard deviation of absolute/relative error and quantiles for error values.

See also

related tutorial

Computed metrics:

  • Mean Absolute Error (MAE)

  • Root of Mean Square Error (RMSE)

  • Mean Relative Error (MRE)

  • Root of Mean Square Relative Error (RMSRE)

Parameters:

  • groups – Groups of image or annotation attributes to use to partition evaluation results to compute multiple PR curves. Must be a group_list . Defaults to “category_id”.

  • quantiles – quantile values to get with respect to confidence threshold, aggregated per image. Must contain the median value (i.e. 0.5). Defaults to (0.25, 0.5, 0.75).

  • confidence_index – sequence of confidence thresholds to compute the metric on. If set to None, will be 101 equidistant points, from 0 to 1. Defaults to None.

Returns:

A pair of DataFrames.

  • a DataFrame with computed metrics, with multiindex columns, for absolute and relative metrics with respect to confidence

  • a DataFrame with detailed error values with respect to confidence for each image, in order to compute statistics manually.

Example

>>> from lours.utils.doc_utils import dummy_dataset
>>> groundtruth = dummy_dataset(
...     10, 1000, label_map={0: "person", 1: "car"}, keypoints_share=1
... )
>>> predictions = dummy_dataset(
...     10,
...     10000,
...     label_map=groundtruth.label_map,
...     images=groundtruth.images,
...     keypoints_share=1,
...     add_confidence=True,
... )
>>> evaluator = CrowdDetectionEvaluator(
...     groundtruth=groundtruth, predictions=predictions
... )
>>> errors, detailed = evaluator.compute_count_error()
>>> errors
                       absolute              ...  relative
                            MAE        RMSE  ...     q0.75        model
category_id confidence                       ...
0           0.00          452.0  452.378824  ...  9.913239  predictions
            0.01          447.8  448.170057  ...  9.779314  predictions
            0.02          442.3  442.670645  ...  9.655792  predictions
            0.03          437.6  437.945887  ...  9.569740  predictions
            0.04          433.5  433.856774  ...  9.412411  predictions
...                         ...         ...  ...       ...          ...
1           0.96           34.2   35.883144  ... -0.540094  predictions
            0.97           38.0   39.549968  ... -0.630391  predictions
            0.98           42.6   43.395852  ... -0.768797  predictions
            0.99           46.3   46.764303  ... -0.911782  predictions
            1.00           50.7   51.073476  ... -1.000000  predictions

[202 rows x 14 columns]

Get the confidence threshold where the Mean Average Error is the lowest, and show the corresponding rows (one per category).

>>> mae = errors[("absolute", "MAE")]
>>> mae
category_id  confidence
0            0.00          452.0
             0.01          447.8
             0.02          442.3
             0.03          437.6
             0.04          433.5
                           ...
1            0.96           34.2
             0.97           38.0
             0.98           42.6
             0.99           46.3
             1.00           50.7
Name: (absolute, MAE), Length: 202, dtype: float64
>>> best_mae = errors.loc[mae.groupby(level=0).idxmin()]
>>> best_mae
                       absolute            ...  relative
                            MAE      RMSE  ...     q0.75        model
category_id confidence                     ...
0           0.89            5.4   7.655064  ...  0.116153  predictions
1           0.88           10.9  13.939153  ...  0.310000  predictions

[2 rows x 14 columns]
>>> best_mae.reset_index().iloc[0]
category_id                     0
confidence                   0.89
absolute     MAE              5.4
             RMSE        7.655064
             std         7.788881
             q0.25            0.0
             q0.50            2.5
             q0.75            5.5
             model    predictions
relative     MRE          0.10748
             RMSRE       0.145579
             std         0.145805
             q0.25            0.0
             q0.50       0.055717
             q0.75       0.116153
             model    predictions
Name: 0, dtype: object