skada.metrics.MaNoScorer

class skada.metrics.MaNoScorer(p=4, threshold=5, greater_is_better=True)[source]

MaNo scorer inspired by [37], an approach for unsupervised accuracy estimation.

This scorer used the model's predictions on target data to estimate the accuracy of the model. The original implementation in [37] is tailored to neural networks and consist of three steps: 1) Recover target logits (inference step), 2) Normalize them as probabilities (e.g., with softmax), 3) Aggregate by averaging the p-norms of the target normalized logits.

To ensure compatibility with any estimator, we adapt the original implementation. If the estimator is a neural network, follow 1) --> 2) --> 3) like in [37]. Else, directly use the probabilities predicted by the estimator and then do 3).

See [37] for details.

Parameters:
pint, default=4

Order for the p-norm normalization. It must be non-negative.

thresholdint, default=5

Threshold value to determine which normalization to use. If threshold <= 0, softmax normalization is always used. See Eq.(6) of [37] for more details.

greater_is_betterbool, default=True

Whether higher scores are better.

Returns:
scorefloat in [0, 1] (or [-1, 0] depending on the value of self._sign).

References

[37] (1,2,3,4,5)

Renchunzi Xie et al. MaNo: Matrix Norm for Unsupervised Accuracy Estimation under Distribution Shifts. In NeurIPS, 2024.

set_score_request(*, sample_domain: bool | None | str = '$UNCHANGED$') → MaNoScorer

Request metadata passed to the score method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters:
sample_domainstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_domain parameter in score.

Returns:
selfobject

The updated object.