skada.metrics.MaNoScorer

class skada.metrics.MaNoScorer(p=4, threshold=5, greater_is_better=True)[source]

MaNo scorer inspired by [37], an approach for unsupervised accuracy estimation.

This scorer used the model's predictions on target data to estimate the accuracy of the model. The original implementation in [37] is tailored to neural networks and consist of three steps: 1) Recover target logits (inference step), 2) Normalize them as probabilities (e.g., with softmax), 3) Aggregate by averaging the p-norms of the target normalized logits.

To ensure compatibility with any estimator, we adapt the original implementation. If the estimator is a neural network, follow 1) --> 2) --> 3) like in [37]. Else, directly use the probabilities predicted by the estimator and then do 3).

See [37] for details.

Parameters:
pint, default=4

Order for the p-norm normalization. It must be non-negative.

thresholdint, default=5

Threshold value to determine which normalization to use. If threshold <= 0, softmax normalization is always used. See Eq.(6) of [37] for more details.

greater_is_betterbool, default=True

Whether higher scores are better.

Returns:
scorefloat in [0, 1] (or [-1, 0] depending on the value of self._sign).

References

[37] (1,2,3,4,5)

Renchunzi Xie et al. MaNo: Matrix Norm for Unsupervised Accuracy Estimation under Distribution Shifts. In NeurIPS, 2024.

set_score_request(*, sample_domain: bool | None | str = '$UNCHANGED$') → MaNoScorer

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:
sample_domainstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_domain parameter in score.

Returns:
selfobject

The updated object.