Metrics¶

Base metric¶

class argus.metrics.Metric[source]¶

Base metric class.

One needs to create a class inherited from the Metric class, to define a custom metric. In the basic use case scenarios, the following should be done:

Override three methods: reset, update, and compute.
Set class attribute: name, better.

name¶

Unique metric name. The name is used to reference the metric by other components, like Callbacks. Defaults to ‘’.

Type: str

better¶

Minimization or maximization is better. Should be ‘min’ or ‘max’. It will be used, for example, by argus.callbacks.MonitorCheckpoint. Defaults to ‘min’.

Type: str

Example

MAP@k implementation:

import torch
import numpy as np
from argus.metrics import Metric


def apk(actual, predicted, k=3):
    if len(predicted) > k:
        predicted = predicted[:k]

    score = 0.0
    num_hits = 0.0

    for i, p in enumerate(predicted):
        if p in actual and p not in predicted[:i]:
            num_hits += 1.0
            score += num_hits / (i+1.0)

    if not actual:
        return 0.0

    return score / min(len(actual), k)


def mapk(actual, predicted, k=3):
    return np.mean([apk(a, p, k) for a, p in zip(actual, predicted)])


class MAPatK(Metric):
    name = 'map_at_k'
    better = 'max'

    def __init__(self, k=3):
        super().__init__()
        self.k = k
        self.scores = []

    def reset(self):
        self.scores = []

    def update(self, step_output: dict):
        preds = step_output['prediction'].cpu().numpy()
        trgs = step_output['target'].cpu().numpy()

        preds_idx = preds.argsort(axis=1)
        preds_idx = np.fliplr(preds_idx)[:, :self.k]

        self.scores += [apk([a], p, self.k) for a, p in zip(trgs, preds_idx)]

    def compute(self):
        return np.mean(self.scores)

Then you can use the metric like this:

callbacks = [
    MonitorCheckpoint(dir_path='mnist', monitor='val_map_at_k')
]

model.fit(train_loader,
          val_loader=val_loader,
          metrics=['map_at_k'],  # or the same: metrics=[MAPatK(k=3)]
          callbacks=callbacks)

In the case of name-based custom metric reference, it is enough to define or import the metric class in the module to use it. Note that the metric values saved into argus.engine.State are prepended with val_ or train_, so, the full metric name, like val_map_at_k in the example, should be used to retrieve the metric value, for instance, as a value to monitor by argus.callbacks.MonitorCheckpoint

compute()[source]¶: Compute the custom metric and return the result.

epoch_complete(state: argus.engine.engine.State)[source]¶: Store metric value to argus.engine.State. You can override this method if you want, for example, to save several metrics values in the state.

reset()[source]¶: Init or reset internal variables and accumulators.

update(step_output: dict)[source]¶

Update internal variables with a provided step_output.

step_output from default argus.model.Model.train_step() and argus.model.Model.val_step() looks like:

{
    'prediction': The batch predictions,
    'target': The batch targets,
    'loss': Loss function value
}

Metrics¶

class argus.metrics.loss.Loss[source]¶: Accumulates the average of a loss function. By default used by argus.model.Model.fit().

class argus.metrics.CategoricalAccuracy[source]¶: Calculates the accuracy for multiclass classification.

Logging

Engine