Skip to main content

Scoring Metrics

To host a competition or set a goal while working with a dataset, it can be helpful to have a clear scoring metric as a north star. Multiple scoring mechanisms can be combined, and in most cases, we aim to use some proxy for accuracy with a loss based on the privacy budget spent to achieve it. Below are the scoring metrics currently supported on the platform:

Metrics of Accuracy

All the metrics of accuracy involve an aggregation, either a sum or a mean. During a competition, these metrics are calculated using a differentially private measurement to ensure that information is not unintentionally leaked based on the leaderboard, serving as a feedback loop to the user. More details are available here.

Below are the current metrics of accuracy. More metrics will be added as required for competitions.


Assuming yy represents the predictions you have made and y^\hat{y} represents the true outputs, both indexed by ii in (1,n)(1, n), L2-loss is defined as:

L2=1ni(yiy^i)2.L_2 = \frac{1}{n} \sum_i (y_i - \hat{y}_i)^2.

The L2L_2-loss is typically used in regression problems.


Similar to L2L_2-loss, assuming yy represents the predictions you have made and y^\hat{y} represents the true outputs, both indexed by ii in (1,n)(1, n), the L1L_1-loss is defined as:

L1=1niyiy^i.L_1 = \frac{1}{n} \sum_i |y_i - \hat{y}_i|.

The L1L_1-loss is typically used in regression problems.

Classification Accuracy

Classification accuracy is simply the number of correctly labelled samples divided by the total number of samples (nn):

acc=1n(yi==y^i).\text{acc} = \frac{1}{n} (y_i == \hat{y}_i).

This is only applicable for classification problems.

Metrics of Privacy Loss

The performance metrics in Antigranular also take into account the privacy budget used by the user to achieve a particular accuracy metric. As the focus is on (ϵδ)(\epsilon - \delta) differential privacy, both of these two parameters of privacy loss can be considered. For more of differential privacy please refer to here.

Linear Epsilon Loss

This is the most straightforward loss, simply calculated as the product of the utilised epsilon and a weight:

Lϵ,w=wϵ.L_{\epsilon, w} = w \epsilon.

Soft Threshold Delta Loss

There are many guidelines for determining the safe amount of δ\delta to be used in differential privacy. Typically, it is suggested to keep it below 1n\frac{1}{n}. The Soft Threshold Delta Loss acts as an smoothened step function, heavily penalising the user when δ>1n\delta > \frac{1}{n} but having little effect otherwise:

Lδ,α,β=(1expαδn)β,L_{\delta, \alpha, \beta} = \left( 1 - \exp^{- \alpha \delta n} \right)^\beta,

where α\alpha and β\beta are scale parameters.