module: dynamic

class annotlib.dynamic.DynamicAnnot(annotator_model, y_unique, learning_rates=None, adversarial=False, random_state=None)

Bases: annotlib.base.BaseAnnot

An instance of this class emulates annotators with a dynamic labelling performance. Therefore, it requires learning rates which describe the progress of the labelling performance. Such a learning rate is either positive or negative. If it is positive, the corresponding annotator’s labelling performance improves during the labelling process. In contrast, a negative learning rate results in a decreasing labelling performance.

A very important remark is the fact that the labelling performance of adversarial and non-adversarial annotators is oppositional. A good labelling performance implies a high labelling accuracy for a non-adversarial annotator, whereas a good labelling performance of an adversarial annotator implies a low labelling accuracy. There is a option which defines whether an annotator is allowed to be adversarial. To realise the development of the labelling performance, the predicted label of an annotator is flipped with a probability depending on the state of the labelling progress which is represented by the number of queries.

The flip probability is computed by \(p_{flip}(\mu_i, q_i) = min(|\mu_i| \cdot q_i, 1)\), where \(\mu\) is the learning rate of an annotator \(a_i\) and \(q_i\) is the number of queries processed by the annotator \(a_i\).

Parameters:

annotator_model: BaseAnnotators

An object of the type Annotators.

y_unique: array-like, shape (n_classes)

The array of available class labels.

learning_rates: array-like, shape (n_annotators)

A learning rate for each annotator. The default learning rate of an annotator is sampled from a uniform distribution \(U(-0.001,0.001)\).

adversarial: boolean | array or list of booleans, shape (n_annotators)

Flag, whether adversarial annotators are allowed. By default, this parameter is false, so that the non-adversarial annotators tend to make random guesses.

random_state: None | int | instance of numpy.Random.RandomState

The random_state is applied for generating the default learning rates and for flipping class labels.

Examples

>>> from sklearn.datasets import load_iris
>>> from annotlib import ClassifierBasedAnnot
>>> # load iris data set
>>> X, y_true = load_iris(return_X_y=True)
>>> y_unique = np.unique(y_true)
>>> # simulate three annotators on the iris data set
>>> annotator_model = ClassifierBasedAnnot(X=X, y_true=y_true, n_annotators=3)
>>> # make the simulated annotators dynamic with default learning rates
>>> dynamic_annotators = DynamicAnnot(annotator_model=annotator_model, y_unique=y_unique)
>>> # query class labels of 2 samples from 3 annotators
>>> dynamic_annotators.class_labels(X=X[0:2], y_true=y_true[0:2]).shape
(2, 3)
>>> # check query values
>>> dynamic_annotators.n_queries()
array([1, 1, 1])

Attributes:

annotator_model_: BaseAnnotators

An object of the type Annotators.

y_unique_: array-like, shape (n_classes)

The array of available class labels.

learning_rates_: numpy.ndarray, shape (n_annotators)

A learning rate for each annotator. The default learning rate of an annotator is sampled from a uniform distribution \(U(-0.001,0.001)\).

adversarial_: numpy.ndarray, shape (n_annotators)

Flags specifying which annotators are allowed to adversarial. By default, this parameter is false for each annotator, so that the non-adversarial annotators makes only random guesses for originally wrong decisions.

random_state_: None | int | instance of numpy.Random.RandomState

The random_state is applied for generating the default learning rates and for flipping class labels.

class_labels(self, X, annotator_ids=None, query_value=1, **kwargs)

Method returning the class labels of the given samples.

Parameters:

X: array-like, shape (n_samples, n_features)

Samples whose class labels are queried.

y_true: array-like, shape (n_samples)

The true class label of each given sample.

annotator_ids: array-like, shape (n_queried_annotators)

The indices of the annotators whose class labels are queried.

query_value: int

The query value represents the increment of the query statistics of the queried annotators.

Returns:

Y: numpy.ndarray, shape (n_samples, n_annotators)

Class labels of the given samples which were provided by the queried annotators. The non queried annotators return np.nan values.

confidence_scores(self, X, annotator_ids=None, **kwargs)

Method returning the confidence scores for labelling the given samples.

Parameters:

X: array-like, shape (n_samples, n_features)

Samples whose class labels are queried.

annotator_ids: array-like, shape (n_queried_annotators)

The indices of the annotators whose confidence scores are queried.

Returns:

C: numpy.ndarray, shape (n_samples, n_annotators)

Confidence scores of the queried annotators for labelling the given samples. The non queried annotators should return np.nan values.

n_annotators(self)

Method for computing the number of annotators.

Returns:

n_annotators_: int

Number of BaseAnnot.

n_queries(self)

Method for computing the number of queries posed to an annotator.

Returns:

n_queries_: numpy.ndarray, shape (n_annotators)

An entry n_queries_[a] indicates how many queries annotator a has processed.

queried_samples(self)

Abstract method for returning the samples for which the annotators were queried to provide class labels.

Returns:

X_queried_: numpy.ndarray, shape (n_annotators, n_samples, n_features)

An entry X_queried_[a] represents the samples for which the annotator a was queried to provide class labels.