Table Of Contents

Previous topic

mvpa.clfs.svm

Next topic

mvpa.clfs.warehouse

This Page

Quick search

mvpa.clfs.transerror

Utility class to compute the transfer error of classifiers.

The comprehensive API documentation for this module, including all technical details, is available in the Epydoc-generated API reference for mvpa.clfs.transerror (for developers).

Classes

ClassifierError

class mvpa.clfs.transerror.ClassifierError(clf, labels=None, train=True, **kwargs)

Bases: mvpa.misc.state.ClassWithCollections

Compute (or return) some error of a (trained) classifier on a dataset.

Initialization.

Parameters:
  • clf (Classifier) – Either trained or untrained classifier
  • labels (list) – if provided, should be a set of labels to add on top of the ones present in testdata
  • train (bool) – unless train=False, classifier gets trained if trainingdata provided to __call__
clf
labels

See also

Derived classes might provide additional methods via their base classes. Please refer to the list of base classes (if it exists) at the begining of the ClassifierError documentation.

Full API documentation of ClassifierError in module mvpa.clfs.transerror.

ConfusionBasedError

class mvpa.clfs.transerror.ConfusionBasedError(clf, labels=None, confusion_state='training_confusion', **kwargs)

Bases: mvpa.clfs.transerror.ClassifierError

For a given classifier report an error based on internally computed error measure (given by some ConfusionMatrix stored in some state variable of Classifier).

This way we can perform feature selection taking as the error criterion either learning error, or transfer to splits error in the case of SplitClassifier

Initialization.

Parameters:
  • clf (Classifier) – Either trained or untrained classifier
  • confusion_state – Id of the state variable which stores ConfusionMatrix
  • labels (list) – if provided, should be a set of labels to add on top of the ones present in testdata

See also

Derived classes might provide additional methods via their base classes. Please refer to the list of base classes (if it exists) at the begining of the ConfusionBasedError documentation.

Full API documentation of ConfusionBasedError in module mvpa.clfs.transerror.

ConfusionMatrix

class mvpa.clfs.transerror.ConfusionMatrix(labels=None, labels_map=None, **kwargs)

Bases: mvpa.clfs.transerror.SummaryStatistics

Class to contain information and display confusion matrix.

Implementation of the SummaryStatistics in the case of classification problem. Actual computation of confusion matrix is delayed until all data is acquired (to figure out complete set of labels). If testing data doesn’t have a complete set of labels, but you like to include all labels, provide them as a parameter to the constructor.

Confusion matrix provides a set of performance statistics (use asstring(description=True) for the description of abbreviations), as well ROC curve (http://en.wikipedia.org/wiki/ROC_curve) plotting and analysis (AUC) in the limited set of problems: binary, multiclass 1-vs-all.

Initialize ConfusionMatrix with optional list of labels

Parameters:
  • labels (list) – Optional set of labels to include in the matrix
  • labels_map (None or dict) – Dictionary from original dataset to show mapping into numerical labels
  • targets – Optional set of targets
  • predictions – Optional set of predictions
asstring(short=False, header=True, summary=True, description=False)

‘Pretty print’ the matrix

Parameters:
  • short (bool) – if True, ignores the rest of the parameters and provides consise 1 line summary
  • header (bool) – print header of the table
  • summary (bool) – print summary (accuracy)
  • description (bool) – print verbose description of presented statistics
error
getLabels_map()
labels
labels_map
matrices
Return a list of separate confusion matrix per each stored set
matrix
percentCorrect
plot(labels=None, numbers=False, origin='upper', numbers_alpha=None, xlabels_vertical=True, numbers_kwargs={}, **kwargs)

Provide presentation of confusion matrix in image

Parameters:
  • labels (list of int or basestring) – Optionally provided labels guarantee the order of presentation. Also value of None places empty column/row, thus provides visual groupping of labels (Thanks Ingo)
  • numbers (bool) – Place values inside of confusion matrix elements
  • numbers_alpha (None or float) – Controls textual output of numbers. If None – all numbers are plotted in the same intensity. If some float – it controls alpha level – higher value would give higher contrast. (good value is 2)
  • origin (basestring) – Which left corner diagonal should start
  • xlabels_vertical (bool) – Either to plot xlabels vertical (benefitial if number of labels is large)
  • numbers_kwargs (dict) – Additional keyword parameters to be added to numbers (if numbers is True)
  • **kwargs – Additional arguments given to imshow (eg me cmap)
Return type:

(fig, im, cb) – figure, imshow, colorbar

setLabels_map(val)

See also

Derived classes might provide additional methods via their base classes. Please refer to the list of base classes (if it exists) at the begining of the ConfusionMatrix documentation.

Full API documentation of ConfusionMatrix in module mvpa.clfs.transerror.

ROCCurve

class mvpa.clfs.transerror.ROCCurve(labels, sets=None)

Bases: object

Generic class for ROC curve computation and plotting

Parameters:
  • labels (list) – labels which were used (in order of values if multiclass, or 1 per class for binary problems (e.g. in SMLR))
  • sets (list of tuples) – list of sets for the analysis
ROCs
aucs
Compute and return set of AUC values 1 per label
plot(label_index=0)
TODO: make it friendly to labels given by values?
should we also treat labels_map?

See also

Derived classes might provide additional methods via their base classes. Please refer to the list of base classes (if it exists) at the begining of the ROCCurve documentation.

Full API documentation of ROCCurve in module mvpa.clfs.transerror.

RegressionStatistics

class mvpa.clfs.transerror.RegressionStatistics(**kwargs)

Bases: mvpa.clfs.transerror.SummaryStatistics

Class to contain information and display on regression results.

Initialize RegressionStatistics

Parameters:
  • targets – Optional set of targets
  • predictions – Optional set of predictions
asstring(short=False, header=True, summary=True, description=False)
‘Pretty print’ the statistics
error

See also

Derived classes might provide additional methods via their base classes. Please refer to the list of base classes (if it exists) at the begining of the RegressionStatistics documentation.

Full API documentation of RegressionStatistics in module mvpa.clfs.transerror.

SummaryStatistics

class mvpa.clfs.transerror.SummaryStatistics(targets=None, predictions=None, values=None, sets=None)

Bases: object

Basic class to collect targets/predictions and report summary statistics

It takes care about collecting the sets, which are just tuples (targets, predictions, values). While ‘computing’ the matrix, all sets are considered together. Children of the class are responsible for computation and display.

Initialize SummaryStatistics

targets or predictions cannot be provided alone (ie targets without predictions)

Parameters:
  • targets – Optional set of targets
  • predictions – Optional set of predictions
  • values – Optional set of values (which served for prediction)
  • sets – Optional list of sets
add(targets, predictions, values=None)
Add new results to the set of known results
asstring(short=False, header=True, summary=True, description=False)

‘Pretty print’ the matrix

Parameters:
  • short (bool) – if True, ignores the rest of the parameters and provides consise 1 line summary
  • header (bool) – print header of the table
  • summary (bool) – print summary (accuracy)
  • description (bool) – print verbose description of presented statistics
compute()
Actually compute the confusion matrix based on all the sets
error
reset()
Cleans summary – all data/sets are wiped out
sets
stats
summaries
Return a list of separate summaries per each stored set

See also

Derived classes might provide additional methods via their base classes. Please refer to the list of base classes (if it exists) at the begining of the SummaryStatistics documentation.

Full API documentation of SummaryStatistics in module mvpa.clfs.transerror.

TransferError

class mvpa.clfs.transerror.TransferError(clf, errorfx=MeanMismatchErrorFx(), labels=None, null_dist=None, **kwargs)

Bases: mvpa.clfs.transerror.ClassifierError

Compute the transfer error of a (trained) classifier on a dataset.

The actual error value is computed using a customizable error function. Optionally the classifier can be trained by passing an additional training dataset to the __call__() method.

Initialization.

Parameters:
  • clf (Classifier) – Either trained or untrained classifier
  • errorfx – Functor that computes a scalar error value from the vectors of desired and predicted values (e.g. subclass of ErrorFunction)
  • labels (list) – if provided, should be a set of labels to add on top of the ones present in testdata
  • null_dist (instance of distribution estimator) –
errorfx
null_dist

See also

Derived classes might provide additional methods via their base classes. Please refer to the list of base classes (if it exists) at the begining of the TransferError documentation.

Full API documentation of TransferError in module mvpa.clfs.transerror.