[Mne_analysis] CSP for Multi-Classification

Brunner, Clemens (clemens.brunner@uni-graz.at) clemens.brunner at uni-graz.at
Mon Feb 25 02:19:06 EST 2019
Search archives:

        External Email - Use Caution        

Hi!

If you use accuracy as your performance measure, note that the chance level depends on the number of classes (among other factors such as class balance). For 2 balanced classes, chance level is 50%, for 3 classes its 33.33%, for 4 classes 25%, and so on. Therefore, given equally well separable data, accuracy will decrease with increasing number of classes (but there are other definitions on how to average accuracy in the multi class case).

Here's a good summary of performance measures that might be helpful: https://scikit-learn.org/stable/modules/model_evaluation.html. You might want to try Cohen's kappa, which is 0 at chance level and 1 in the best case (irrespective of the number of classes you have).

CSP is not a classifier, it preprocesses the data so that the variance in one class is maximized vs. the other class. In the multi class case, CSP uses a 1-vs-rest scheme. All this happens automatically, so you don't have to worry about that. Your code looks good, you're correctly using a combination of CSP and a classifier.

Clemens




> On 24.02.2019, at 22:32, Xi Zh <timsasah at gmail.com> wrote:
> 
>         External Email - Use Caution        
> 
> 
> Hi Clemens,
> Thank you very much for your quesitons. I tried 2,3,4,6, and 18 classes. I found out that the best accuracy is in case of 2 classes and that by increasing the number of classes the accuracy decreases.
> I used cross_validation. for all cases (2,3,4,6, or 18 classes) Following is part of my code as I copied some from https://mne-tools.github.io/0.13/auto_examples/decoding/plot_decoding_csp_eeg.html
> 
> from mne.decoding import CSP
> from sklearn.cross_validation import cross_val_score
> 
> X_train, X_test, y_train, y_test = train_test_split(my_array, y, test_size=0.25,random_state=0)
> csp = CSP(n_components=15, reg=None, log=True, norm_trace=False)
> log_reg = LogisticRegression()
> clf = Pipeline([('CSP', csp), ('Reg', log_reg)])
> my_score = cross_val_score(clf, X_train, y_train, cv=10, n_jobs=1, scoring='accuracy')
> 
> Is it right this way for multi-classification? What would you suggest? What do you mean by  CSP is a spatial filter, and you can train filters e.g. in a 1-vs-rest scheme 
> 
> Many thanks for your kind help
> 
>  
> 
> 
> El sáb., 23 feb. 2019 a las 23:20, Brunner, Clemens (clemens.brunner at uni-graz.at) (<clemens.brunner at uni-graz.at>) escribió:
>         External Email - Use Caution        
> 
> 
> Hi!
> 
>  
> 
> Could you please be more specific? What have you tried for the multi-class case (how many classes?) and which method yielded better results? Which performance metric did you consider? Which data (cross-validation)? CSP is a spatial filter, and you can train filters e.g. in a 1-vs-rest scheme.
> 
>  
> 
> Clemens
> 
>  
> 
>  
> 
> From: mne_analysis-bounces at nmr.mgh.harvard.edu <mne_analysis-bounces at nmr.mgh.harvard.edu> On Behalf Of tr rt
> Sent: Saturday, 23 February 2019 11:59
> To: mne_analysis at nmr.mgh.harvard.edu
> Subject: [Mne_analysis] CSP for Multi-Classification
> 
>  
> 
>         External Email - Use Caution        
> 
> Hi everyone,
> 
> CSP (from mne.decoding import CSP) (Common Spatial Patterns) for Multi-Classification gives clearly worse accuracy than the binary classification. Would you suggest a better solution for the case of more than two classes? Maybe another kind of CSP? Another kind of Machine Learning methods? or even Deep Learning?
> 
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis




More information about the Mne_analysis mailing list