[Mne_analysis] Source Space Decoding Classification Timecourse

[alexandre.barachant at gmail.com]

Hi Cody,

Depending on your number of trials, the number of feature and the cross
validation procedure, you can have fairly high decoding results just by
chance.
You should never interpret a results without running a statistical test.
One good way to get the chance level of your classification pipeline is to
run a permutation test :
http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.permutation_test_score.html
the idea is to shuffle the labels, and train again the model to see what
score you get 'by chance'. it is sometimes surprising how high you can get.

If you have an unbalanced number of trial per class, i would also suggest
to use the AUC as a metric instead of the accuracy.

Alex

On Sat, Aug 5, 2017 at 9:11 AM JR KING <jeanremi.king at gmail.com> wrote:

> Hi Cody,
>
> Overall, your baseline doesn't look too bad - you would need to do a
> statistical test to check whether it is just noise variation or
> above-chance decoding scores.
>
> Still there could be multiple reasons behind a significant accuracy before
> t0 here:
> - accuracy is biased for imbalanced datasets. You can either
> use epochs.equalize_event_counts before your cross validation, or better,
> use a 'roc_auc' scoring metrics
> - filtering the data can spread information over time. Try changing your
> filtering parameters
> - IIRC, the 'sample' protocol is actually not randomized, and it is
> possible to predict the simulus category in advance.
>
> If you're using the MNE master branch, then I would recommend simplfy
> using this instead of your big loop (see
> https://martinos.org/mne/dev/auto_tutorials/plot_sensors_decoding.html#temporal-decoding
> for more details):
>
> clf = make_pipeline(StandardScaler(), SelectKBest(f_classif, k=500),
> SVC(kernel='linear'))
> time_decod = SlidingEstimator(clf, scoring='roc_auc')
> scores = cross_val_multiscore(clf, X, y, cv=5)
> plt.plot(times, scores.mean(0))
>
> (Note that I would personnally recommend clf =
> make_pipeline(StandardScaler(), LogisticRegression(C=1)) which should be
> better)
>
> Else, I believe we will be releasing the next version of MNE this month,
> so you'll just have to update MNE.
>
> Hope that helps,
>
> Jean-Rémi
>
>
>
>
>
> On 4 August 2017 at 17:19, Ghuman, Avniel <ghumana at upmc.edu> wrote:
>
>> Hi Cody,
>>
>> Do you have the same number of trials in each condition after any trial
>> rejection you do? If not, then the issue might be that 50% is not the
>> correct chance level to think about, rather the correct chance level is the
>> proportion of trials that is in your more frequent condition (eyeballing,
>> maybe like 55%?). There are unbiased classifiers you can use, but I am not
>> sure if they are built into MNE python...
>>
>> Best wishes,
>> Avniel
>>
>> ________________________________
>> From: mne_analysis-bounces at nmr.mgh.harvard.edu [
>> mne_analysis-bounces at nmr.mgh.harvard.edu] on behalf of Cushing, Cody [
>> CCUSHING1 at mgh.harvard.edu]
>> Sent: Friday, August 04, 2017 5:11 PM
>> To: mne_analysis at nmr.mgh.harvard.edu
>> Subject: [Mne_analysis] Source Space Decoding Classification Timecourse
>>
>> Hi,
>>
>> I've been trying to modify the following example:
>>
>>
>> http://martinos.org/mne/dev/auto_examples/decoding/plot_decoding_spatio_temporal_source.html
>>
>> to yield a time resolved classification accuracy.  I'm new to decoding so
>> I've done it in a fairly brute way (just iterating this script over every
>> time point), which yields a fairly convincing classification accuracy
>> timecourse.  However, I'm a bit concerned at how high the accuracy is
>> during the baseline, pre-stim period.  See attached for the modified script
>> using the sample data and an example of the output.  I'm new to decoding,
>> but the best answer I've been able to find for abnormally high pre-stim
>> accuracy is failing to cross validate, but that shouldn't be the case as
>> cross validation is being performed (but perhaps I'm doing it wrong) .  Is
>> there something improper about my strategy here?  Thanks for any input.
>>
>> Cheers,
>> Cody
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>>
>> The information in this e-mail is intended only for the person to whom it
>> is
>> addressed. If you believe this e-mail was sent to you in error and the
>> e-mail
>> contains patient information, please contact the Partners Compliance
>> HelpLine at
>> http://www.partners.org/complianceline . If the e-mail was sent to you
>> in error
>> but does not contain patient information, please contact the sender and
>> properly
>> dispose of the e-mail.
>>
>>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to whom it
> is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20170805/6c85ff6b/attachment.html