[Mne_analysis] Temporal Generalization - Different results with and without using cross validation

[Alexandre Gramfort]

        External Email - Use Caution        

hi,

when you do:

*cv = StratifiedKFold(n_splits=5, shuffle=True)*

*you make the cross-validation folds random (shuffle=True). It means*
*that everytime you run the code you will get a different value. This the
inherent*

*variance of the statistic that cross-validation reports.*

*To avoid this randomness (which should not be neglected) you can fix the
random state eg*

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

to avoid this you should use a StratifiedShuffleSplit and use many folds eg
100 to limit
the variance due to the choice of the cv splits.

HTH
Alex



On Wed, Mar 25, 2020 at 4:39 PM Maryam Zolfaghar <
Maryam.Zolfaghar at colorado.edu> wrote:

>         External Email - Use Caution
>
> Hi MNE experts,
>
> I am using the temporal generalization
> <https://mne.tools/dev/auto_tutorials/machine-learning/plot_sensors_decoding.html#temporal-generalization> approach.
> I have plotted scores' output from cross_val_multiscore
> <https://mne.tools/dev/generated/mne.decoding.cross_val_multiscore.html#mne.decoding.cross_val_multiscore>
> with a RepeatedStratifiedKFold
> <https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RepeatedStratifiedKFold.html#sklearn.model_selection.RepeatedStratifiedKFold>
>  cv parameter.  I have also plotted training scores
>
> I assumed that I should get the same result and the only difference would
> be the diagonal results where the diagonal training scores will be all 1.
> However, the general results are quite different (you can still see some
> fade underlying pattern similar in both).
>
> Any idea of why plotting scores using cross-validation v.s. only
> plotting fitting/training scores will give different results?
>
> This is my understanding of what should be going on: in the training case
> without using any cross-validation, on each time point, there was a
> classifier/decoder that was trained by seeing all EEG channels' data over
> all epochs at that train time point, therefore it would give a perfect
> score on the same test time point.  However, a different time point
> (testing times) has different data that can be seen as a test set for this
> decoder. Right? (Even if there was an autocorrelation between EEG data over
> time and still see some meaningful pattern in time generalization matrix,
> it means that EEG data had task-related information over time which is
> still meaningful).
>
> ---------
> I have also put my code here:
>
> *Scores using cross-validation:*
>
> *clf_SVC  = make_pipeline(*
> *                        StandardScaler(),*
> *                        LinearModel(LinearSVC(random_state=0,
> max_iter=10000)))*
>
> *temp_gen = GeneralizingEstimator(clf_SVC, scoring='roc_auc',
> n_jobs=1,verbose=True)*
>
> *cv = StratifiedKFold(n_splits=5, shuffle=True)*
> *scores = cross_val_multiscore(temp_gen, X, y, cv=cv, n_jobs=1)*
>
> *Only fitting scores:*
>
> *temp_gen.fit(X=X ,y=y)*
> *scores = temp_gen.score(X=X, y=y) #scores without cv*
> *-----------*
>
> - I will appreciate any comments,
> Thanks
>
>
>
>
>
>
>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200325/f627b409/attachment.html