[Mne_analysis] Temporal Generalization - Different results with and without using cross validation

[Maryam Zolfaghar]

        External Email - Use Caution        

Hi Alex,

Thanks a lot for the helpful information. I have run my code according to
your suggestion but I want to make sure I understood your point correctly.

As you suggested, the difference in results might be due to "shuffle=True".
However, I do not understand why you suggested "*StratifiedShuffleSplit* "?
what would be the difference between doing "cv = *StratifiedKFold*
*(n_splits=5*, shuffle=True, random_state=42) or cv =
*StratifiedKFold**(*n_splits=5,
*shuffle=False*)"    vs   "cv = *StratifiedShuffleSplit*(*n_splits=1000*,
random_state=42)"? Why not only setting random state and just using "cv =
*StratifiedShuffleSplit*(*n_splits=5*, random_state=42)"  with the same
number of splits?

Thanks,
-Maryam

On Wed, Mar 25, 2020 at 4:15 PM Alexandre Gramfort <
alexandre.gramfort at inria.fr> wrote:

> hi,
>
> when you do:
>
> *cv = StratifiedKFold(n_splits=5, shuffle=True)*
>
> *you make the cross-validation folds random (shuffle=True). It means*
> *that everytime you run the code you will get a different value. This the
> inherent*
>
> *variance of the statistic that cross-validation reports.*
>
> *To avoid this randomness (which should not be neglected) you can fix the
> random state eg*
>
> cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
>
> to avoid this you should use a StratifiedShuffleSplit and use many folds
> eg 100 to limit
> the variance due to the choice of the cv splits.
>
> HTH
> Alex
>
>
>
> On Wed, Mar 25, 2020 at 4:39 PM Maryam Zolfaghar <
> Maryam.Zolfaghar at colorado.edu> wrote:
>
>>         External Email - Use Caution
>>
>> Hi MNE experts,
>>
>> I am using the temporal generalization
>> <https://secure-web.cisco.com/1jWVu354v-yk2PHG6scpRuNlqi-Xei3wH-moV9-Av0fstdMSB7aUnzzNvmfnF3JuIo6EC0A07IdflTTdP4WpUqpUrZSclzoSV8WBksahu0ODpDgFOsHmm6eHO0i8QuTC6bAvL9yb4XuOmHH4wqg_ubbDuEIZUQriZvysI-ayKjx5o4jndHXz4UqLxv0CVaHv0YafoA8c9X29SwV52DKgOaPlf2v1oiEOTcd-Wu6QtvRMnn5AgMVc4_QxbHaXjM5qAZz8u1r_wFixIzhsSgFa89d05jmOT0BVNWMZ1-sXCPt0by2krKKqiEyV7OLYBLLMrTTnWZ00KZGIGX_VkqbIA4uArHPJxE7w_U3mNvn0YFXUUhVuopodRcQf_ZA81GJA5E5Qy5s6PW0ZP8SjUtICoajBkEHmm6LpIKycpscbg0W7uVdtFNyDkDTMXVq0OemdyrFxvdqDi6VnKPtgefbyyNw/https%3A%2F%2Fmne.tools%2Fdev%2Fauto_tutorials%2Fmachine-learning%2Fplot_sensors_decoding.html%23temporal-generalization> approach.
>> I have plotted scores' output from cross_val_multiscore
>> <https://secure-web.cisco.com/1h1fBYrY82nc7Ha3ohR5V_l8-7FgWv1EKHCEhN_hAJm8A-NfF2V7uxTWmy5IwO-cOk33FiOpkywzhHZcxjWQfQez3GPysxi2nGr5NkcO-UbypLFGD_4d5yopqahopnYqwDy-b5yOA56uFYbRzHJLYyu6NsGNICQkbBWbwDt1EXOgxAFYg6wK5WIcxNYuZ9Jg_4G88dlreZRD8p-yNwFZ7D7z_DBKIo8eudHagdQhME0woFuPo63yrIoM4zLcAWLhFta-xNl2IvYh9iTIk5U5zUwFIbyRBa4HIWRD-Es3MGFPRcDmjke4u-MSG4yKuwjcIIfy3K4MCoiMF79wTrt2u-SutYTmMNte1IFDKYhw-kx7oH-XbtFkLZH1xigiLkdICoIFeZnptCIYr0Cu1mb75-E8SZD17l2y3xvLlgqZQ2aOYCPoFgCdlbtaBN7GhzUzeunr0EMGN7y6_W-4o-9ASig/https%3A%2F%2Fmne.tools%2Fdev%2Fgenerated%2Fmne.decoding.cross_val_multiscore.html%23mne.decoding.cross_val_multiscore>
>> with a RepeatedStratifiedKFold
>> <https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RepeatedStratifiedKFold.html#sklearn.model_selection.RepeatedStratifiedKFold>
>>  cv parameter.  I have also plotted training scores
>>
>> I assumed that I should get the same result and the only difference would
>> be the diagonal results where the diagonal training scores will be all 1.
>> However, the general results are quite different (you can still see some
>> fade underlying pattern similar in both).
>>
>> Any idea of why plotting scores using cross-validation v.s. only
>> plotting fitting/training scores will give different results?
>>
>> This is my understanding of what should be going on: in the training case
>> without using any cross-validation, on each time point, there was a
>> classifier/decoder that was trained by seeing all EEG channels' data over
>> all epochs at that train time point, therefore it would give a perfect
>> score on the same test time point.  However, a different time point
>> (testing times) has different data that can be seen as a test set for this
>> decoder. Right? (Even if there was an autocorrelation between EEG data over
>> time and still see some meaningful pattern in time generalization matrix,
>> it means that EEG data had task-related information over time which is
>> still meaningful).
>>
>> ---------
>> I have also put my code here:
>>
>> *Scores using cross-validation:*
>>
>> *clf_SVC  = make_pipeline(*
>> *                        StandardScaler(),*
>> *                        LinearModel(LinearSVC(random_state=0,
>> max_iter=10000)))*
>>
>> *temp_gen = GeneralizingEstimator(clf_SVC, scoring='roc_auc',
>> n_jobs=1,verbose=True)*
>>
>> *cv = StratifiedKFold(n_splits=5, shuffle=True)*
>> *scores = cross_val_multiscore(temp_gen, X, y, cv=cv, n_jobs=1)*
>>
>> *Only fitting scores:*
>>
>> *temp_gen.fit(X=X ,y=y)*
>> *scores = temp_gen.score(X=X, y=y) #scores without cv*
>> *-----------*
>>
>> - I will appreciate any comments,
>> Thanks
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200403/556f65b1/attachment.html