<p><span style="padding: 3px 10px; border-radius: 5px; color: #ffffff; font-weight: bold; display: inline-block; background-color: #ff0000;"> External Email - Use Caution </span></p><p></p><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Alex,<div><br></div><div>Thanks a lot for the helpful information. I have run my code according to your suggestion but I want to make sure I understood your point correctly.</div><div><br></div><div>As you suggested, the difference in results might be due to "shuff<font face="arial, sans-serif">le=True". However, I do not understand why you suggested "<b>StratifiedShuffleSplit</b> "? what would be the difference between doing "cv = <b>StratifiedKFold</b><b>(n_splits=5</b>, shuffle=True, random_state=42) or <span style="color:rgb(0,0,0)">cv = </span><b style="color:rgb(0,0,0)">StratifiedKFold</b><font style="color:rgb(0,0,0)"><b>(</b>n_splits=5, <b>shuffle=False</b>)</font>" vs "cv = <b>StratifiedShuffleSplit</b>(<b>n_splits=1000</b>, random_state=42)"? Why not only setting random state and just using <span style="color:rgb(0,0,0)">"cv = <b>StratifiedShuffleSplit</b>(</span><b style="color:rgb(0,0,0)">n_splits=5</b><span style="color:rgb(0,0,0)">, random_state=42)" with the same number of splits?</span></font></div><div><br></div><div>Thanks, </div><div>-Maryam</div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 25, 2020 at 4:15 PM Alexandre Gramfort <<a href="mailto:alexandre.gramfort@inria.fr">alexandre.gramfort@inria.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">hi,<div><br></div><div>when you do:</div><div><br></div><div><i style="color:rgb(0,0,0)"><font face="arial, sans-serif">cv = StratifiedKFold(n_splits=5, shuffle=True)</font></i><br></div><div><i style="color:rgb(0,0,0)"><font face="arial, sans-serif"><br></font></i></div><div><i style="color:rgb(0,0,0)"><font face="arial, sans-serif">you make the cross-validation folds random (shuffle=True). It means</font></i></div><div><font color="#000000" face="arial, sans-serif"><i>that everytime you run the code you will get a different value. This the inherent</i></font></div><div><font color="#000000" face="arial, sans-serif"><i>variance of the statistic that cross-validation reports.<br></i></font></div><div><font color="#000000" face="arial, sans-serif"><i><br></i></font></div><div><font color="#000000" face="arial, sans-serif"><i>To avoid this randomness (which should not be neglected) you can fix the random state eg</i></font></div><div><font color="#000000" face="arial, sans-serif"><i><br></i></font></div><div>cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)<font color="#000000" face="arial, sans-serif"><i><br></i></font></div><div><br></div><div>to avoid this you should use a StratifiedShuffleSplit and use many folds eg 100 to limit</div><div>the variance due to the choice of the cv splits.</div><div><br></div><div>HTH</div><div>Alex</div><div><br></div><div><font color="#000000" face="arial, sans-serif"><i><br></i></font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 25, 2020 at 4:39 PM Maryam Zolfaghar <<a href="mailto:Maryam.Zolfaghar@colorado.edu" target="_blank">Maryam.Zolfaghar@colorado.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><p><span style="padding:3px 10px;border-top-left-radius:5px;border-top-right-radius:5px;border-bottom-right-radius:5px;border-bottom-left-radius:5px;color:rgb(255,255,255);font-weight:bold;display:inline-block;background-color:rgb(255,0,0)"> External Email - Use Caution </span></p><p></p><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi MNE experts,<div><br></div><div>I am using the<a href="https://secure-web.cisco.com/1jWVu354v-yk2PHG6scpRuNlqi-Xei3wH-moV9-Av0fstdMSB7aUnzzNvmfnF3JuIo6EC0A07IdflTTdP4WpUqpUrZSclzoSV8WBksahu0ODpDgFOsHmm6eHO0i8QuTC6bAvL9yb4XuOmHH4wqg_ubbDuEIZUQriZvysI-ayKjx5o4jndHXz4UqLxv0CVaHv0YafoA8c9X29SwV52DKgOaPlf2v1oiEOTcd-Wu6QtvRMnn5AgMVc4_QxbHaXjM5qAZz8u1r_wFixIzhsSgFa89d05jmOT0BVNWMZ1-sXCPt0by2krKKqiEyV7OLYBLLMrTTnWZ00KZGIGX_VkqbIA4uArHPJxE7w_U3mNvn0YFXUUhVuopodRcQf_ZA81GJA5E5Qy5s6PW0ZP8SjUtICoajBkEHmm6LpIKycpscbg0W7uVdtFNyDkDTMXVq0OemdyrFxvdqDi6VnKPtgefbyyNw/https%3A%2F%2Fmne.tools%2Fdev%2Fauto_tutorials%2Fmachine-learning%2Fplot_sensors_decoding.html%23temporal-generalization" target="_blank"> temporal generalization</a> approach. I have plotted scores' output from<font face="arial, sans-serif"> <a href="https://secure-web.cisco.com/1h1fBYrY82nc7Ha3ohR5V_l8-7FgWv1EKHCEhN_hAJm8A-NfF2V7uxTWmy5IwO-cOk33FiOpkywzhHZcxjWQfQez3GPysxi2nGr5NkcO-UbypLFGD_4d5yopqahopnYqwDy-b5yOA56uFYbRzHJLYyu6NsGNICQkbBWbwDt1EXOgxAFYg6wK5WIcxNYuZ9Jg_4G88dlreZRD8p-yNwFZ7D7z_DBKIo8eudHagdQhME0woFuPo63yrIoM4zLcAWLhFta-xNl2IvYh9iTIk5U5zUwFIbyRBa4HIWRD-Es3MGFPRcDmjke4u-MSG4yKuwjcIIfy3K4MCoiMF79wTrt2u-SutYTmMNte1IFDKYhw-kx7oH-XbtFkLZH1xigiLkdICoIFeZnptCIYr0Cu1mb75-E8SZD17l2y3xvLlgqZQ2aOYCPoFgCdlbtaBN7GhzUzeunr0EMGN7y6_W-4o-9ASig/https%3A%2F%2Fmne.tools%2Fdev%2Fgenerated%2Fmne.decoding.cross_val_multiscore.html%23mne.decoding.cross_val_multiscore" target="_blank">cross_val_multiscore</a> with <span style="color:rgb(0,0,0)">a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RepeatedStratifiedKFold.html#sklearn.model_selection.RepeatedStratifiedKFold" target="_blank">RepeatedStratifiedKFold</a> </span>cv parameter. </font><span style="font-family:arial,sans-serif">I have also plotted training scores </span></div><div><span style="font-family:arial,sans-serif"><br></span></div><div><div>I assumed that I should get the same result and the only difference would be the diagonal results where the diagonal training scores will be all 1. However, the general results are quite different (you can still see so<font face="arial, sans-serif">me fade underlying pattern similar in both).</font></div><div><font face="arial, sans-serif"><br></font></div><div><span style="color:rgb(0,0,0)"><font face="arial, sans-serif">Any idea of why plotting scores using cross-validation v.s. only plotting fitting/training scores will give different results?</font></span><br></div><div><br></div><div>This is my understanding of what should be going on: in the training case without using any cross-validation, on each time point, there was a classifier/decoder that was trained by seeing all EEG channels' data over all epochs at that train time point, therefore it would give a perfect score on the same test time poin<font face="arial, sans-serif">t. However, a different time point (testing times) has different data that can be seen as a test set for this decoder. Right? (Even if there was an autocorrelation between EEG data over time and still see some meaningful pattern <span style="color:rgb(0,0,0)">in time generalization matrix</span>, it means that EEG data had task-related information over time which is still meaningful).</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">---------</font></div><div><font face="arial, sans-serif">I have also put my code here:</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"><b>Scores using cross-validation:</b></font></div><div><font face="arial, sans-serif"><br></font></div><div><div style="color:rgb(0,0,0)"><div><i><font face="arial, sans-serif">clf_SVC = make_pipeline(</font></i></div><div><i><font face="arial, sans-serif"> StandardScaler(),</font></i></div><div><i><font face="arial, sans-serif"> LinearModel(LinearSVC(random_state=0, max_iter=10000)))</font></i></div><div><i><font face="arial, sans-serif"><br></font></i></div></div><div style="color:rgb(0,0,0)"><font face="arial, sans-serif"><i>temp_gen = GeneralizingEstimator(clf_SVC, scoring='roc_auc', n_jobs=1,verbose=True)</i><i><br></i></font></div><div style="color:rgb(0,0,0)"><i><font face="arial, sans-serif">cv = StratifiedKFold(n_splits=5, shuffle=True)<br></font></i></div><div style="color:rgb(0,0,0)"><i><font face="arial, sans-serif">scores = cross_val_multiscore(temp_gen, X, y, cv=cv, n_jobs=1)</font></i></div></div><div style="color:rgb(0,0,0)"><i><font face="arial, sans-serif"><br></font></i></div><div style="color:rgb(0,0,0)"><font face="arial, sans-serif"><b>Only fitting scores:</b></font></div><div style="color:rgb(0,0,0)"><font face="arial, sans-serif"><b><br></b></font></div><div style="color:rgb(0,0,0)"><div><i><font face="arial, sans-serif">temp_gen.fit(X=X ,y=y)</font></i></div><div><i><font face="arial, sans-serif">scores = temp_gen.score(X=X, y=y) #scores without cv</font></i></div><div style="font-family:-webkit-standard"><i>-----------</i></div></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">- I will appreciate any comments,</font></div><div><font face="arial, sans-serif">Thanks</font></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div><br></div><div><i></i></div><div><br></div><div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Mne_analysis mailing list<br>
<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu" target="_blank">Mne_analysis@nmr.mgh.harvard.edu</a><br>
<a href="https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis" rel="noreferrer" target="_blank">https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis</a></blockquote></div>
</blockquote></div>