[Mne_analysis] Stats on decoding scores {Disarmed}

[Maryam Zolfaghar]

        External Email - Use Caution        

Hi JR,

Thank you for your helpful response. This function is amazing and fast but
it did not give a reasonable result (the significant time shape is the same
as the actual timing, all times came out significant!! ). I am wondering
whether I should use all the default parameters or set threshold for
example to 0.5 (still low and all time points are significant, I appreciate
any suggestions) because there were two classes for the decoder?

Thank you,
-Maryam


On Mon, Feb 17, 2020 at 7:30 AM JR KING <jeanremi.king at gmail.com> wrote:

> Hi Maryam,
>
> you can use
> https://mne.tools/stable/generated/mne.stats.spatio_temporal_cluster_1samp_test.html
> <https://secure-web.cisco.com/1GFv_UWYrD1CKRWNpWB3dWMfsmjGVsYMIgWcj4SVk2IAieVzSgIAdCf4HQ7tflBGibZerLEEI7ZjXBDdgDLrQaX6Vu3laFEuN8igRdSbpCReq_TeRmEnPF1z3y9ifg9xSCXQ47mgtmY0jLRWp2-bbNN7AIB1u7CjbfKTZiAqgDgEox1oaRWSaziTPwFUZHpcWzOnAZRfiMUqztjGn03riSoLFeaKOFVXjgHImgX45BOU0kmJxTfJT1qWLa1_N7JajVqVnOD-nWiJsig5rFRP88F3ZiLKmM2ZMtwe4gntMaMY2jNrZRuCpkWSeQ4H1_NFIW1h2uBObKeZp7cXsrn3rvPzEM_DOCM0yvo5gEevlD99tV3mMn3NfTAu--lUhLNjVUfV4LxESvJM-nvLgX1KceEu6-NjQM-DzQXLOKlngM8IMQ17UCRmrl2eqPIaR3uxr9xdM_cvNGyVZ1CzHY_4FN6FRd0b7slASeQWRLp4yYU8/https%3A%2F%2Fmne.tools%2Fstable%2Fgenerated%2Fmne.stats.spatio_temporal_cluster_1samp_test.html>
>
> where X is the accuracy array of shape n_subjects x n_times x 1,
>
> hope that helps,
>
> Kindest regards
>
> JR
>
> On Tue, 11 Feb 2020 at 21:24, Maryam Zolfaghar <
> Maryam.Zolfaghar at colorado.edu> wrote:
>
>>         External Email - Use Caution
>>
>> Hi Phillip,
>>
>> Thank you for the response.
>>
>> More specifically, I am trying to use the "STATISTICAL ANALYSIS OF
>> DECODING ACCURACY" in this paper
>> <https://www.jneurosci.org/content/38/2/409.long>. They used MATLAB and
>> I am trying to use MNE Python.
>>
>> They did the following steps and report clusters of time points in which
>> the decoding was significantly greater than chance after correction for
>> multiple comparisons (e.g. Figure 3)
>> In Step 1, they tested whether the obtained decoding accuracy at each
>> individual time point during the delay interval was greater than chance
>> using one-sample t-tests comparing the mean accuracy across participants
>> to chance.
>> In Step 2, they constructed a Monte Carlo null distribution of
>> cluster-level t mass values.
>> In Step 3, they obtained a null distribution for the cluster mass.
>>
>> P.s. I am doing this analysis on my own project and data but I also want
>> to present MNE in a neuroscience department who are only using MATLAB to
>> show using python and MNE could be another great option and save their
>> time. That is why I am trying to see if I can use MNE for all steps instead
>> of implementing them by myself in python).
>>
>> Thank you,
>> -Maryam
>>
>> On Tue, Feb 11, 2020 at 8:55 AM Phillip Alday <phillip.alday at mpi.nl>
>> wrote:
>>
>>> Hi Maryam,
>>>
>>> First: cluster-based permutation tests won't tell you whether any
>>> particular times/clusters are actually significant (see
>>> http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/).
>>> The null-hypotheses of these tests is exchangeability of conditions, at
>>> least when they're defined over the two-sample t-test; there is some debate
>>> over on the FieldTrip mailing list as to whether they make any sense for
>>> the one-sample t-tests. I do think you can construct a meaningful
>>> cluster-based permutation test using one-sample t-tests in some situations,
>>> including in decoding situations, but you have to be careful.
>>>
>>> Second: You have to transform your decoding scores to be on an unbounded
>>> scale before using the t-test or use a different test to construct your
>>> permutation test. This follows directly from the assumptions of the t-test
>>> (unboundedness and equal variance) and will be especially problematic when
>>> your decoding scores in some temporal regions are close to one, but close
>>> to 0.5 in other temporal regions, because these cannot have equal variance
>>> (the variance of the binomial distribution is a function of its mean). This
>>> is discussed in decoding analyzes of fMRI in Allefeld et al. 2015 (
>>> https://doi.org/10.1016/j.neuroimage.2016.07.040). For a simpler way
>>> around this, you could use the Fisher transformation (for correlations) or
>>> the logistic function to get decoding scores on an unbounded scale.
>>>
>>> If you still think you want to try to use a cluster-based permutation
>>> test, let me know and I'll see if I can extract the relevant code from a
>>> study I'm currently working on.
>>>
>>> Best,
>>>
>>> Phillip
>>>
>>>
>>>
>>> On 09/02/2020 03:01, Maryam Zolfaghar wrote:
>>>
>>>         External Email - Use Caution
>>> Hi all,
>>>
>>> I'm trying to analyze whether my decoding scores over time (
>>>
>>> https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time
>>> <https://secure-web.cisco.com/1uAwyFyh-uhac3a6v-MmauPyNYYC6SxvkCR3741x8dwDrlTdxIa9ip8AqJlU12SFnVkxvjajPdI2dc3FQaZ6-baqxv6ZzFeT0of9gsGUowGgxYi74cw6D5KMKjrvqJWjap7GYWQRPE4k4aj63uVMpv7dlAjlGOzGrmPT6Hl4e4TaYhAKbyIyFJRyT_2Pf_RuIJnYf4fOzn4WDayRBYD3SwhOLOewwCya_j2xenK0c2sTCItgNG69_lpw6eS4B1aCElF3p8rxxERxZnF-Q0I2yYESOpzeB8qj6eNL-j4ViN3cX3T-MeEib-MKxHDxGWWRrLEFGtxkcEyCmHberq5NWG2ppGHWE2wFn-P8MHpLhjDpYHNSTGEM4Enq3JJZ6TOrCZVugJdYUQnb0n_ejs4ykkxEiHdpWj4EU4K_seISzcwGCcSELHLJMA7QxAXavtR2I1XNa8D-Lc95eOxeEdDNLbPy0xSSYGBDNhci6H7xkTxI/https%3A%2F%2Fmne.tools%2Fstable%2Fauto_tutorials%2Fmachine-learning%2Fplot_sensors_decoding.html%23decoding-over-time>)
>>> are "significant" or not, doing permutation testing and cluster-based
>>> correction. Does anyone have any idea how to do it in MNE?
>>>
>>> Thank you,
>>> -Maryam
>>>
>>>
>>> _______________________________________________
>>> Mne_analysis mailing listMne_analysis at nmr.mgh.harvard.eduhttps://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>
>>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200219/a52c6b69/attachment-0001.html