[Mne_analysis] Stats on decoding scores

[JR KING]

        External Email - Use Caution        

Hi Maryam,

you can use
https://mne.tools/stable/generated/mne.stats.spatio_temporal_cluster_1samp_test.html

where X is the accuracy array of shape n_subjects x n_times x 1,

hope that helps,

Kindest regards

JR

On Tue, 11 Feb 2020 at 21:24, Maryam Zolfaghar <
Maryam.Zolfaghar at colorado.edu> wrote:

>         External Email - Use Caution
>
> Hi Phillip,
>
> Thank you for the response.
>
> More specifically, I am trying to use the "STATISTICAL ANALYSIS OF
> DECODING ACCURACY" in this paper
> <https://www.jneurosci.org/content/38/2/409.long>. They used MATLAB and I
> am trying to use MNE Python.
>
> They did the following steps and report clusters of time points in which
> the decoding was significantly greater than chance after correction for
> multiple comparisons (e.g. Figure 3)
> In Step 1, they tested whether the obtained decoding accuracy at each
> individual time point during the delay interval was greater than chance
> using one-sample t-tests comparing the mean accuracy across participants
> to chance.
> In Step 2, they constructed a Monte Carlo null distribution of
> cluster-level t mass values.
> In Step 3, they obtained a null distribution for the cluster mass.
>
> P.s. I am doing this analysis on my own project and data but I also want
> to present MNE in a neuroscience department who are only using MATLAB to
> show using python and MNE could be another great option and save their
> time. That is why I am trying to see if I can use MNE for all steps instead
> of implementing them by myself in python).
>
> Thank you,
> -Maryam
>
> On Tue, Feb 11, 2020 at 8:55 AM Phillip Alday <phillip.alday at mpi.nl>
> wrote:
>
>> Hi Maryam,
>>
>> First: cluster-based permutation tests won't tell you whether any
>> particular times/clusters are actually significant (see
>> http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/).
>> The null-hypotheses of these tests is exchangeability of conditions, at
>> least when they're defined over the two-sample t-test; there is some debate
>> over on the FieldTrip mailing list as to whether they make any sense for
>> the one-sample t-tests. I do think you can construct a meaningful
>> cluster-based permutation test using one-sample t-tests in some situations,
>> including in decoding situations, but you have to be careful.
>>
>> Second: You have to transform your decoding scores to be on an unbounded
>> scale before using the t-test or use a different test to construct your
>> permutation test. This follows directly from the assumptions of the t-test
>> (unboundedness and equal variance) and will be especially problematic when
>> your decoding scores in some temporal regions are close to one, but close
>> to 0.5 in other temporal regions, because these cannot have equal variance
>> (the variance of the binomial distribution is a function of its mean). This
>> is discussed in decoding analyzes of fMRI in Allefeld et al. 2015 (
>> https://doi.org/10.1016/j.neuroimage.2016.07.040). For a simpler way
>> around this, you could use the Fisher transformation (for correlations) or
>> the logistic function to get decoding scores on an unbounded scale.
>>
>> If you still think you want to try to use a cluster-based permutation
>> test, let me know and I'll see if I can extract the relevant code from a
>> study I'm currently working on.
>>
>> Best,
>>
>> Phillip
>>
>>
>>
>> On 09/02/2020 03:01, Maryam Zolfaghar wrote:
>>
>>         External Email - Use Caution
>> Hi all,
>>
>> I'm trying to analyze whether my decoding scores over time (
>>
>> https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time)
>> are "significant" or not, doing permutation testing and cluster-based
>> correction. Does anyone have any idea how to do it in MNE?
>>
>> Thank you,
>> -Maryam
>>
>>
>> _______________________________________________
>> Mne_analysis mailing listMne_analysis at nmr.mgh.harvard.eduhttps://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200217/926cba86/attachment.html