[Mne_analysis] Stats on decoding scores

[Phillip Alday]

Hi Maryam,

First: cluster-based permutation tests won't tell you whether any
particular times/clusters are actually significant (see
http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/).
The null-hypotheses of these tests is exchangeability of conditions, at
least when they're defined over the two-sample t-test; there is some
debate over on the FieldTrip mailing list as to whether they make any
sense for the one-sample t-tests. I do think you can construct a
meaningful cluster-based permutation test using one-sample t-tests in
some situations, including in decoding situations, but you have to be
careful.

Second: You have to transform your decoding scores to be on an unbounded
scale before using the t-test or use a different test to construct your
permutation test. This follows directly from the assumptions of the
t-test (unboundedness and equal variance) and will be especially
problematic when your decoding scores in some temporal regions are close
to one, but close to 0.5 in other temporal regions, because these cannot
have equal variance (the variance of the binomial distribution is a
function of its mean). This is discussed in decoding analyzes of fMRI in
Allefeld et al. 2015 (https://doi.org/10.1016/j.neuroimage.2016.07.040).
For a simpler way around this, you could use the Fisher transformation
(for correlations) or the logistic function to get decoding scores on an
unbounded scale.

If you still think you want to try to use a cluster-based permutation
test, let me know and I'll see if I can extract the relevant code from a
study I'm currently working on.

Best,

Phillip



On 09/02/2020 03:01, Maryam Zolfaghar wrote:
>
>         External Email - Use Caution        
>
> Hi all,
>
> I'm trying to analyze whether my decoding scores over time (
> https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time)
> are "significant" or not, doing permutation testing and cluster-based
> correction. Does anyone have any idea how to do it in MNE? 
>
> Thank you, 
> -Maryam
>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200211/9a6701bf/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
Url : http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200211/9a6701bf/attachment.bin 
-------------- next part --------------
        External Email - Use Caution        
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200211/9a6701bf/attachment-0001.html