[Mne_analysis] Stats on decoding scores

[Maryam Zolfaghar]

        External Email - Use Caution        

Hi Phillip,

Thank you for the response.

More specifically, I am trying to use the "STATISTICAL ANALYSIS OF DECODING
ACCURACY" in this paper <https://www.jneurosci.org/content/38/2/409.long>.
They used MATLAB and I am trying to use MNE Python.

They did the following steps and report clusters of time points in which
the decoding was significantly greater than chance after correction for
multiple comparisons (e.g. Figure 3)
In Step 1, they tested whether the obtained decoding accuracy at each
individual time point during the delay interval was greater than chance
using one-sample t-tests comparing the mean accuracy across participants to
chance.
In Step 2, they constructed a Monte Carlo null distribution of
cluster-level t mass values.
In Step 3, they obtained a null distribution for the cluster mass.

P.s. I am doing this analysis on my own project and data but I also want to
present MNE in a neuroscience department who are only using MATLAB to show
using python and MNE could be another great option and save their time.
That is why I am trying to see if I can use MNE for all steps instead of
implementing them by myself in python).

Thank you,
-Maryam

On Tue, Feb 11, 2020 at 8:55 AM Phillip Alday <phillip.alday at mpi.nl> wrote:

> Hi Maryam,
>
> First: cluster-based permutation tests won't tell you whether any
> particular times/clusters are actually significant (see
> http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/).
> The null-hypotheses of these tests is exchangeability of conditions, at
> least when they're defined over the two-sample t-test; there is some debate
> over on the FieldTrip mailing list as to whether they make any sense for
> the one-sample t-tests. I do think you can construct a meaningful
> cluster-based permutation test using one-sample t-tests in some situations,
> including in decoding situations, but you have to be careful.
>
> Second: You have to transform your decoding scores to be on an unbounded
> scale before using the t-test or use a different test to construct your
> permutation test. This follows directly from the assumptions of the t-test
> (unboundedness and equal variance) and will be especially problematic when
> your decoding scores in some temporal regions are close to one, but close
> to 0.5 in other temporal regions, because these cannot have equal variance
> (the variance of the binomial distribution is a function of its mean). This
> is discussed in decoding analyzes of fMRI in Allefeld et al. 2015 (
> https://doi.org/10.1016/j.neuroimage.2016.07.040). For a simpler way
> around this, you could use the Fisher transformation (for correlations) or
> the logistic function to get decoding scores on an unbounded scale.
>
> If you still think you want to try to use a cluster-based permutation
> test, let me know and I'll see if I can extract the relevant code from a
> study I'm currently working on.
>
> Best,
>
> Phillip
>
>
>
> On 09/02/2020 03:01, Maryam Zolfaghar wrote:
>
>         External Email - Use Caution
> Hi all,
>
> I'm trying to analyze whether my decoding scores over time (
>
> https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time)
> are "significant" or not, doing permutation testing and cluster-based
> correction. Does anyone have any idea how to do it in MNE?
>
> Thank you,
> -Maryam
>
>
> _______________________________________________
> Mne_analysis mailing listMne_analysis at nmr.mgh.harvard.eduhttps://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200211/f1f3ffae/attachment.html