[Mne_analysis] Stats on decoding scores {Disarmed}

Thu Feb 20 03:23:24 EST 2020

        External Email - Use Caution        

Hi Maryam,

It is difficult to help you further but here are a few remarks:

- If I recall correctly this function expects a 3d array: make sure that
you didn't swap space and time dimension.

- It is possible for a cluster analysis, depending on its parameters
(threshold) and on the data, to over-estimate the size of a cluster and to
thus suggest that everything is significant. This is because, formally,
such cluster statistics can't tell you whether each individual dimension
(e.g. time) is significant (e.g. see fieldtrip tutorial
<http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/>).
Instead, this analysis indicates whether a significant cluster can be
detected. Conclusions can be also drawn about the centroid of the cluster
under gaussian assumption: i.e. we can meaningfully report the mean and
variance of that cluster. To minimize the impact of cluster
parameterization, MNE has a threshold-free clustering method (tfce
parameters
<https://mne.tools/dev/auto_tutorials/stats-sensor-space/plot_stats_cluster_erp.html>),
but it is computationally expensive.

- What I would do in your case, is check significance with a non parametric
(e.g. wilcoxon) mass univariate analysis corrected with FDR on decimated
data

Hope that helps,

JR

On Thu, 20 Feb 2020 at 07:10, Maryam Zolfaghar <
Maryam.Zolfaghar at colorado.edu> wrote:

> Hi JR,
>
> Thank you for your helpful response. This function is amazing and fast but
> it did not give a reasonable result (the significant time shape is the same
> as the actual timing, all times came out significant!! ). I am wondering
> whether I should use all the default parameters or set threshold for
> example to 0.5 (still low and all time points are significant, I appreciate
> any suggestions) because there were two classes for the decoder?
>
> Thank you,
> -Maryam
>
>
> On Mon, Feb 17, 2020 at 7:30 AM JR KING <jeanremi.king at gmail.com> wrote:
>
>> Hi Maryam,
>>
>> you can use
>> https://mne.tools/stable/generated/mne.stats.spatio_temporal_cluster_1samp_test.html
>> <https://secure-web.cisco.com/1GFv_UWYrD1CKRWNpWB3dWMfsmjGVsYMIgWcj4SVk2IAieVzSgIAdCf4HQ7tflBGibZerLEEI7ZjXBDdgDLrQaX6Vu3laFEuN8igRdSbpCReq_TeRmEnPF1z3y9ifg9xSCXQ47mgtmY0jLRWp2-bbNN7AIB1u7CjbfKTZiAqgDgEox1oaRWSaziTPwFUZHpcWzOnAZRfiMUqztjGn03riSoLFeaKOFVXjgHImgX45BOU0kmJxTfJT1qWLa1_N7JajVqVnOD-nWiJsig5rFRP88F3ZiLKmM2ZMtwe4gntMaMY2jNrZRuCpkWSeQ4H1_NFIW1h2uBObKeZp7cXsrn3rvPzEM_DOCM0yvo5gEevlD99tV3mMn3NfTAu--lUhLNjVUfV4LxESvJM-nvLgX1KceEu6-NjQM-DzQXLOKlngM8IMQ17UCRmrl2eqPIaR3uxr9xdM_cvNGyVZ1CzHY_4FN6FRd0b7slASeQWRLp4yYU8/https%3A%2F%2Fmne.tools%2Fstable%2Fgenerated%2Fmne.stats.spatio_temporal_cluster_1samp_test.html>
>>
>> where X is the accuracy array of shape n_subjects x n_times x 1,
>>
>> hope that helps,
>>
>> Kindest regards
>>
>> JR
>>
>> On Tue, 11 Feb 2020 at 21:24, Maryam Zolfaghar <
>> Maryam.Zolfaghar at colorado.edu> wrote:
>>
>>>         External Email - Use Caution
>>>
>>> Hi Phillip,
>>>
>>> Thank you for the response.
>>>
>>> More specifically, I am trying to use the "STATISTICAL ANALYSIS OF
>>> DECODING ACCURACY" in this paper
>>> <https://www.jneurosci.org/content/38/2/409.long>. They used MATLAB and
>>> I am trying to use MNE Python.
>>>
>>> They did the following steps and report clusters of time points in
>>> which the decoding was significantly greater than chance after correction
>>> for multiple comparisons (e.g. Figure 3)
>>> In Step 1, they tested whether the obtained decoding accuracy at each
>>> individual time point during the delay interval was greater than chance
>>> using one-sample t-tests comparing the mean accuracy across
>>> participants to chance.
>>> In Step 2, they constructed a Monte Carlo null distribution of
>>> cluster-level t mass values.
>>> In Step 3, they obtained a null distribution for the cluster mass.
>>>
>>> P.s. I am doing this analysis on my own project and data but I also
>>> want to present MNE in a neuroscience department who are only using MATLAB
>>> to show using python and MNE could be another great option and save their
>>> time. That is why I am trying to see if I can use MNE for all steps instead
>>> of implementing them by myself in python).
>>>
>>> Thank you,
>>> -Maryam
>>>
>>> On Tue, Feb 11, 2020 at 8:55 AM Phillip Alday <phillip.alday at mpi.nl>
>>> wrote:
>>>
>>>> Hi Maryam,
>>>>
>>>> First: cluster-based permutation tests won't tell you whether any
>>>> particular times/clusters are actually significant (see
>>>> http://www.fieldtriptoolbox.org/faq/how_not_to_interpret_results_from_a_cluster-based_permutation_test/).
>>>> The null-hypotheses of these tests is exchangeability of conditions, at
>>>> least when they're defined over the two-sample t-test; there is some debate
>>>> over on the FieldTrip mailing list as to whether they make any sense for
>>>> the one-sample t-tests. I do think you can construct a meaningful
>>>> cluster-based permutation test using one-sample t-tests in some situations,
>>>> including in decoding situations, but you have to be careful.
>>>>
>>>> Second: You have to transform your decoding scores to be on an
>>>> unbounded scale before using the t-test or use a different test to
>>>> construct your permutation test. This follows directly from the assumptions
>>>> of the t-test (unboundedness and equal variance) and will be especially
>>>> problematic when your decoding scores in some temporal regions are close to
>>>> one, but close to 0.5 in other temporal regions, because these cannot have
>>>> equal variance (the variance of the binomial distribution is a function of
>>>> its mean). This is discussed in decoding analyzes of fMRI in Allefeld et
>>>> al. 2015 (https://doi.org/10.1016/j.neuroimage.2016.07.040). For a
>>>> simpler way around this, you could use the Fisher transformation (for
>>>> correlations) or the logistic function to get decoding scores on an
>>>> unbounded scale.
>>>>
>>>> If you still think you want to try to use a cluster-based permutation
>>>> test, let me know and I'll see if I can extract the relevant code from a
>>>> study I'm currently working on.
>>>>
>>>> Best,
>>>>
>>>> Phillip
>>>>
>>>>
>>>>
>>>> On 09/02/2020 03:01, Maryam Zolfaghar wrote:
>>>>
>>>>         External Email - Use Caution
>>>> Hi all,
>>>>
>>>> I'm trying to analyze whether my decoding scores over time (
>>>>
>>>> https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time
>>>> <https://secure-web.cisco.com/1uAwyFyh-uhac3a6v-MmauPyNYYC6SxvkCR3741x8dwDrlTdxIa9ip8AqJlU12SFnVkxvjajPdI2dc3FQaZ6-baqxv6ZzFeT0of9gsGUowGgxYi74cw6D5KMKjrvqJWjap7GYWQRPE4k4aj63uVMpv7dlAjlGOzGrmPT6Hl4e4TaYhAKbyIyFJRyT_2Pf_RuIJnYf4fOzn4WDayRBYD3SwhOLOewwCya_j2xenK0c2sTCItgNG69_lpw6eS4B1aCElF3p8rxxERxZnF-Q0I2yYESOpzeB8qj6eNL-j4ViN3cX3T-MeEib-MKxHDxGWWRrLEFGtxkcEyCmHberq5NWG2ppGHWE2wFn-P8MHpLhjDpYHNSTGEM4Enq3JJZ6TOrCZVugJdYUQnb0n_ejs4ykkxEiHdpWj4EU4K_seISzcwGCcSELHLJMA7QxAXavtR2I1XNa8D-Lc95eOxeEdDNLbPy0xSSYGBDNhci6H7xkTxI/https%3A%2F%2Fmne.tools%2Fstable%2Fauto_tutorials%2Fmachine-learning%2Fplot_sensors_decoding.html%23decoding-over-time>)
>>>> are "significant" or not, doing permutation testing and cluster-based
>>>> correction. Does anyone have any idea how to do it in MNE?
>>>>
>>>> Thank you,
>>>> -Maryam
>>>>
>>>>
>>>> _______________________________________________
>>>> Mne_analysis mailing listMne_analysis at nmr.mgh.harvard.eduhttps://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>>
>>>> _______________________________________________
>>> Mne_analysis mailing list
>>> Mne_analysis at nmr.mgh.harvard.edu
>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200220/3f9c77da/attachment-0001.html 