[Mne_analysis] Cluster-based Permutation T-test for Decoders

JR KING jeanremi.king at gmail.com
Fri Oct 11 06:09:40 EDT 2019
Search archives:

        External Email - Use Caution        

Hi, no, AUC is just a metrics for effect size. You'll need to do
permutation tests to get a p-value.

On Thu, 10 Oct 2019 at 20:23, Maryam Zolfaghar <
Maryam.Zolfaghar at colorado.edu> wrote:

> Thanks for the response.
> Does AUC take care of the multiple comparison issues? How I will be sure
> that the accuracy is significant (above chance)?
>
> -Mary
>
> On Thu, Oct 10, 2019 at 4:04 AM JR KING <jeanremi.king at gmail.com> wrote:
>
>> You can use a one-versus-all classifier and compute the average AUC
>> across  categories
>>
>> HTH
>> JR
>>
>> On Thu, 10 Oct 2019 at 05:24, Maryam Zolfaghar <
>> Maryam.Zolfaghar at colorado.edu> wrote:
>>
>>>         External Email - Use Caution
>>>
>>> Hi all,
>>>
>>> I am trying to
>>>
>>>    - use decoders to decode whether ERP or time-frequency signals have
>>>    any meaninful information of four classes (location of the target on the
>>>    screen) in my experiment *over time *(according to this example
>>>    <https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time>
>>>    ).
>>>    -  and then test whether the output of the decoder is significantly
>>>    above the chance (in my case: 1/4=0.25) using a permutation t-test with
>>>    cluster-based correction.
>>>
>>> My question is:
>>>
>>>    - In the example
>>>    <https://mne.tools/stable/auto_tutorials/machine-learning/plot_sensors_decoding.html#decoding-over-time>
>>>    there are only two classes, so AUC was used. However, what if there are
>>>    more than two classes? How I can analyze the significance of the decoder's
>>>    output with the cluster-based correction?
>>>
>>>
>>> Thanks,
>>> -Mary
>>>
>>> _______________________________________________
>>> Mne_analysis mailing list
>>> Mne_analysis at nmr.mgh.harvard.edu
>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20191011/2cea9205/attachment.html 


More information about the Mne_analysis mailing list