These results look reasonable given what each method is doing. The FDR blob is there because it is very bright (significant). It is lost in the cluster-wise correction because it is small, and the cluster-wise correction does not care how significant something is as long as it meets threshold. The RMT blob is there in the cluster-wise correction because it is big; it is lost in FDR because it is not significant. As you mention, these are very different methods, and it is incorrect to say that you use a p=0.05 in both methods. For the simulation, you set the voxel-wise threshold to .05. For FDR, you set the false discovery rate to .05; this is not a p-value and it is not a voxel-wise threshold.

A p-value is a False Positive Rate (FPR) which is interpreted differently than an FDR. For your FDR blob, the interpretation is that 5% of the voxels that are remaining are false positives (and so 95% are true positives); but you don't know which ones. If voxels remain active in FDR after correction, then you draw the conclusion that there is activation SOMEWHERE IN THE SEARCH SPACE. In this case, you can say that there is an effect somewhere in the left hemisphere. Where exactly? In your case, it is a little easier since you have a single blob. So you have to ask yourself if you remove any 5% of the active voxels whether you would change your conclusion. In this case, probably not because you only have one blob. If you had another smaller blob that was 5% of the total, you probably would not want to declare an effect for the smaller one. Note that this does not address the FPR (ie, the probability that that blob itself is there by chance in the first place).

For the cluster-wise correction, the interpretation is that a cluster of the given size in the RMT would occur by chance only 5% of the time.

It's a good question as to which method, if either or both, you should use to draw your conclusions. I'm not sure, I'd have to think about it more. Maybe some of the statisticians would like to weigh in.

doug

Stefan Brauns wrote:

Hi there,

we are struggling with one question and haven't yet heard back. We would be very grateful for any advice or recommendation.

We were running an analysis (95 subjects) examining a group effect (two groups, controlling for the effects of a covariate and two cofactors, DODS model) on cortical thickness with mri_glmfit. We then corrected our results for multiple comparisons using two methods, FDR and Monte Carlo simulation with 4000 repeats. The threshold was set at p=0.05 respectively. Although examining the same population we got results in very different regions.

command for the simulation:

mri_glmfit-sim --glmdir xxxx.glmdir --sim mc-z 4000 1.301 mc-z abs.1.301 --sim-sign abs

the fdr results were obtained setting the fdr-threshold at p=0.05 with the tksurfer script command (sclv_set current_threshold_using_fdr 0.05 0)

With the fdr method we found a spot in the left supramarginal gyrus and with the clusterwise correction we got a cluster in the right middle temporal lobe (see pictures of corrected and uncorrected results attached). No other clusters/findings survived one of the correction methods.

We are well aware of the fact that FDR and Monte Carlo simulation are very different statistical methods and that FDR is more conservative. Does that explain the discrepant results? Would you expect a highly robust FDR finding to not show up at all when using Monte Carlo? What additional information could be used to decide which method to use for the final models?

Thank you,

Stefan

---

---

---

Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

Who says FDR is more conservative? It all depends on the thresholds you choose.

Note that the FDR threshold is different from the cluster-based p threshold, and it isn't necessarily appropriate to use the same for both.

The cluster-based p threshold controls the overall probability that you will get 1 or more false positives. With p<0.05, the idea is that if you repeated your study 20 times, 19 times out of 20, you would have NO false positives.

With an FDR threshold of 0.05, you are guaranteed that 5% of your "significant" vertices are false positives. A more conservative threshold would be 0.01.

By the way, for cluster thresholding, there are two p thresholds that are important. The first is the p threshold applied to the z-stats or t-stats before finding clusters. 0.05 would usually be too liberal. 0.01 or 0.001 would be better. The second is the multiple comparison corrected p value, for which you would usually use 0.05.

Date: Fri, 29 Jan 2010 10:02:45 -0500 From: stefan.brauns@googlemail.com To: Freesurfer@nmr.mgh.harvard.edu Subject: [Freesurfer] thickness maps: FDR versus Monte Carlo - different results

Hi there,

we are struggling with one question and haven't yet heard back. We would be very grateful for any advice or recommendation. We were running an analysis (95 subjects) examining a group effect (two groups, controlling for the effects of a covariate and two cofactors, DODS model) on cortical thickness with mri_glmfit. We then corrected our results for multiple comparisons using two methods, FDR and Monte Carlo simulation with 4000 repeats. The threshold was set at p=0.05 respectively. Although examining the same population we got results in very different regions.

command for the simulation:

mri_glmfit-sim --glmdir xxxx.glmdir --sim mc-z 4000 1.301 mc-z abs.1.301 --sim-sign abs

the fdr results were obtained setting the fdr-threshold at p=0.05 with the tksurfer script command (sclv_set current_threshold_using_fdr 0.05 0)

With the fdr method we found a spot in the left supramarginal gyrus and with the clusterwise correction we got a cluster in the right middle temporal lobe (see pictures of corrected and uncorrected results attached). No other clusters/findings survived one of the correction methods.

We are well aware of the fact that FDR and Monte Carlo simulation are very different statistical methods and that FDR is more conservative. Does that explain the discrepant results? Would you expect a highly robust FDR finding to not show up at all when using Monte Carlo? What additional information could be used to decide which method to use for the final models?

Thank you, Stefan

_________________________________________________________________ Hotmail: Free, trusted and rich email service. http://clk.atdmt.com/GBL/go/196390708/direct/01/

In my experience, which is more in fMRI than thickness analyses, 0.05 can yield very large mega-clusters that should actually be considered a conglomeration of smaller clusters. I found that regardless of whether one does Monte Carlo or GRF. Because 0.05 for uncorrected p values is quite liberal, only the very large clusters survive the multiple comparisons correction. So you can end up losing the smaller, but still significant clusters. For very smooth data (e.g. dSPMs from MEG/EEG), you may just get one huge cluster if you use 0.05.

Date: Fri, 29 Jan 2010 14:08:04 -0500 From: greve@nmr.mgh.harvard.edu To: dhaglerjr@hotmail.com CC: freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] thickness maps: FDR versus Monte Carlo - different results

Don, why do you say that .05 is too liberal? We use a simulation-based test, not GRF, so we don't have to worry about the GRF assumptions breaking down at higher p-values.

doug

Don Hagler wrote:

...

Who says FDR is more conservative? It all depends on the thresholds you choose.

Note that the FDR threshold is different from the cluster-based p threshold, and it isn't necessarily appropriate to use the same for both.

The cluster-based p threshold controls the overall probability that you will get 1 or more false positives. With p<0.05, the idea is that if you repeated your study 20 times, 19 times out of 20, you would have NO false positives.

With an FDR threshold of 0.05, you are guaranteed that 5% of your "significant" vertices are false positives. A more conservative threshold would be 0.01.

By the way, for cluster thresholding, there are two p thresholds that are important. The first is the p threshold applied to the z-stats or t-stats before finding clusters. 0.05 would usually be too liberal. 0.01 or 0.001 would be better. The second is the multiple comparison corrected p value, for which you would usually use 0.05.

---

Date: Fri, 29 Jan 2010 10:02:45 -0500 From: stefan.brauns@googlemail.com To: Freesurfer@nmr.mgh.harvard.edu Subject: [Freesurfer] thickness maps: FDR versus Monte Carlo - different results

Hi there,

we are struggling with one question and haven't yet heard back. We would be very grateful for any advice or recommendation.

We were running an analysis (95 subjects) examining a group effect (two groups, controlling for the effects of a covariate and two cofactors, DODS model) on cortical thickness with mri_glmfit. We then corrected our results for multiple comparisons using two methods, FDR and Monte Carlo simulation with 4000 repeats. The threshold was set at p=0.05 respectively. Although examining the same population we got results in very different regions.

command for the simulation:

mri_glmfit-sim --glmdir xxxx.glmdir --sim mc-z 4000 1.301 mc-z abs.1.301 --sim-sign abs

the fdr results were obtained setting the fdr-threshold at p=0.05 with the tksurfer script command (sclv_set current_threshold_using_fdr 0.05 0)

With the fdr method we found a spot in the left supramarginal gyrus and with the clusterwise correction we got a cluster in the right middle temporal lobe (see pictures of corrected and uncorrected results attached). No other clusters/findings survived one of the correction methods.

We are well aware of the fact that FDR and Monte Carlo simulation are very different statistical methods and that FDR is more conservative. Does that explain the discrepant results? Would you expect a highly robust FDR finding to not show up at all when using Monte Carlo? What additional information could be used to decide which method to use for the final models?

Thank you,

Stefan

---

Hotmail: Free, trusted and rich email service. Get it now.

http://clk.atdmt.com/GBL/go/196390708/direct/01/

---

Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422

Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop: www.nmr.mgh.harvard.edu/facility/filedrop/index.html

---

Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

_________________________________________________________________ Hotmail: Trusted email with Microsoft’s powerful SPAM protection. http://clk.atdmt.com/GBL/go/196390706/direct/01/