Thank you all for the very good and detailed advice. We reran our cluster-wise correction after lowering the initial p-value (the one which is applied to the T-maps) to 0.01 or 0.001. Interestingly we then get the very same blob like in the FDR controlled approach (supramarginal gyrus). However, when using an initial p of 0.001 we loose the finding in the right middle temporal lobe. 
So I guess there are 2 remaining questions:
 
1)      What is an appropriate initial p values (to define the clusters before simulation)?
2)      Is there any way to use the proposed fancier methods like TFCE or cluster-mass method within freesurfer?
 
Thank you so much! Stefan

 

 
 
Message: 1
Date: Fri, 29 Jan 2010 14:57:34 -0800
From: Don Hagler <dhaglerjr@hotmail.com>
Subject: Re: [Freesurfer] thickness maps: FDR versus Monte Carlo -
different results
To: Doug Greve <greve@nmr.mgh.harvard.edu>
Cc: freesurfer maillist <freesurfer@nmr.mgh.harvard.edu>
Message-ID: <SNT129-W21E7C86C55DB35F30A018FA65B0@phx.gbl>
Content-Type: text/plain; charset="windows-1252"


In my experience, which is more in fMRI than thickness analyses, 0.05 can yield very large mega-clusters that should actually be considered a conglomeration of smaller clusters.  I found that regardless of whether one does Monte Carlo or GRF.  Because 0.05 for uncorrected p values is quite liberal, only the very large clusters survive the multiple comparisons correction.  So you can end up losing the smaller, but still significant clusters.  For very smooth data (e.g. dSPMs from MEG/EEG), you may just get one huge cluster if you use 0.05.



> Date: Fri, 29 Jan 2010 14:08:04 -0500
> From: greve@nmr.mgh.harvard.edu
> To: dhaglerjr@hotmail.com
> CC: freesurfer@nmr.mgh.harvard.edu
> Subject: Re: [Freesurfer] thickness maps: FDR versus Monte Carlo - different results
>
> Don, why do you say that  .05 is too liberal? We use a simulation-based
> test, not GRF, so we don't have to worry about the GRF assumptions
> breaking down at higher p-values.
>
> doug
>
> Don Hagler wrote:
> > Who says FDR is more conservative?  It all depends on the thresholds
> > you choose.
> >
> > Note that the FDR threshold is different from the cluster-based p
> > threshold, and it isn't necessarily appropriate to use the same for both.
> >
> > The cluster-based p threshold controls the overall probability that
> > you will get 1 or more false positives.  With p<0.05, the idea is that
> > if you repeated your study 20 times, 19 times out of 20, you would
> > have NO false positives.
> >
> > With an FDR threshold of 0.05, you are guaranteed that 5% of your
> > "significant" vertices are false positives.  A more conservative
> > threshold would be 0.01.
> >
> > By the way, for cluster thresholding, there are two p thresholds that
> > are important.  The first is the p threshold applied to the z-stats or
> > t-stats before finding clusters.  0.05 would usually be too liberal. 
> > 0.01 or 0.001 would be better.  The second is the multiple comparison
> > corrected p value, for which you would usually use 0.05.
> >
> > ------------------------------------------------------------------------
> > Date: Fri, 29 Jan 2010 10:02:45 -0500
> > From: stefan.brauns@googlemail.com
> > To: Freesurfer@nmr.mgh.harvard.edu
> > Subject: [Freesurfer] thickness maps: FDR versus Monte Carlo -
> > different results
> >
> > Hi there,
> >
> > we are struggling with one question and haven't yet heard back. We
> > would be very grateful for any advice or recommendation.
> >
> > We were running an analysis (95 subjects) examining a group effect
> > (two groups, controlling for the effects of a covariate and two
> > cofactors, DODS model) on cortical thickness with mri_glmfit. We then
> > corrected our results for multiple comparisons using two methods, FDR
> > and Monte Carlo simulation with 4000 repeats. The threshold was set at
> > p=0.05 respectively. Although examining the same population we got
> > results in very different regions.
> >
> > command for the simulation:
> >
> > mri_glmfit-sim --glmdir xxxx.glmdir --sim mc-z 4000 1.301 mc-z
> > abs.1.301 --sim-sign abs
> >
> > the fdr results were obtained setting the fdr-threshold at p=0.05 with
> > the tksurfer script command (sclv_set current_threshold_using_fdr 0.05 0)
> >
> > With the fdr method we found a spot in the left supramarginal gyrus
> > and with the clusterwise correction we got a cluster in the right
> > middle temporal lobe (see pictures of corrected and uncorrected
> > results attached). No other clusters/findings survived one of the
> > correction methods.
> >
> > We are well aware of the fact that FDR and Monte Carlo simulation are
> > very different statistical methods and that FDR is more conservative.
> > Does that explain the discrepant results? Would you expect a highly
> > robust FDR finding to not show up at all when using Monte Carlo? What
> > additional information could be used to decide which method to use for
> > the final models? 
> > 
> > Thank you,
> >
> > Stefan