Hello Freesurfer Experts, Recently there were two article published regarding clusterwise simulations for volumetric fmri analyses and potential errors for underestimating clusterwise extent thresholds.
1) http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes 2) biorxiv.org/content/early/2016/07/26/065862
One issue pointed out from these articles seems software specific, however the second issue is determining the proper clustersize. The heavy-tail nature of spatial smoothness seems to be ignored and a gaussian shape is generally assumed, leading to an underestimation of the spatial smoothness which can affect cluster size calculations. The issues are highlighted in the second article above.
I created my own monte carlo simulation in Freesurfer for a specific brain template and I wanted to find out if these concerns also apply to my surface based simulations? I am not sure if it does since the monte carlo tool is a GRF simulation as opposed to an analytic equation, however given that these articles were highlighted very recently, I wanted to ensure I am running things appropriately for surface based cortical thickness/dti analyses.
Thanks, Ajay
I have been doing simulations similar to #1 (Eklund) using surface-based analysis on both thickness and fMRI. I'll prepare a report of the results, but the early indications are that the same effect is in play, though it does not look like the effects are as bad as in Eklund.
For thickness analysis using applied smoothing of 5 or 10 mm FWHM, for a voxel-wise threshold of .001, the false positives are appropriate (ie, 5%). For a voxel-wise threshold of .01, the false positives is only a little off (about 7%); for a voxel-wise threshold of .05, the FPR is about 13%. If the data are not smoothed at all, then the false positive rates go way up. The reason appears to be the same as found in Eklund (ie, the autocorrelation function has a heavier-than-Gaussian tail). I did the analysis by randomly selecting 40 subjects from a homogeneous data set of 809 subjects aged 18-25. I then made two groups of 20 subjects each and ran a two-group test, then found clusters significant based on our Monte Carlo (Gaussian) simulations. I repeated this several thousand times. Any significant clusters were interpreted as false positives. These results are much better than Eklund, but Eklund was analyzing fMRI data.
I'm still working on the fMRI data. It is much more complicated because the results depend on the assumed stimulus schedule (eg, 10 sec blocks vs 30 sec blocks) and whether a one-group or two-group anaysis is done; nuisance variables also play a role. At very low cluster-forming thresholds (ie, .05), the FPR is roughly 20-30%. At a threshold of .01, the FPR is about 3-13%. At a threshold of .001 are about 1-6%. This is all for an applied smoothing level of 5mm.
All of these results are preliminary, so don't take them as true and established yet. As a reminder, you can always do a permutation test using mri_glmfit-sim. Eklund found that permutation did pretty well in most cases.
doug
On 8/2/16 12:43 AM, Ajay Kurani wrote:
Hello Freesurfer Experts, Recently there were two article published regarding clusterwise simulations for volumetric fmri analyses and potential errors for underestimating clusterwise extent thresholds.
- http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes
- biorxiv.org/content/early/2016/07/26/065862
http://biorxiv.org/content/early/2016/07/26/065862
One issue pointed out from these articles seems software specific, however the second issue is determining the proper clustersize. The heavy-tail nature of spatial smoothness seems to be ignored and a gaussian shape is generally assumed, leading to an underestimation of the spatial smoothness which can affect cluster size calculations. The issues are highlighted in the second article above.
I created my own monte carlo simulation in Freesurfer for a specific brain template and I wanted to find out if these concerns also apply to my surface based simulations? I am not sure if it does since the monte carlo tool is a GRF simulation as opposed to an analytic equation, however given that these articles were highlighted very recently, I wanted to ensure I am running things appropriately for surface based cortical thickness/dti analyses.
Thanks, Ajay
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Hi Doug, Thank you very much for your update regarding this issue.
1)Just curious, will LGI be included in this report as this is another analysis of interest?
2)As for the cortical thickness I originally used 15mm in the analysis so based on your email I think using 5-10mm may be more prudent in order to minimize FPR. From your email, I understand that mris_surf2surf (command I use to convert individual subject to fsaverage or template and smooth to 10-15mm) assumes an ACF estimation of smoothness which DOES NOT take into account the long tail distribution. Does this mean that when using mri_mcsim on my own template, the cluster extents for a given smoothness will be undersampled due to the fact that the "true" smoothness is more than what is estimated in the simulation, correct? For instance, when I select 15mm in qdec, it would point to the 21mm folder (fwhm.dat=20.8mm estimate), and I would select a given cluster extent for p=0.05. However, in this case, 15mm may translate to a larger FWHM than the estimated 21mm, correct?
3)You mentioned that I can use mri_glmfit-sim which is permutation testing based. I am struggling a bit in understanding how this differs from the simulation ran with mri_mcsim/qdec? Does qdec monte carlo simulation option run mri_glmfit-sim in the background to estimate the smoothness which looks up the cluster extent within the mri_mcsim based on the estimated FWHM? If so, is this estimate incorrect due to the fact that the long tails are not taken into account?
Thanks, Ajay
On Mon, Aug 1, 2016 at 11:43 PM, Ajay Kurani dr.ajay.kurani@gmail.com wrote:
Hello Freesurfer Experts, Recently there were two article published regarding clusterwise simulations for volumetric fmri analyses and potential errors for underestimating clusterwise extent thresholds.
- http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes
- biorxiv.org/content/early/2016/07/26/065862
One issue pointed out from these articles seems software specific, however the second issue is determining the proper clustersize. The heavy-tail nature of spatial smoothness seems to be ignored and a gaussian shape is generally assumed, leading to an underestimation of the spatial smoothness which can affect cluster size calculations. The issues are highlighted in the second article above.
I created my own monte carlo simulation in Freesurfer for a specific brain template and I wanted to find out if these concerns also apply to my surface based simulations? I am not sure if it does since the monte carlo tool is a GRF simulation as opposed to an analytic equation, however given that these articles were highlighted very recently, I wanted to ensure I am running things appropriately for surface based cortical thickness/dti analyses.
Thanks, Ajay
On 8/3/16 3:45 AM, Ajay Kurani wrote:
Hi Doug, Thank you very much for your update regarding this issue.
1)Just curious, will LGI be included in this report as this is another analysis of interest?
I was not planning to. The 809 subjects that I used for thickness do not have lGI run on them, but I think it is possible to run it on a subset without too much trouble. Not sure when I'll get to it.
2)As for the cortical thickness I originally used 15mm in the analysis so based on your email I think using 5-10mm may be more prudent in order to minimize FPR. From your email, I understand that mris_surf2surf (command I use to convert individual subject to fsaverage or template and smooth to 10-15mm) assumes an ACF estimation of smoothness which DOES NOT take into account the long tail distribution. Does this mean that when using mri_mcsim on my own template, the cluster extents for a given smoothness will be undersampled due to the fact that the "true" smoothness is more than what is estimated in the simulation, correct? For instance, when I select 15mm in qdec, it would point to the 21mm folder (fwhm.dat=20.8mm estimate), and I would select a given cluster extent for p=0.05. However, in this case, 15mm may translate to a larger FWHM than the estimated 21mm, correct?
Sort of. It is the Gaussian assumption that is incorrect, so there is no one FWHM that is correct (it is not a question of it simply being too small).
3)You mentioned that I can use mri_glmfit-sim which is permutation testing based. I am struggling a bit in understanding how this differs from the simulation ran with mri_mcsim/qdec? Does qdec monte carlo simulation option run mri_glmfit-sim in the background to estimate the smoothness which looks up the cluster extent within the mri_mcsim based on the estimated FWHM? If so, is this estimate incorrect due to the fact that the long tails are not taken into account?
Permutation uses your data and permutes (ie, randomly swaps) the class label associated with a subject, the data with this new labeling are analyzed and clusters computed. Under the null, the label is irrelevant, so clusters are interpreted as false positives. After several thousand of these swaps, one builds a list of the probability of seeing a cluster of a certain size under the null, and this is used to generate the p-value. Permutation will then naturally take into account all the non-Gaussian aspects of the data. The monte carlo (MC) simulation is similar, but it uses smoothed synthesized gaussian noise instead of the real data and so the gaussian assumption is built into it.
Thanks, Ajay
On Mon, Aug 1, 2016 at 11:43 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hello Freesurfer Experts, Recently there were two article published regarding clusterwise simulations for volumetric fmri analyses and potential errors for underestimating clusterwise extent thresholds. 1) http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes 2) biorxiv.org/content/early/2016/07/26/065862 <http://biorxiv.org/content/early/2016/07/26/065862> One issue pointed out from these articles seems software specific, however the second issue is determining the proper clustersize. The heavy-tail nature of spatial smoothness seems to be ignored and a gaussian shape is generally assumed, leading to an underestimation of the spatial smoothness which can affect cluster size calculations. The issues are highlighted in the second article above. I created my own monte carlo simulation in Freesurfer for a specific brain template and I wanted to find out if these concerns also apply to my surface based simulations? I am not sure if it does since the monte carlo tool is a GRF simulation as opposed to an analytic equation, however given that these articles were highlighted very recently, I wanted to ensure I am running things appropriately for surface based cortical thickness/dti analyses. Thanks, Ajay
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Hi Doug, I had some additional questions regarding multiple comparisons in Freesurfer.
1) Do you correct the left and right hemispheres separately or combine both together for muliple comparison correction?
2) Say you are testing multiple contrasts in your model: A > B, A< B etc. Do you correct for multiple contrasts and if not, is there any particular reason why not.
Thanks, Ajay
On Wed, Aug 3, 2016 at 2:45 AM, Ajay Kurani dr.ajay.kurani@gmail.com wrote:
Hi Doug, Thank you very much for your update regarding this issue.
1)Just curious, will LGI be included in this report as this is another analysis of interest?
2)As for the cortical thickness I originally used 15mm in the analysis so based on your email I think using 5-10mm may be more prudent in order to minimize FPR. From your email, I understand that mris_surf2surf (command I use to convert individual subject to fsaverage or template and smooth to 10-15mm) assumes an ACF estimation of smoothness which DOES NOT take into account the long tail distribution. Does this mean that when using mri_mcsim on my own template, the cluster extents for a given smoothness will be undersampled due to the fact that the "true" smoothness is more than what is estimated in the simulation, correct? For instance, when I select 15mm in qdec, it would point to the 21mm folder (fwhm.dat=20.8mm estimate), and I would select a given cluster extent for p=0.05. However, in this case, 15mm may translate to a larger FWHM than the estimated 21mm, correct?
3)You mentioned that I can use mri_glmfit-sim which is permutation testing based. I am struggling a bit in understanding how this differs from the simulation ran with mri_mcsim/qdec? Does qdec monte carlo simulation option run mri_glmfit-sim in the background to estimate the smoothness which looks up the cluster extent within the mri_mcsim based on the estimated FWHM? If so, is this estimate incorrect due to the fact that the long tails are not taken into account?
Thanks, Ajay
On Mon, Aug 1, 2016 at 11:43 PM, Ajay Kurani dr.ajay.kurani@gmail.com wrote:
Hello Freesurfer Experts, Recently there were two article published regarding clusterwise simulations for volumetric fmri analyses and potential errors for underestimating clusterwise extent thresholds.
- http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes
- biorxiv.org/content/early/2016/07/26/065862
One issue pointed out from these articles seems software specific, however the second issue is determining the proper clustersize. The heavy-tail nature of spatial smoothness seems to be ignored and a gaussian shape is generally assumed, leading to an underestimation of the spatial smoothness which can affect cluster size calculations. The issues are highlighted in the second article above.
I created my own monte carlo simulation in Freesurfer for a specific brain template and I wanted to find out if these concerns also apply to my surface based simulations? I am not sure if it does since the monte carlo tool is a GRF simulation as opposed to an analytic equation, however given that these articles were highlighted very recently, I wanted to ensure I am running things appropriately for surface based cortical thickness/dti analyses.
Thanks, Ajay
On 10/17/16 2:16 AM, Ajay Kurani wrote:
Hi Doug, I had some additional questions regarding multiple comparisons in Freesurfer.
- Do you correct the left and right hemispheres separately or combine
both together for muliple comparison correction?
We correct each hemi independently and then bonferroni correct both. Eg, if you have a cluster in the left hemi that has a pvalue = .01, then we would multiply the pvalue by 2 to account for both hemis. When you run mri_glmfit-sim with --2space, it will do this for you automatically.
- Say you are testing multiple contrasts in your model: A > B, A< B
etc. Do you correct for multiple contrasts and if not, is there any particular reason why not.
I think you should, but most people probably don't. This is an unsigned test. In FS, you would specify "abs" as the sign (absolute, vs pos (positive) or neg), and FS will do the right correction. In other packages, eg, FSL, you must specify an F-test (even then it might not do the right thing).
Thanks, Ajay
On Wed, Aug 3, 2016 at 2:45 AM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Doug, Thank you very much for your update regarding this issue. 1)Just curious, will LGI be included in this report as this is another analysis of interest? 2)As for the cortical thickness I originally used 15mm in the analysis so based on your email I think using 5-10mm may be more prudent in order to minimize FPR. From your email, I understand that mris_surf2surf (command I use to convert individual subject to fsaverage or template and smooth to 10-15mm) assumes an ACF estimation of smoothness which DOES NOT take into account the long tail distribution. Does this mean that when using mri_mcsim on my own template, the cluster extents for a given smoothness will be undersampled due to the fact that the "true" smoothness is more than what is estimated in the simulation, correct? For instance, when I select 15mm in qdec, it would point to the 21mm folder (fwhm.dat=20.8mm estimate), and I would select a given cluster extent for p=0.05. However, in this case, 15mm may translate to a larger FWHM than the estimated 21mm, correct? 3)You mentioned that I can use mri_glmfit-sim which is permutation testing based. I am struggling a bit in understanding how this differs from the simulation ran with mri_mcsim/qdec? Does qdec monte carlo simulation option run mri_glmfit-sim in the background to estimate the smoothness which looks up the cluster extent within the mri_mcsim based on the estimated FWHM? If so, is this estimate incorrect due to the fact that the long tails are not taken into account? Thanks, Ajay On Mon, Aug 1, 2016 at 11:43 PM, Ajay Kurani <dr.ajay.kurani@gmail.com <mailto:dr.ajay.kurani@gmail.com>> wrote: Hello Freesurfer Experts, Recently there were two article published regarding clusterwise simulations for volumetric fmri analyses and potential errors for underestimating clusterwise extent thresholds. 1) http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes <http://www.pnas.org/content/113/28/7900.full.pdf?with-ds=yes> 2) biorxiv.org/content/early/2016/07/26/065862 <http://biorxiv.org/content/early/2016/07/26/065862> One issue pointed out from these articles seems software specific, however the second issue is determining the proper clustersize. The heavy-tail nature of spatial smoothness seems to be ignored and a gaussian shape is generally assumed, leading to an underestimation of the spatial smoothness which can affect cluster size calculations. The issues are highlighted in the second article above. I created my own monte carlo simulation in Freesurfer for a specific brain template and I wanted to find out if these concerns also apply to my surface based simulations? I am not sure if it does since the monte carlo tool is a GRF simulation as opposed to an analytic equation, however given that these articles were highlighted very recently, I wanted to ensure I am running things appropriately for surface based cortical thickness/dti analyses. Thanks, Ajay
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
freesurfer@nmr.mgh.harvard.edu
-
Ajay Kurani -
Douglas Greve -
Douglas N Greve