Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post: http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong...
1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education.
2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16
Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0:
cmdline mri_glmfit.bin --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Female-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group-Intercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y /home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.mgh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016
Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on?
b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails
c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016
I am not sure if this means my simulation is incorrect?
Thanks, Ajay
Can you tar up the glmdir and drop it to me at our filedrop (link below)? Also,what version of FS are you using? Also, can you try running it without backgrounding on a few (say, 100) iterations to see if it completes? Then try with backgrounding (problem may be backgrounding)
On 08/25/2016 01:18 PM, Ajay Kurani wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post: http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong...
- I first ran QDec to generate a folder for the analysis which would
create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education.
- I ran the following command:
/mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16
Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0:
cmdline mri_glmfit.bin --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Female-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group-Intercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y /home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.mgh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016
Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on?
b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails
c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016
I am not sure if this means my simulation is incorrect?
Thanks, Ajay
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Something strange is going on in your data. You are using fsaverage, but the size of the data in PDvsMCI_lh_thickness_10mm/y.mgh does not match (it has 148278 vertices, not 163842 of fsaverage). This is causing the processes to die, but the backgrounding software does not know that the process died and so just keeps polling.
What I don't understand is why the original call to mri_glmfit did not die too. When I run it as such, it does die. It may be that the data in that folder did not get deleted by QDEC, and there is a strange mix of data in there. Can you delete that folder and try it again? Also, don't run it in background mode until you know that it is working.
On 08/25/2016 01:18 PM, Ajay Kurani wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post: http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong...
- I first ran QDec to generate a folder for the analysis which would
create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education.
- I ran the following command:
/mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16
Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0:
cmdline mri_glmfit.bin --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Female-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group-Intercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y /home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.mgh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016
Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on?
b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails
c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016
I am not sure if this means my simulation is incorrect?
Thanks, Ajay
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Hi Doug, Thanks for the help! I think I figured out the issue based on your response. 1) I created a template to use for this group and named it fsaverage (including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
2) I deleted the folder and restarted without background processes. The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani dr.ajay.kurani@gmail.com wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post: http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard. edu/msg48653.html )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong...
- I first ran QDec to generate a folder for the analysis which would
create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education.
- I ran the following command:
/mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16
Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0:
cmdline mri_glmfit.bin --C ./HCvsPAT_lh_thickness_10mm// tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh- Diff-Male-Female-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm// tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X- Gender-Group-Intercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y /home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.mgh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016
Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on?
b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails
c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016
I am not sure if this means my simulation is incorrect?
Thanks, Ajay
On 08/25/2016 05:13 PM, Ajay Kurani wrote:
Hi Doug, Thanks for the help! I think I figured out the issue based on your response.
- I created a template to use for this group and named it fsaverage
(including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
- I deleted the folder and restarted without background processes.
The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
There are ways to do this, but I don't have them implemented yet. See here http://www.ncbi.nlm.nih.gov/pubmed/24530839
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
No,non-orthogonalality has no effect on the monte carlo analyis
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post: http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html <http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html> ) Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong... 1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education. 2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16 Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0: cmdline mri_glmfit.bin --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Female-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group-Intercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y /home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.mgh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013 3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016 Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on? b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 I am not sure if this means my simulation is incorrect? Thanks, Ajay
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Hi all,
Is using forced permutation for non-orthogonal design matrices wrong or is it allowed to do this instead of using tools like palm (what happens eg with the covariates when using forced permutation)? I used forced permutation and it seemed to work, results were (partly) comparable to what I found with monte carlo simulations.
Thanks, Maaike
________________________________ Van: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu namens Harms, Michael mharms@wustl.edu Verzonden: vrijdag 26 augustus 2016 01:00:13 Aan: Freesurfer support list Onderwerp: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi, You might want to check out FSL’s PALM tool, which has a bit more sophisticated permutation framework, and allows for permutation in the context of non-orthogonal covariates.
cheers, -MH
-- Michael Harms, Ph.D. ----------------------------------------------------------- Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave. Tel: 314-747-6173 St. Louis, MO 63110 Email: mharms@wustl.edu
From: <freesurfer-bounces@nmr.mgh.harvard.edumailto:freesurfer-bounces@nmr.mgh.harvard.edu> on behalf of Ajay Kurani <dr.ajay.kurani@gmail.commailto:dr.ajay.kurani@gmail.com> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edumailto:freesurfer@nmr.mgh.harvard.edu> Date: Thursday, August 25, 2016 at 4:13 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edumailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi Doug, Thanks for the help! I think I figured out the issue based on your response. 1) I created a template to use for this group and named it fsaverage (including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
2) I deleted the folder and restarted without background processes. The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.commailto:dr.ajay.kurani@gmail.com> wrote: Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post: http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong...
1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education.
2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16
Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0:
cmdline mri_glmfit.bin --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Female-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group-Intercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y /home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.mgh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016
Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on?
b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails
c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016
I am not sure if this means my simulation is incorrect?
Thanks, Ajay
________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
It is hard to say. Since the subjects are not exchangeable, the permutation is technically not appropriate. Check the winkler paper, I think he talks about what happens if you just don't do anything.
On 08/29/2016 11:07 AM, maaike rive wrote:
Hi all,
Is using forced permutation for non-orthogonal design matrices wrong or is it allowed to do this instead of using tools like palm (what happens eg with the covariates when using forced permutation)? I used forced permutation and it seemed to work, results were (partly) comparable to what I found with monte carlo simulations.
Thanks, Maaike
*Van:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu namens Harms, Michael mharms@wustl.edu *Verzonden:* vrijdag 26 augustus 2016 01:00:13 *Aan:* Freesurfer support list *Onderwerp:* Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi, You might want to check out FSL’s PALM tool, which has a bit more sophisticated permutation framework, and allows for permutation in the context of non-orthogonal covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
From: <freesurfer-bounces@nmr.mgh.harvard.edu mailto:freesurfer-bounces@nmr.mgh.harvard.edu> on behalf of Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Date: Thursday, August 25, 2016 at 4:13 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi Doug, Thanks for the help! I think I figured out the issue based on your response.
- I created a template to use for this group and named it fsaverage
(including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
- I deleted the folder and restarted without background processes.
The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post: http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html <http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html> ) Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong... 1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education. 2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16 Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0: cmdline mri_glmfit.bin --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Female-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Intercept-thickness.mtx --C ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group-Intercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y /home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.mgh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013 3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016 Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on? b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 I am not sure if this means my simulation is incorrect? Thanks, Ajay
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Hi Maaike, Why not just use PALM? Then you don’t have to worry about this (since PALM appropriately handles the situation of correlated covariates).
cheers, -MH
-- Michael Harms, Ph.D.
----------------------------------------------------------- Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 4:45 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
It is hard to say. Since the subjects are not exchangeable, the permutation is technically not appropriate. Check the winkler paper, I think he talks about what happens if you just don't do anything.
On 08/29/2016 11:07 AM, maaike rive wrote:
Hi all,
Is using forced permutation for non-orthogonal design matrices wrong or is it allowed to do this instead of using tools like palm (what happens eg with the covariates when using forced permutation)? I used forced permutation and it seemed to work, results were (partly) comparable to what I found with monte carlo simulations.
Thanks, Maaike
*Van:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu namens Harms, Michael mharms@wustl.edu *Verzonden:* vrijdag 26 augustus 2016 01:00:13 *Aan:* Freesurfer support list *Onderwerp:* Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi, You might want to check out FSL’s PALM tool, which has a bit more sophisticated permutation framework, and allows for permutation in the context of non-orthogonal covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
From: <freesurfer-bounces@nmr.mgh.harvard.edu mailto:freesurfer-bounces@nmr.mgh.harvard.edu> on behalf of Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Date: Thursday, August 25, 2016 at 4:13 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi Doug, Thanks for the help! I think I figured out the issue based on your response.
- I created a template to use for this group and named it fsaverage
(including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
- I deleted the folder and restarted without background processes.
The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post:http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html
http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.htm l )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong... 1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education. 2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16 Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0: cmdline mri_glmfit.bin --C./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-thi ckness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Female- Intercept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Inter cept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group-In tercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y
/home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.mgh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016 Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on? b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 I am not sure if this means my simulation is incorrect? Thanks, Ajay
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop:https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing:ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Does PALM do surface-based? Also, there is no way to appropriately handle this. For permutation, non-orthogonal designs are wrong. There are ways to try to compensate for it, which is what PALM is doing. Sorry to be nit-picky!
On 08/29/2016 06:12 PM, Harms, Michael wrote:
Hi Maaike, Why not just use PALM? Then you don’t have to worry about this (since PALM appropriately handles the situation of correlated covariates).
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 4:45 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
It is hard to say. Since the subjects are not exchangeable, the permutation is technically not appropriate. Check the winkler paper, I think he talks about what happens if you just don't do anything.
On 08/29/2016 11:07 AM, maaike rive wrote:
Hi all,
Is using forced permutation for non-orthogonal design matrices wrong or is it allowed to do this instead of using tools like palm (what happens eg with the covariates when using forced permutation)? I used forced permutation and it seemed to work, results were (partly) comparable to what I found with monte carlo simulations.
Thanks, Maaike
*Van:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu namens Harms, Michael mharms@wustl.edu *Verzonden:* vrijdag 26 augustus 2016 01:00:13 *Aan:* Freesurfer support list *Onderwerp:* Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi, You might want to check out FSL’s PALM tool, which has a bit more sophisticated permutation framework, and allows for permutation in the context of non-orthogonal covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
From: <freesurfer-bounces@nmr.mgh.harvard.edu mailto:freesurfer-bounces@nmr.mgh.harvard.edu> on behalf of Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Date: Thursday, August 25, 2016 at 4:13 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi Doug, Thanks for the help! I think I figured out the issue based on your response.
- I created a template to use for this group and named it fsaverage
(including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
- I deleted the folder and restarted without background processes.
The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post:http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.html
http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.htm l )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong... 1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education. 2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16 Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0: cmdline mri_glmfit.bin --C./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-thi ckness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Female- Intercept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Inter cept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group-In tercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y
/home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.mgh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016 Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on? b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 I am not sure if this means my simulation is incorrect? Thanks, Ajay
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop:https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing:ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
PALM handles GIFTI and CIFTI data.
Peace,
Matt.
On 8/29/16, 6:21 PM, "Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
Does PALM do surface-based? Also, there is no way to appropriately handle this. For permutation, non-orthogonal designs are wrong. There are ways to try to compensate for it, which is what PALM is doing. Sorry to be nit-picky!
On 08/29/2016 06:12 PM, Harms, Michael wrote:
Hi Maaike, Why not just use PALM? Then you don¹t have to worry about this (since PALM appropriately handles the situation of correlated covariates).
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 4:45 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
It is hard to say. Since the subjects are not exchangeable, the permutation is technically not appropriate. Check the winkler paper, I think he talks about what happens if you just don't do anything.
On 08/29/2016 11:07 AM, maaike rive wrote:
Hi all,
Is using forced permutation for non-orthogonal design matrices wrong or is it allowed to do this instead of using tools like palm (what happens eg with the covariates when using forced permutation)? I used forced permutation and it seemed to work, results were (partly) comparable to what I found with monte carlo simulations.
Thanks, Maaike
*Van:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu namens Harms, Michael mharms@wustl.edu *Verzonden:* vrijdag 26 augustus 2016 01:00:13 *Aan:* Freesurfer support list *Onderwerp:* Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi, You might want to check out FSL¹s PALM tool, which has a bit more sophisticated permutation framework, and allows for permutation in the context of non-orthogonal covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
From: <freesurfer-bounces@nmr.mgh.harvard.edu mailto:freesurfer-bounces@nmr.mgh.harvard.edu> on behalf of Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Date: Thursday, August 25, 2016 at 4:13 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi Doug, Thanks for the help! I think I figured out the issue based on your response.
- I created a template to use for this group and named it fsaverage
(including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
- I deleted the folder and restarted without background processes.
The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post:http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.ht ml
http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.h tm l )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong... 1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education. 2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16 Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0: cmdline mri_glmfit.bin --C./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-t hi ckness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Femal e- Intercept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Int er cept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group- In tercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y
/home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.m gh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016 Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on? b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 I am not sure if this means my simulation is incorrect? Thanks, Ajay
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop:https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing:ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Hi, I wouldn’t say that non-orthogonal designs are “wrong” to use with permutation. Rather, there are different approaches to handling that situation and produce approximate p-values. See Table 2 in Winkler’s 2014 paper, and the results therein comparing the various approaches:
http://www.ncbi.nlm.nih.gov/pubmed/24530839
PALM actually gives you control over the method used, with the default (and recommended) approach being that of “Freedman-Lane", which is the same approach used by FSL’s ‘randomise’ tool to handle correlated covariates.
cheers, -MH
-- Michael Harms, Ph.D.
----------------------------------------------------------- Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 7:49 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Matt Glasser" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of matt@ma-tea.com> wrote:
PALM handles GIFTI and CIFTI data.
Peace,
Matt.
On 8/29/16, 6:21 PM, "Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
Does PALM do surface-based? Also, there is no way to appropriately handle this. For permutation, non-orthogonal designs are wrong. There are ways to try to compensate for it, which is what PALM is doing. Sorry to be nit-picky!
On 08/29/2016 06:12 PM, Harms, Michael wrote:
Hi Maaike, Why not just use PALM? Then you don¹t have to worry about this (since PALM appropriately handles the situation of correlated covariates).
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 4:45 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
It is hard to say. Since the subjects are not exchangeable, the permutation is technically not appropriate. Check the winkler paper, I think he talks about what happens if you just don't do anything.
On 08/29/2016 11:07 AM, maaike rive wrote:
Hi all,
Is using forced permutation for non-orthogonal design matrices wrong or is it allowed to do this instead of using tools like palm (what happens eg with the covariates when using forced permutation)? I used forced permutation and it seemed to work, results were (partly) comparable to what I found with monte carlo simulations.
Thanks, Maaike
*Van:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu namens Harms, Michael mharms@wustl.edu *Verzonden:* vrijdag 26 augustus 2016 01:00:13 *Aan:* Freesurfer support list *Onderwerp:* Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi, You might want to check out FSL¹s PALM tool, which has a bit more sophisticated permutation framework, and allows for permutation in the context of non-orthogonal covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
From: <freesurfer-bounces@nmr.mgh.harvard.edu mailto:freesurfer-bounces@nmr.mgh.harvard.edu> on behalf of Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Date: Thursday, August 25, 2016 at 4:13 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi Doug, Thanks for the help! I think I figured out the issue based on your response.
- I created a template to use for this group and named it fsaverage
(including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
- I deleted the folder and restarted without background processes.
The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post:http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.ht ml
http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.h tm l )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong... 1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education. 2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16 Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0: cmdline mri_glmfit.bin --C./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-t hi ckness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Femal e- Intercept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Int er cept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group- In tercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y
/home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.m gh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016 Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on? b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 I am not sure if this means my simulation is incorrect? Thanks, Ajay
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop:https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing:ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
By "wrong" I meant that permutation no longer gives exact p-values in expectation with non-orthogonal designs. There is no theory to characterize the accuracy of Freeman-Lane of the other methods. In this sense, they are not approximations but ad hoc methods that people hope do a better job than parametric methods. Anderson tested them on a wide range of designs, but it was, of course, not exhaustive. The accuracy of his results may not extend to other designs, so it is a buyer-beware situation (as with all neuroimaging).
On 8/29/16 10:16 PM, Harms, Michael wrote:
Hi, I wouldn’t say that non-orthogonal designs are “wrong” to use with permutation. Rather, there are different approaches to handling that situation and produce approximate p-values. See Table 2 in Winkler’s 2014 paper, and the results therein comparing the various approaches:
http://www.ncbi.nlm.nih.gov/pubmed/24530839
PALM actually gives you control over the method used, with the default (and recommended) approach being that of “Freedman-Lane", which is the same approach used by FSL’s ‘randomise’ tool to handle correlated covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 7:49 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Matt Glasser" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of matt@ma-tea.com> wrote:
PALM handles GIFTI and CIFTI data.
Peace,
Matt.
On 8/29/16, 6:21 PM, "Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
Does PALM do surface-based? Also, there is no way to appropriately handle this. For permutation, non-orthogonal designs are wrong. There are ways to try to compensate for it, which is what PALM is doing. Sorry to be nit-picky!
On 08/29/2016 06:12 PM, Harms, Michael wrote:
Hi Maaike, Why not just use PALM? Then you don¹t have to worry about this (since PALM appropriately handles the situation of correlated covariates).
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 4:45 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
It is hard to say. Since the subjects are not exchangeable, the permutation is technically not appropriate. Check the winkler paper, I think he talks about what happens if you just don't do anything.
On 08/29/2016 11:07 AM, maaike rive wrote:
Hi all,
Is using forced permutation for non-orthogonal design matrices wrong or is it allowed to do this instead of using tools like palm (what happens eg with the covariates when using forced permutation)? I used forced permutation and it seemed to work, results were (partly) comparable to what I found with monte carlo simulations.
Thanks, Maaike
*Van:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu namens Harms, Michael mharms@wustl.edu *Verzonden:* vrijdag 26 augustus 2016 01:00:13 *Aan:* Freesurfer support list *Onderwerp:* Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi, You might want to check out FSL¹s PALM tool, which has a bit more sophisticated permutation framework, and allows for permutation in the context of non-orthogonal covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
From: <freesurfer-bounces@nmr.mgh.harvard.edu mailto:freesurfer-bounces@nmr.mgh.harvard.edu> on behalf of Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Date: Thursday, August 25, 2016 at 4:13 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi Doug, Thanks for the help! I think I figured out the issue based on your response.
- I created a template to use for this group and named it fsaverage
(including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
- I deleted the folder and restarted without background processes.
The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommended by Doug in my previous post:http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.ht ml
http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.h tm l )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong... 1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The following is for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education. 2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16 Prior to running the command above, from the y.fsdg file I deleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0: cmdline mri_glmfit.bin --C./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-t hi ckness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Femal e- Intercept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Int er cept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group- In tercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y
/home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.m gh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete but currently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016 Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do with the number of iterations it is on? b) Did I run this procedure correctly? Was I incorrect in deleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 I am not sure if this means my simulation is incorrect? Thanks, Ajay
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop:https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing:ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Hi all,
In the presence of nuisance (orthogonal or not) the permutation test is only approximate, whatever is the method used. Regressing out the nuisance introduces dependencies between the residuals that render them strictly not exchangeable. However, in practice this is not an issue, and the FL method has been assessed extensively by us and also by others (e.g., Anderson & Legendre 1999; Anderson & Robinson, 2001; Anderson & ter Braak, 2003; among others) and does lead to error rates that approach the test level.
In the Winkler et al 2014 paper, Table 7 gives a sense of how good or bad some of the regression and permutation methods can be, and even the FL method, that we advocate, can lead to error rates below or above the 95% CI, although the chances of invalid results are very small. This is for a single test. When we consider the thousands of voxels/vertices, this gets further diluted in the distribution of the maximum, such that the FWER is even closer to perfectly exact (this information isn't in the paper, though).
In the presence of non-orthogonality between regressors of interest and nuisance, the result remains approximately exact, only the power to detect a true effect is reduced, as it cannot be disambiguated from the nuisance.
Cheers,
Anderson
On 30 August 2016 at 09:33, Douglas Greve greve@nmr.mgh.harvard.edu wrote:
By "wrong" I meant that permutation no longer gives exact p-values in expectation with non-orthogonal designs. There is no theory to characterize the accuracy of Freeman-Lane of the other methods. In this sense, they are not approximations but ad hoc methods that people hope do a better job than parametric methods. Anderson tested them on a wide range of designs, but it was, of course, not exhaustive. The accuracy of his results may not extend to other designs, so it is a buyer-beware situation (as with all neuroimaging).
On 8/29/16 10:16 PM, Harms, Michael wrote:
Hi, I wouldn’t say that non-orthogonal designs are “wrong” to use with permutation. Rather, there are different approaches to handling that situation and produce approximate p-values. See Table 2 in Winkler’s
2014
paper, and the results therein comparing the various approaches:
http://www.ncbi.nlm.nih.gov/pubmed/24530839
PALM actually gives you control over the method used, with the default (and recommended) approach being that of “Freedman-Lane", which is the same approach used by FSL’s ‘randomise’ tool to handle correlated covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 7:49 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf
of
Matt Glasser" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of matt@ma-tea.com> wrote:
PALM handles GIFTI and CIFTI data.
Peace,
Matt.
On 8/29/16, 6:21 PM, "Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
Does PALM do surface-based? Also, there is no way to appropriately handle this. For permutation, non-orthogonal designs are wrong. There are ways to try to compensate for it, which is what PALM is doing. Sorry to be nit-picky!
On 08/29/2016 06:12 PM, Harms, Michael wrote:
Hi Maaike, Why not just use PALM? Then you don¹t have to worry about this (since PALM appropriately handles the situation of correlated covariates).
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 4:45 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
It is hard to say. Since the subjects are not exchangeable, the permutation is technically not appropriate. Check the winkler paper, I think he talks about what happens if you just don't do anything.
On 08/29/2016 11:07 AM, maaike rive wrote:
Hi all,
Is using forced permutation for non-orthogonal design matrices wrong or is it allowed to do this instead of using tools like palm (what happens eg with the covariates when using forced permutation)? I used forced permutation and it seemed to work, results were (partly) comparable to what I found with monte carlo simulations.
Thanks, Maaike
*Van:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu namens Harms, Michael mharms@wustl.edu *Verzonden:* vrijdag 26 augustus 2016 01:00:13 *Aan:* Freesurfer support list *Onderwerp:* Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi, You might want to check out FSL¹s PALM tool, which has a bit more sophisticated permutation framework, and allows for permutation in the context of non-orthogonal covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
From: <freesurfer-bounces@nmr.mgh.harvard.edu mailto:freesurfer-bounces@nmr.mgh.harvard.edu> on behalf of Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Date: Thursday, August 25, 2016 at 4:13 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi Doug, Thanks for the help! I think I figured out the issue based on
your
response.
- I created a template to use for this group and named it fsaverage
(including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
- I deleted the folder and restarted without background processes.
The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommendedby
Doug in my previous post:edu/msg48653.ht
ml
edu/msg48653.h
tm l> )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong... 1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The followingis
for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education. 2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16 Prior to running the command above, from the y.fsdg file Ideleted
the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing thisestimate,
it would force the permutation test to calculate based on thedata
but when looking at the log output I see the following whichsays
fwhm 0: cmdline mri_glmfit.bin --C./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-
Avg-Intercept-t
hi ckness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-
Diff-Male-Femal
e- Intercept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-
Diff-PD-MCI-Int
er cept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-
Gender-Group-
In tercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//
csd/perm.abs.2.j013
--y/home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_
thickness_10mm/y.m
gh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs
--fwhm
0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surffsaverage
lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013 3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete butcurrently
I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016 Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do withthe
number of iterations it is on? b) Did I run this procedure correctly? Was I incorrect indeleting
the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 I am not sure if this means my simulation is incorrect? Thanks, Ajay
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop:https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing:ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/
flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/
greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The materials in this message are private and may contain Protected
Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Sure, any simulation can’t be exhaustive, but my understanding, chatting with Anderson, is that their 2014 paper simulated a quite large space of designs and Freedman-Lane is appropriate for the vast majority of the GLMs that we would encounter in neuroimaging.
Also, again per Anderson, I believe that method implemented in mri_glmfit-sim is that of Manly for those interested in linking what mri_glmfit-sim is doing to the results in Winkler (e.g., Table 7).
cheers, -MH
-- Michael Harms, Ph.D.
----------------------------------------------------------- Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/30/16, 8:33 AM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
By "wrong" I meant that permutation no longer gives exact p-values in expectation with non-orthogonal designs. There is no theory to characterize the accuracy of Freeman-Lane of the other methods. In this sense, they are not approximations but ad hoc methods that people hope do a better job than parametric methods. Anderson tested them on a wide range of designs, but it was, of course, not exhaustive. The accuracy of his results may not extend to other designs, so it is a buyer-beware situation (as with all neuroimaging).
On 8/29/16 10:16 PM, Harms, Michael wrote:
Hi, I wouldn’t say that non-orthogonal designs are “wrong” to use with permutation. Rather, there are different approaches to handling that situation and produce approximate p-values. See Table 2 in Winkler’s 2014 paper, and the results therein comparing the various approaches:
http://www.ncbi.nlm.nih.gov/pubmed/24530839
PALM actually gives you control over the method used, with the default (and recommended) approach being that of “Freedman-Lane", which is the same approach used by FSL’s ‘randomise’ tool to handle correlated covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 7:49 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Matt Glasser" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of matt@ma-tea.com> wrote:
PALM handles GIFTI and CIFTI data.
Peace,
Matt.
On 8/29/16, 6:21 PM, "Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
Does PALM do surface-based? Also, there is no way to appropriately handle this. For permutation, non-orthogonal designs are wrong. There are ways to try to compensate for it, which is what PALM is doing. Sorry to be nit-picky!
On 08/29/2016 06:12 PM, Harms, Michael wrote:
Hi Maaike, Why not just use PALM? Then you don¹t have to worry about this (since PALM appropriately handles the situation of correlated covariates).
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
On 8/29/16, 4:45 PM, "freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N Greve" <freesurfer-bounces@nmr.mgh.harvard.edu on behalf of greve@nmr.mgh.harvard.edu> wrote:
It is hard to say. Since the subjects are not exchangeable, the permutation is technically not appropriate. Check the winkler paper, I think he talks about what happens if you just don't do anything.
On 08/29/2016 11:07 AM, maaike rive wrote:
Hi all,
Is using forced permutation for non-orthogonal design matrices wrong or is it allowed to do this instead of using tools like palm (what happens eg with the covariates when using forced permutation)? I used forced permutation and it seemed to work, results were (partly) comparable to what I found with monte carlo simulations.
Thanks, Maaike
*Van:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu namens Harms, Michael mharms@wustl.edu *Verzonden:* vrijdag 26 augustus 2016 01:00:13 *Aan:* Freesurfer support list *Onderwerp:* Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi, You might want to check out FSL¹s PALM tool, which has a bit more sophisticated permutation framework, and allows for permutation in the context of non-orthogonal covariates.
cheers, -MH
-- Michael Harms, Ph.D.
Conte Center for the Neuroscience of Mental Disorders Washington University School of Medicine Department of Psychiatry, Box 8134 660 South Euclid Ave.Tel: 314-747-6173 St. Louis, MO 63110Email: mharms@wustl.edu
From: <freesurfer-bounces@nmr.mgh.harvard.edu mailto:freesurfer-bounces@nmr.mgh.harvard.edu> on behalf of Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Date: Thursday, August 25, 2016 at 4:13 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running after 3 days!
Hi Doug, Thanks for the help! I think I figured out the issue based on your response.
- I created a template to use for this group and named it fsaverage
(including creating monte carlo simulations) for simplicity of integrating with freesurfer as I am newer to it. This is why the sizes didn't match up as you expected but the mri_glmfit still ran.
- I deleted the folder and restarted without background processes.
The error became apparent. Of my covariates (2 fix factors and 3 quantitative), not all were orthogonal. In looking at the error more, it seems that i need to add the --perm-force if I wanted the simulation to run, however the background processes were not aware of this error and kept polling as you mentioned.
This brings me to a new but related issue. From what I have read in other freesurfer posts, it is statistically incorrect to use --perm-force for non-orthogonal covariates (or continuous covariates). I am unsure how to proceed. a) If I ran permutation testing (to overcome the issue of incorrect smoothness estimations from the gaussian distribution assumption), then I run into the issue of non-orthogonal covariates. Is there a way to orthogonalize the data in freesurfer, or a solution to this issue?
b) If orthogonalizing is difficult to implement, another option is running Qdec with the montecarlo simulation at a more conservative p value (p< 0.001). From your previous posts, the testing at this p value for 10mm seems to meet the 5% FPR. One question is if the non-orthogonal data affects this analysis as well for this model?
Thanks, Ajay
On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani <dr.ajay.kurani@gmail.com mailto:dr.ajay.kurani@gmail.com> wrote:
Hi Freesurfer Experts, I am trying to use freesurfer's mri_glmfit-sim tool to run permutation testing on cortical thickness data (as recommendedby Doug in my previous post:
http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.h t ml
http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653. h tm l )
Most of the tutorials I found were not related to permutation testing so the subsequent steps may be incorrect. Please let me know where I go wrong... 1) I first ran QDec to generate a folder for the analysis which would create the subsequent fsgd and y files needed my mri_glmfit-sim. I am running both left and right hemisphere cortical thickness analysis with 10mm smoothing. The followingis for just the left hemisphere. Note I am doing a 3 group comparison, but for this 2 group ttest I manually centered the data based on the 3 group mean for age and education.
2) I ran the following command: /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm 10000 2 perm.abs.2 --sim-sign abs --bg 16 Prior to running the command above, from the y.fsdg file Ideleted the fwhm estimate of 13mm since this was not correctly estimated (ACF with long tails). I assumed that by removing this estimate, it would force the permutation test to calculate based on the data but when looking at the log output I see the following which says fwhm 0:
cmdline mri_glmfit.bin --C./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept- t hi ckness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Fema l e- Intercept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-In t er cept-thickness.mtx --C
./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group
In tercept-thickness.mtx --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 --y
/home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y. m gh --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage lh white --sim-done ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013
3)I started this a few days ago on a 16 core machine and it is still running in the terminal. I have 150 subjects in the analysis and specified 10000 iterations. In the terminal I assumed when I reach Poll 10000 it would be complete butcurrently I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016
Questions: a) I am curious, is this going to run 10,000 simulations X 150 patients or does the Poll number not have anything to do withthe number of iterations it is on?
b) Did I run this procedure correctly? Was I incorrect indeleting the fwhm estimate from y.fsgd file generated by Qdec even though we know the estimate is incorrect since smoothness assumed a gaussian distribution as opposed to gaussian with heavy tails
c) I noticed in the logfile the following warning: INFO: gd2mtx_method is dods Computing normalized matrix Normalized matrix condition is 5.65727 Matrix condition is 935.597 Found 136777 voxels in mask Reshaping mriglm->mask... search space = 89675.729228 ERROR: design matrix is not orthogonal, cannot be used with permutation. If this something you really want to do, run with --perm-force Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 I am not sure if this means my simulation is incorrect? Thanks, Ajay
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop:https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing:ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
freesurfer@nmr.mgh.harvard.edu
-
Ajay Kurani -
Anderson M. Winkler -
Douglas Greve -
Douglas N Greve
-
Harms, Michael -
maaike rive -
Matt Glasser