Hi FreeSurfer Developers,
I'm running freesurfer from the HCP pipeline developed by the Washington University in St. Louis. It ran just fine on our university's cluster, but when I migrated the pipeline to Exacloud, Intel's HPC cluster, the recon-all command began randomly failing. Occasionally it will work on the first try, but more often than not the same command has to be re-run upwards of 30 times before it will succeed. I've provided the recon-all command and the relevant section of the log file including the error here (attached is the full recon-all.log file):
recon-all -i ./T1w/T1w_acpc_dc_restore_1mm.nii.gz -subjid ${SUBJECT_ID} -sd . /T1w -motioncor -tal airach -nuintensitycor -normalization
... [perronea@exanode-3-9.local${PWD}/T1w/washu_INV0WC5U4JA/mri/] [2016-11-17 13:08:38] running: /home/exacloud/lustre1/users/mirandad/usr/local/freesurfer53/mni/bin/make_template -quiet -shrink 3 ./tmp.mri_nu_correct.mni.223851/nu1.mnc ./tmp.mri_nu_correct.mni.223851/1//template.mnc
Transforming slices:......................................................................................Done Transforming slices:.........................................................................................................................................................mincresample: posixio.c:210: px_pgin: Assertion `*posp == ((off_t)(-1)) || *posp == lseek(nciop->fd, 0, 1)' failed. nu_evaluate: crashed while running mincresample (termination status=134) nu_correct: crashed while running nu_evaluate (termination status=65280) ERROR: nu_correct Linux exanode-3-9.local 2.6.32-504.30.3.el6.x86_64 #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
recon-all -s washu_INV0WC5U4JA exited with ERRORS at Thu Nov 17 13:08:40 PST 2016
It doesn't appear than any similar errors have been reported on freesurfer. Our current work-around is to simply re-run the job on different nodes until it works, but this is not a sustainable long-term solution. Any guidance on troubleshooting this error would be greatly appreciated.
FreeSurfer version: freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0 Platform: CentOS release 6.5 (FINAL) uname -a: Linux exalab3.ohsu.edu 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Thanks, Anders
Anders Perrone, Research Assistant I Fair Neuroimaging Lab
perronea@ohsu.edumailto:perronea@ohsu.edu 503-418-1897
Oregon Health & Science University Mail code:L470 3181 SW Sam Jackson Park Road Portland, Oregon 97239-3098
[cid:image001.png@01D240D7.C6F2C170]
"If there is no solution to the problem then don't waste time worrying about it. If there is a solution to the problem then don't waste time worrying about it." - Dalai Lama XIV
That is a very odd error and I havent seen anything like it that I can recall.
My initial thoughts are that the nu_evaluate and nu_correct commands are part of the mni tools shipped with freesurfer. The mni tools do a lot of reading, writing and deleting of temporary files. (You can see how it mentions 'tmp.mri_nu_correct.mni.223851' in the log file you sent).
It's possible that your cluster has limits on the I/O (our does and the mni tools were the culprit on our case). Maybe its deleting some of these temp files or perhaps the temporary space is just getting filled. This could cause the nu_evaluate command to fail in an ungraceful way.
On 11/17/2016 05:14 PM, Anders Perrone wrote:
Hi FreeSurfer Developers,
I’m running freesurfer from the HCP pipeline developed by the Washington University in St. Louis. It ran just fine on our university’s cluster, but when I migrated the pipeline to Exacloud, Intel’s HPC cluster, the recon-all command began randomly failing. Occasionally it will work on the first try, but more often than not the same command has to be re-run upwards of 30 times before it will succeed. I’ve provided the recon-all command and the relevant section of the log file including the error here (attached is the full recon-all.log file):
recon-all -i ./T1w/T1w_acpc_dc_restore_1mm.nii.gz -subjid ${SUBJECT_ID} -sd . /T1w -motioncor -tal airach -nuintensitycor –normalization
…
[perronea@exanode-3-9.local${PWD}/T1w/washu_INV0WC5U4JA/mri/] [2016-11-17 13:08:38] running:
/home/exacloud/lustre1/users/mirandad/usr/local/freesurfer53/mni/bin/make_template -quiet -shrink 3 ./tmp.mri_nu_correct.mni.223851/nu1.mnc ./tmp.mri_nu_correct.mni.223851/1//template.mnc
Transforming slices:......................................................................................Done
Transforming slices:.........................................................................................................................................................mincresample: posixio.c:210: px_pgin: Assertion `*posp == ((off_t)(-1)) || *posp == lseek(nciop->fd, 0, 1)' failed.
nu_evaluate: crashed while running mincresample (termination status=134)
nu_correct: crashed while running nu_evaluate (termination status=65280)
ERROR: nu_correct
Linux exanode-3-9.local 2.6.32-504.30.3.el6.x86_64 #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
recon-all -s washu_INV0WC5U4JA exited with ERRORS at Thu Nov 17 13:08:40 PST 2016
It doesn’t appear than any similar errors have been reported on freesurfer. Our current work-around is to simply re-run the job on different nodes until it works, but this is not a sustainable long-term solution. Any guidance on troubleshooting this error would be greatly appreciated.
FreeSurfer version: freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0
Platform: CentOS release 6.5 (FINAL)
uname –a: Linux exalab3.ohsu.edu 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Thanks,
Anders
Anders Perrone,
Research Assistant I
Fair Neuroimaging Lab
perronea@ohsu.edu mailto:perronea@ohsu.edu
503-418-1897
Oregon Health & Science University
Mail code:L470
3181 SW Sam Jackson Park Road
Portland, Oregon 97239-3098
“If there is no solution to the problem then don’t waste time worrying about it. If there is a solution to the problem then don’t waste time worrying about it.” – Dalai Lama XIV
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Pardon me for poking in here. I hope this is helpful. It looks like the failure is on an lseek command which means the file isn't there or was truncated, as Zeke suggests. When you deploy your jobs to the cloud cluster, perhaps there's a directive which can be tweaked to give you a bigger playpen.
Best - Don
-----Original Message----- From: freesurfer-bounces@nmr.mgh.harvard.edu [mailto:freesurfer- bounces@nmr.mgh.harvard.edu] On Behalf Of Z K Sent: Friday, November 18, 2016 10:36 AM To: freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Recon-all: stochastic mri_nu_correct error when run on HPC cluster
That is a very odd error and I havent seen anything like it that I can recall.
My initial thoughts are that the nu_evaluate and nu_correct commands are part of the mni tools shipped with freesurfer. The mni tools do a lot of reading, writing and deleting of temporary files. (You can see how it mentions 'tmp.mri_nu_correct.mni.223851' in the log file you sent).
It's possible that your cluster has limits on the I/O (our does and the mni tools were the culprit on our case). Maybe its deleting some of these temp files or perhaps the temporary space is just getting filled. This could cause the nu_evaluate command to fail in an ungraceful way.
On 11/17/2016 05:14 PM, Anders Perrone wrote:
Hi FreeSurfer Developers,
I'm running freesurfer from the HCP pipeline developed by the Washington University in St. Louis. It ran just fine on our university's cluster, but when I migrated the pipeline to Exacloud, Intel's HPC cluster, the recon-all command began randomly failing. Occasionally it will work on the first try, but more often than not the same command has to be re-run upwards of 30 times before it will succeed. I've provided the recon-all command and the relevant section of the log file including the error here (attached is the full recon-all.log file):
recon-all -i ./T1w/T1w_acpc_dc_restore_1mm.nii.gz -subjid ${SUBJECT_ID} -sd . /T1w -motioncor -tal airach -nuintensitycor -normalization
...
[perronea@exanode-3-9.local${PWD}/T1w/washu_INV0WC5U4JA/mri/] [2016-11-17 13:08:38] running:
/home/exacloud/lustre1/users/mirandad/usr/local/freesurfer53/mni/bin/m ake_template -quiet -shrink 3 ./tmp.mri_nu_correct.mni.223851/nu1.mnc ./tmp.mri_nu_correct.mni.223851/1//template.mnc
Transforming slices:............................................................... .......................Done
Transforming
slices:....................................................................................................................... ..................................mincresample:
posixio.c:210: px_pgin: Assertion `*posp == ((off_t)(-1)) || *posp == lseek(nciop->fd, 0, 1)' failed.
nu_evaluate: crashed while running mincresample (termination status=134)
nu_correct: crashed while running nu_evaluate (termination status=65280)
ERROR: nu_correct
Linux exanode-3-9.local 2.6.32-504.30.3.el6.x86_64 #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
recon-all -s washu_INV0WC5U4JA exited with ERRORS at Thu Nov 17 13:08:40 PST 2016
It doesn't appear than any similar errors have been reported on freesurfer. Our current work-around is to simply re-run the job on different nodes until it works, but this is not a sustainable long-term solution. Any guidance on troubleshooting this error would be greatly appreciated.
FreeSurfer version: freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0
Platform: CentOS release 6.5 (FINAL)
uname -a: Linux exalab3.ohsu.edu 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Thanks,
Anders
Anders Perrone,
Research Assistant I
Fair Neuroimaging Lab
perronea@ohsu.edu mailto:perronea@ohsu.edu
503-418-1897
Oregon Health & Science University
Mail code:L470
3181 SW Sam Jackson Park Road
Portland, Oregon 97239-3098
"If there is no solution to the problem then don't waste time worrying about it. If there is a solution to the problem then don't waste time worrying about it." - Dalai Lama XIV
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
freesurfer@nmr.mgh.harvard.edu
-
Anders Perrone -
Krieger, Donald N. -
Z K