Hi FreeSurfer Developers,
I'm running freesurfer from the HCP pipeline developed by the Washington University in St. Louis. It ran just fine on our university's cluster, but when I migrated the pipeline to Exacloud, Intel's HPC cluster, the recon-all command began randomly failing. Occasionally it will work on the first try, but more often than not the same command has to be re-run upwards of 30 times before it will succeed. I've provided the recon-all command and the relevant section of the log file including the error here (attached is the full recon-all.log file):
recon-all -i ./T1w/T1w_acpc_dc_restore_1mm.nii.gz -subjid ${SUBJECT_ID} -sd . /T1w -motioncor -tal airach -nuintensitycor -normalization
... [perronea@exanode-3-9.local${PWD}/T1w/washu_INV0WC5U4JA/mri/] [2016-11-17 13:08:38] running: /home/exacloud/lustre1/users/mirandad/usr/local/freesurfer53/mni/bin/make_template -quiet -shrink 3 ./tmp.mri_nu_correct.mni.223851/nu1.mnc ./tmp.mri_nu_correct.mni.223851/1//template.mnc
Transforming slices:......................................................................................Done Transforming slices:.........................................................................................................................................................mincresample: posixio.c:210: px_pgin: Assertion `*posp == ((off_t)(-1)) || *posp == lseek(nciop->fd, 0, 1)' failed. nu_evaluate: crashed while running mincresample (termination status=134) nu_correct: crashed while running nu_evaluate (termination status=65280) ERROR: nu_correct Linux exanode-3-9.local 2.6.32-504.30.3.el6.x86_64 #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
recon-all -s washu_INV0WC5U4JA exited with ERRORS at Thu Nov 17 13:08:40 PST 2016
It doesn't appear than any similar errors have been reported on freesurfer. Our current work-around is to simply re-run the job on different nodes until it works, but this is not a sustainable long-term solution. Any guidance on troubleshooting this error would be greatly appreciated.
FreeSurfer version: freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0 Platform: CentOS release 6.5 (FINAL) uname -a: Linux exalab3.ohsu.edu 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Thanks, Anders
Anders Perrone, Research Assistant I Fair Neuroimaging Lab
perronea@ohsu.edumailto:perronea@ohsu.edu 503-418-1897
Oregon Health & Science University Mail code:L470 3181 SW Sam Jackson Park Road Portland, Oregon 97239-3098
[cid:image001.png@01D240D7.C6F2C170]
"If there is no solution to the problem then don't waste time worrying about it. If there is a solution to the problem then don't waste time worrying about it." - Dalai Lama XIV
