External Email - Use Caution
Hi Freesurfer Support,
The past few weeks, I've emailed about my recon-all processing stream never completely finishing. We are still encountering the same problem. For context, our recon-all job submission script runs recon-all for 1 image. Given a folder of 100 MRI images, we have another script that calls upon the recon-all script for each image within the folder via a for loop. The result is 100 jobs running in parallel on the cluster.
I have attached a screenshot of the parameters we set whenever we submit each recon-all job for each image by SLURM to UCSD's cluster. The maximum time we are allowed for each job is 48 hours. Our jobs are shared-node jobs, which means we run more than 1 job on a single node. This time, we increased RAM from 8GB to 16GB in hopes the entire recon-all processing stream can fully run through for each image, but it still stops at "mri_pretess done" and does not go on to the -fill step. Each job submitted to the cluster took around 4 hours and 20 minutes each to run in our most recent attempt.
If increasing the RAM did not change anything for us, how else can we get more verbose error messages? What other reasons could be why our recon-all aborts before finishing? I have attached a recon-all.log from one of our subjects for reference and a screenshot of the parameters we set for submitting jobs in our cluster for reference.
Freesurfer version: 7.2.0, but already available as module on UCSD's cluster system
Platform: Rocky Linux release 8.8 (Green Obsidian)
uname -a: Linux login01 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Wed Jun 28 15:04:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Recon-all log: see attached
Thank you!
Sincerely,
Yilei
not sure what is happening. You can run recon-all with -debug and capture the output (both stdout and stderr) into a file. This will cause it to print out huge amounts of text specifying what it is doing on each line. You should also see if there is some stderr output that is not being caught. You can try running recon-all -fill on the subject that you created to see if the fill will run at all (this may be easier than waiting 4 hours).
On 4/8/2024 3:18 PM, Dong, Yilei wrote:
External Email - Use Caution
Hi Freesurfer Support,
The past few weeks, I've emailed about my recon-all processing stream never completely finishing. We are still encountering the same problem. For context, our recon-all job submission script runs recon-all for 1 image. Given a folder of 100 MRI images, we have another script that calls upon the recon-all script for each image within the folder via a for loop. The result is 100 jobs running in parallel on the cluster.
I have attached a screenshot of the parameters we set whenever we submit each recon-all job for each image by SLURM to UCSD's cluster. The maximum time we are allowed for each job is 48 hours. Our jobs are shared-node jobs, which means we run more than 1 job on a single node. This time, we increased RAM from 8GB to 16GB in hopes the entire recon-all processing stream can fully run through for each image, but it still stops at "mri_pretess done" and does not go on to the -fill step. Each job submitted to the cluster took around 4 hours and 20 minutes each to run in our most recent attempt.
If increasing the RAM did not change anything for us, how else can we get more verbose error messages? What other reasons could be why our recon-all aborts before finishing? I have attached a recon-all.log from one of our subjects for reference and a screenshot of the parameters we set for submitting jobs in our cluster for reference.
Freesurfer version: 7.2.0, but already available as module on UCSD's cluster system
Platform: Rocky Linux release 8.8 (Green Obsidian)
uname -a: Linux login01 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Wed Jun 28 15:04:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Recon-all log: see attached
Thank you!
Sincerely,
Yilei
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
External Email - Use Caution
Hi Freesurfer support,
I have several questions regarding recon-all -fill and -debug.
If we run recon-all -fill, what is the command set-up for that? From the ReconAllDevTable here, https://secure-web.cisco.com/168deFdRWVJ6rqWKdMIwjH_-sTPV9vncDkCBQheBnFXRjWP..., do we structure the recon-all command to be: recon-all -autorecon2 -fill -subjid <subjid>? I attempted to run recon-all -fill with -debug with the structure as follows: recon-all -s $subject -i $1 -fill -debug and I was wondering if this was the wrong format. I have also attached our recon-all-slurm-apr19.sb cluster submission file for reference.
Does recon-all -fill only run the fill step, while ignoring the previous steps before that or does recon-all run through all the initial steps including the -fill step ideally?
Regarding recon-all -debug, does the stdout file have the huge amounts of text specifying what each line is doing? I have included .out and .err files from the same recon-all -fill with -debug command for reference.
Sincerely, Yilei ________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N. Greve dgreve@mgh.harvard.edu Sent: Thursday, April 18, 2024 2:57 PM To: freesurfer@nmr.mgh.harvard.edu freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] recon-all process still not finishing properly
not sure what is happening. You can run recon-all with -debug and capture the output (both stdout and stderr) into a file. This will cause it to print out huge amounts of text specifying what it is doing on each line. You should also see if there is some stderr output that is not being caught. You can try running recon-all -fill on the subject that you created to see if the fill will run at all (this may be easier than waiting 4 hours).
On 4/8/2024 3:18 PM, Dong, Yilei wrote:
External Email - Use Caution
Hi Freesurfer Support,
The past few weeks, I've emailed about my recon-all processing stream never completely finishing. We are still encountering the same problem. For context, our recon-all job submission script runs recon-all for 1 image. Given a folder of 100 MRI images, we have another script that calls upon the recon-all script for each image within the folder via a for loop. The result is 100 jobs running in parallel on the cluster.
I have attached a screenshot of the parameters we set whenever we submit each recon-all job for each image by SLURM to UCSD's cluster. The maximum time we are allowed for each job is 48 hours. Our jobs are shared-node jobs, which means we run more than 1 job on a single node. This time, we increased RAM from 8GB to 16GB in hopes the entire recon-all processing stream can fully run through for each image, but it still stops at "mri_pretess done" and does not go on to the -fill step. Each job submitted to the cluster took around 4 hours and 20 minutes each to run in our most recent attempt.
If increasing the RAM did not change anything for us, how else can we get more verbose error messages? What other reasons could be why our recon-all aborts before finishing? I have attached a recon-all.log from one of our subjects for reference and a screenshot of the parameters we set for submitting jobs in our cluster for reference.
Freesurfer version: 7.2.0, but already available as module on UCSD's cluster system
Platform: Rocky Linux release 8.8 (Green Obsidian)
uname -a: Linux login01 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Wed Jun 28 15:04:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Recon-all log: see attached
Thank you!
Sincerely,
Yilei
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu https://secure-web.cisco.com/10vZ3lyKifsSsLtfZzgKvrT91yFm1-8drLKU54PSfnAnAiE...https://secure-web.cisco.com/1L9JtWIahKTHfxUVwdkxJIsmbw26B2aV-zoJtcXFvPFglatVfthNANOU6m6dINcqgOMcqNe6-NyaJ1AACvR352K7WfmeGGQLRHqelIETwyGo-Ax_nLddafs6AFMte7wozQAQA7-h9QiZAa9Y1ygaAONlIsaEVwAHq2B0Jk8QESmgqPwz4w_LSdAjkVLTuvd-sFzVsZcGTaW5j7FdAF9-lianwwSs1wX2pKmBJJeiImfx1uy946naVFPeEn9gh67Cbvtw25-pCSt2lspl0bliEpRlyxHBKqoO7SztH_LqXZyU506emutMUiW7URO2tyocR/https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer__;!!LLK065n_VXAQ!iYiPFfQGnfb0AUsXFU0O8g63EYT_Ws59pJFwJLy77PZ9mY6xftY2ysOiM9wrt5KFxVIc-uTTtBbho1NHzIeUC2Rq5w$
External Email - Use Caution
Hi Freesurfer Developers,
Just to reply to this thread again, if I want to see if recon-all will just run the -fill step, do I submit it on the files that have already been run? In addition, does this error count as Freesurfer running out of RAM or it's more of a cluster issue I have to talk with the staff that manage it?
Sincerely, Yilei ________________________________ From: Dong, Yilei yid025@health.ucsd.edu Sent: Tuesday, April 23, 2024 3:34 PM To: freesurfer@nmr.mgh.harvard.edu freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] recon-all process still not finishing properly
Hi Freesurfer support,
I have several questions regarding recon-all -fill and -debug.
If we run recon-all -fill, what is the command set-up for that? From the ReconAllDevTable here, https://secure-web.cisco.com/1lmfyKn6Pvakq-RQxgJbE5k4RDGG6d0Vpvb5lGcjzph3h7U..., do we structure the recon-all command to be: recon-all -autorecon2 -fill -subjid <subjid>? I attempted to run recon-all -fill with -debug with the structure as follows: recon-all -s $subject -i $1 -fill -debug and I was wondering if this was the wrong format. I have also attached our recon-all-slurm-apr19.sb cluster submission file for reference.
Does recon-all -fill only run the fill step, while ignoring the previous steps before that or does recon-all run through all the initial steps including the -fill step ideally?
Regarding recon-all -debug, does the stdout file have the huge amounts of text specifying what each line is doing? I have included .out and .err files from the same recon-all -fill with -debug command for reference.
Sincerely, Yilei ________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N. Greve dgreve@mgh.harvard.edu Sent: Thursday, April 18, 2024 2:57 PM To: freesurfer@nmr.mgh.harvard.edu freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] recon-all process still not finishing properly
not sure what is happening. You can run recon-all with -debug and capture the output (both stdout and stderr) into a file. This will cause it to print out huge amounts of text specifying what it is doing on each line. You should also see if there is some stderr output that is not being caught. You can try running recon-all -fill on the subject that you created to see if the fill will run at all (this may be easier than waiting 4 hours).
On 4/8/2024 3:18 PM, Dong, Yilei wrote:
External Email - Use Caution
Hi Freesurfer Support,
The past few weeks, I've emailed about my recon-all processing stream never completely finishing. We are still encountering the same problem. For context, our recon-all job submission script runs recon-all for 1 image. Given a folder of 100 MRI images, we have another script that calls upon the recon-all script for each image within the folder via a for loop. The result is 100 jobs running in parallel on the cluster.
I have attached a screenshot of the parameters we set whenever we submit each recon-all job for each image by SLURM to UCSD's cluster. The maximum time we are allowed for each job is 48 hours. Our jobs are shared-node jobs, which means we run more than 1 job on a single node. This time, we increased RAM from 8GB to 16GB in hopes the entire recon-all processing stream can fully run through for each image, but it still stops at "mri_pretess done" and does not go on to the -fill step. Each job submitted to the cluster took around 4 hours and 20 minutes each to run in our most recent attempt.
If increasing the RAM did not change anything for us, how else can we get more verbose error messages? What other reasons could be why our recon-all aborts before finishing? I have attached a recon-all.log from one of our subjects for reference and a screenshot of the parameters we set for submitting jobs in our cluster for reference.
Freesurfer version: 7.2.0, but already available as module on UCSD's cluster system
Platform: Rocky Linux release 8.8 (Green Obsidian)
uname -a: Linux login01 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Wed Jun 28 15:04:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Recon-all log: see attached
Thank you!
Sincerely,
Yilei
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu https://secure-web.cisco.com/1Pv7GW0cJEBRkDoNxnJuTO5eel1Em4ED-AsDRikPcYZfFw6...https://secure-web.cisco.com/1P7JPsiPrzppfxrMrpMYHuH0WScJN8TFnGcGdTyRnynXLx8536ONkcPa15X7rmvqUcGKIY3kX7aouazjH85-B_dcnd6Zpw9fwJNAm2Jhe52ADvAbpXEsiQwcnuiFUAE_KbNY9GBE2QRBWCMzKSi1hXwaMPLUxQg1Yze7hRcYdnpQfRkhEXMBkXFtLD1p_c-71OOGr5AqcQDHVUA21fZwTm0dTezQRbtGYnoPFTQRlC-Cj2TO89ncFCg1gzInzx58B19jponiJd_RTMTSx555W4rjaYJSMbf9mktwVUD0OZ68JKKNfV95AwVwXw49AwJ1a/https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer__;!!LLK065n_VXAQ!iYiPFfQGnfb0AUsXFU0O8g63EYT_Ws59pJFwJLy77PZ9mY6xftY2ysOiM9wrt5KFxVIc-uTTtBbho1NHzIeUC2Rq5w$
It should just run the fill step. However, I've been slowly reprogramming recon-all to just see what needs to be run and rerun it, so you might get more than just the fill.
On 5/3/2024 4:38 PM, Dong, Yilei wrote:
External Email - Use Caution
Hi Freesurfer Developers,
Just to reply to this thread again, if I want to see if recon-all will just run the -fill step, do I submit it on the files that have already been run? In addition, does this error count as Freesurfer running out of RAM or it's more of a cluster issue I have to talk with the staff that manage it?
Sincerely, Yilei
*From:* Dong, Yilei yid025@health.ucsd.edu *Sent:* Tuesday, April 23, 2024 3:34 PM *To:* freesurfer@nmr.mgh.harvard.edu freesurfer@nmr.mgh.harvard.edu *Subject:* Re: [Freesurfer] recon-all process still not finishing properly Hi Freesurfer support,
I have several questions regarding recon-all -fill and -debug.
If we run recon-all -fill, what is the command set-up for that? From the ReconAllDevTable here, *MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://surfer.nmr.mgh.harvard.edu/fswiki/ReconAllDevTable https://secure-web.cisco.com/1lmfyKn6Pvakq-RQxgJbE5k4RDGG6d0Vpvb5lGcjzph3h7UE3rIOK94h4_vGK8oa4q_FYtrbTmYfXg9SF-sFCkj4DY_oUkqnlqG_VRr6zv-uPviunfO901Vr4Llra2RZoL7k2SaVz5b8sHZwz9tr28nK4K01ItC1QwtyMTw-gA6LKU-lK3jTTjAHXqq0NK-7yqLmD6TCtsqdbNsdoxGLEfnWyV4rlB4ypTtbp8RQCkTKa8EWWLysYONW6-m24iLcIBbOt0l9i4UxjEV-DHag-8gIkxA9SeCthGdnNmHPEyy8ngikQbSR5ckvogYu-nsAi/https%3A%2F%2Fsurfer.nmr.mgh.harvard.edu%2Ffswiki%2FReconAllDevTable, do we structure the recon-all command to be: recon-all -autorecon2 -fill -subjid <subjid>? I attempted to run recon-all -fill with -debug with the structure as follows: recon-all -s $subject -i $1 -fill -debug and I was wondering if this was the wrong format. I have also attached our recon-all-slurm-apr19.sb cluster submission file for reference.
Does recon-all -fill only run the fill step, while ignoring the previous steps before that or does recon-all run through all the initial steps including the -fill step ideally?
Regarding recon-all -debug, does the stdout file have the huge amounts of text specifying what each line is doing? I have included .out and .err files from the same recon-all -fill with -debug command for reference.
Sincerely, Yilei
*From:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N. Greve dgreve@mgh.harvard.edu *Sent:* Thursday, April 18, 2024 2:57 PM *To:* freesurfer@nmr.mgh.harvard.edu freesurfer@nmr.mgh.harvard.edu *Subject:* Re: [Freesurfer] recon-all process still not finishing properly not sure what is happening. You can run recon-all with -debug and capture the output (both stdout and stderr) into a file. This will cause it to print out huge amounts of text specifying what it is doing on each line. You should also see if there is some stderr output that is not being caught. You can try running recon-all -fill on the subject that you created to see if the fill will run at all (this may be easier than waiting 4 hours).
On 4/8/2024 3:18 PM, Dong, Yilei wrote:
External Email - Use Caution
Hi Freesurfer Support,
The past few weeks, I've emailed about my recon-all processing stream never completely finishing. We are still encountering the same problem. For context, our recon-all job submission script runs recon-all for 1 image. Given a folder of 100 MRI images, we have another script that calls upon the recon-all script for each image within the folder via a for loop. The result is 100 jobs running in parallel on the cluster.
I have attached a screenshot of the parameters we set whenever we submit each recon-all job for each image by SLURM to UCSD's cluster. The maximum time we are allowed for each job is 48 hours. Our jobs are shared-node jobs, which means we run more than 1 job on a single node. This time, we increased RAM from 8GB to 16GB in hopes the entire recon-all processing stream can fully run through for each image, but it still stops at "mri_pretess done" and does not go on to the -fill step. Each job submitted to the cluster took around 4 hours and 20 minutes each to run in our most recent attempt.
If increasing the RAM did not change anything for us, how else can we get more verbose error messages? What other reasons could be why our recon-all aborts before finishing? I have attached a recon-all.log from one of our subjects for reference and a screenshot of the parameters we set for submitting jobs in our cluster for reference.
Freesurfer version: 7.2.0, but already available as module on UCSD's cluster system
Platform: Rocky Linux release 8.8 (Green Obsidian)
uname -a: Linux login01 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Wed Jun 28 15:04:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Recon-all log: see attached
Thank you!
Sincerely,
Yilei
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu mailto:Freesurfer@nmr.mgh.harvard.edu *MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer https://secure-web.cisco.com/1BrM8h0098VhZcRBjpL9j6U4ifgVdwSGTXsfoZs5ViUIft9Bmopekic_rw3v5RQgl9792mTOgQeo_EH_jldym9jccJQ0wsoKByBOvMElfBVpURHCvDlJ1H_VFJLMlMSUFDJKQrNa2Pu3O4IkI5aDfudafXVtWbx1wqCAsSOT0JRnTHLrZXAJF0XLN4AJ1E40hUXRqcPevmsW9s-U-E4moLZgFhoc6u-dqiKW_29UJyMLM2cV8llCofwhX1Hk2qz5ucvpeK1KLAYwLiBDgovOxvGItbHirbBXTWfVK6VkHIyVU8usJZQduS0XQI73vmxd5/https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer__%3B%21%21LLK065n_VXAQ%21iYiPFfQGnfb0AUsXFU0O8g63EYT_Ws59pJFwJLy77PZ9mY6xftY2ysOiM9wrt5KFxVIc-uTTtBbho1NHzIeUC2Rq5w%24
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
freesurfer@nmr.mgh.harvard.edu
-
Dong, Yilei -
Douglas N. Greve