External Email - Use Caution
Hi, Freesufer experts
My name is wei shao, I have a few questions regarding the parallel computing of recon-all
1 According to documents, freesurfer is OpenMP code , I use -openmp 8 flag like: recon-all -alli -i input.file -s output.file -openmp 8 to tun the recon command on a server, but it doesn't work for me (the running time is very similar for the same subject). Cloud you tell me what is wrong?
2 Can Freesufer take advantage of several GPUs?
3 Is it possible to open few threads to run recon-all on different sections parallelly?
Best, Wei
On 1/20/2021 3:39 PM, Wei Shao wrote:
External Email - Use Caution
Hi, Freesufer experts
My name is wei shao, I have a few questions regarding the parallel computing of recon-all
1 According to documents, freesurfer is OpenMP code , I use -openmp 8 flag like: recon-all -alli -i input.file -s output.file -openmp 8 to tun the recon command on a server, but it doesn't work for me (the running time is very similar for the same subject). Cloud you tell me what is wrong?
Please specify the difference in times.
2 Can Freesufer take advantage of several GPUs?
No
3 Is it possible to open few threads to run recon-all on different sections parallelly?
Only if you run them by hand
Best, Wei
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
External Email - Use Caution
Hi, Douglas
Thanks for your kind reply.
1 I run the same subject on a server with openmp 8 flags cost 7 hours, without openmp 8 flag would be 7h10minutes.
3 Could you be more specific about run them by hand? is this mean I run different tasks like skull strip or parcellation separately within recon rather than recon-all?
Best, Wei
On Mon, Jan 25, 2021 at 10:17 PM Douglas N. Greve dgreve@mgh.harvard.edu wrote:
On 1/20/2021 3:39 PM, Wei Shao wrote:
External Email - Use CautionHi, Freesufer experts
My name is wei shao, I have a few questions regarding the parallel computing of recon-all
1 According to documents, freesurfer is OpenMP code , I use -openmp 8 flag like: recon-all -alli -i input.file -s output.file -openmp 8 to tun the recon command on a server, but it doesn't work for me (the running time is very similar for the same subject). Cloud you tell me what is wrong?
Please specify the difference in times.
2 Can Freesufer take advantage of several GPUs?
No
3 Is it possible to open few threads to run recon-all on different sections parallelly?
Only if you run them by hand
Best, Wei
Freesurfer mailing listFreesurfer@nmr.mgh.harvard.eduhttps://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://secure-web.cisco.com/1iCbpbp3TlQC67E9mq0PzTzJ5cANh6TRxDqcmmQi1W7zG1j...
On 1/27/2021 2:02 PM, Wei Shao wrote:
External Email - Use Caution
Hi, Douglas
Thanks for your kind reply.
1 I run the same subject on a server with openmp 8 flags cost 7 hours, without openmp 8 flag would be 7h10minutes.
Can you send the two recon-all.log files?
3 Could you be more specific about run them by hand? is this mean I run different tasks like skull strip or parcellation separately within recon rather than recon-all?
Each step in recon-all can be run separately using command line flags. It is not an easy thing to do and not well documented.
Best, Wei
On Mon, Jan 25, 2021 at 10:17 PM Douglas N. Greve <dgreve@mgh.harvard.edu mailto:dgreve@mgh.harvard.edu> wrote:
On 1/20/2021 3:39 PM, Wei Shao wrote:External Email - Use Caution Hi, Freesufer experts My name is wei shao, I have a few questions regarding the parallel computing of recon-all 1 According to documents, freesurfer is OpenMP code , I use -openmp 8 flag like: recon-all -alli -i input.file -s output.file -openmp 8 to tun the recon command on a server, but it doesn't work for me (the running time is very similar for the same subject). Cloud you tell me what is wrong?Please specify the difference in times.2 Can Freesufer take advantage of several GPUs?No3 Is it possible to open few threads to run recon-all on different sections parallelly?Only if you run them by handBest, Wei _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> *MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer <https://secure-web.cisco.com/1iCbpbp3TlQC67E9mq0PzTzJ5cANh6TRxDqcmmQi1W7zG1jM5MCUQqqsXe3QwM7oVOFj_LjCFo27-8CJfoyO91GKrSBzIxnU_YshInte2aLVCn47Vj32-jMMVrD8rT-_76qE6PP6khCI7B6FR2Yp5MsZI8wpfQDHkafQjCjJcDl8xcasuz_GtojXgPZ9v09wpG8dwOWxahtMB8ldo_6U1rwqRG3OFUZCwWs26DrqqA8xjv-XgB_FKSSrhHHo8xSgDo6iwlh8PgiT7FQY9ZGz_TQ/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer>_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> *MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer <https://secure-web.cisco.com/1iCbpbp3TlQC67E9mq0PzTzJ5cANh6TRxDqcmmQi1W7zG1jM5MCUQqqsXe3QwM7oVOFj_LjCFo27-8CJfoyO91GKrSBzIxnU_YshInte2aLVCn47Vj32-jMMVrD8rT-_76qE6PP6khCI7B6FR2Yp5MsZI8wpfQDHkafQjCjJcDl8xcasuz_GtojXgPZ9v09wpG8dwOWxahtMB8ldo_6U1rwqRG3OFUZCwWs26DrqqA8xjv-XgB_FKSSrhHHo8xSgDo6iwlh8PgiT7FQY9ZGz_TQ/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer>
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
External Email - Use Caution
1 According to documents, freesurfer is OpenMP code , I use -openmp 8 flag like: recon-all -alli -i input.file -s output.file -openmp 8 to tun the recon command on a server, but it doesn't work for me (the running time is very similar for the same subject). Cloud you tell me what is wrong?
It's my understanding that you also need to specify `-parallel` in addition to optional flag for the openmp thread count. At least since v6, no?
https://secure-web.cisco.com/1iQveNDdwNTosrnlkwXmvvjJwqsGtnW6NxvHb6cchwZMPFP...
cheers Michael Krause
No, use of -threads or -openmp will always trigger the openmp code. The -parallel may introduce other parallel. I did not write this part of the code so I'm not sure, and I think I might have broken the -parallel implementation when I did the last recon-all rewrite
On 1/28/2021 11:23 AM, Michael Krause wrote:
External Email - Use Caution1 According to documents, freesurfer is OpenMP code , I use -openmp 8 flag like: recon-all -alli -i input.file -s output.file -openmp 8 to tun the recon command on a server, but it doesn't work for me (the running time is very similar for the same subject). Cloud you tell me what is wrong?
It's my understanding that you also need to specify `-parallel` in addition to optional flag for the openmp thread count. At least since v6, no?
https://secure-web.cisco.com/1iQveNDdwNTosrnlkwXmvvjJwqsGtnW6NxvHb6cchwZMPFP...
cheers Michael Krause
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
External Email - Use Caution
On 1/28/21 6:00 PM, Douglas N. Greve wrote:
No, use of -threads or -openmp will always trigger the openmp code. The -parallel may introduce other parallel. I did not write this part of the code so I'm not sure, and I think I might have broken the -parallel implementation when I did the last recon-all rewrite
Ah, that's good to know. I will remove the redundancy in our scripts then.
What do you mean by "may have broken"? Broken as in we have to worry about results? We are currently running a couple of thousand subjects with 7.1.1 and they appear to be running fine so far *with* -parallel and -openmp 4 with an average core utilization of around 2.4 out of the possible 4.
cheers Michael Krause
No, all the results are fine. By "broken" I mean that it might crash in places that it would not have crashed in v6 or that it just would not have been running in parallel when you think it was.
On 1/28/2021 3:48 PM, Michael Krause wrote:
External Email - Use CautionOn 1/28/21 6:00 PM, Douglas N. Greve wrote:
No, use of -threads or -openmp will always trigger the openmp code. The -parallel may introduce other parallel. I did not write this part of the code so I'm not sure, and I think I might have broken the -parallel implementation when I did the last recon-all rewrite
Ah, that's good to know. I will remove the redundancy in our scripts then.
What do you mean by "may have broken"? Broken as in we have to worry about results? We are currently running a couple of thousand subjects with 7.1.1 and they appear to be running fine so far *with* -parallel and -openmp 4 with an average core utilization of around 2.4 out of the possible 4.
cheers Michael Krause
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
External Email - Use Caution
Hi, Freesufer experts
Sorry for the late reply, I have tried again with openmp 8 flag and added the cpus-per-task=16 for a subject. I have cut the recon-all processing time from 06:17:14 to 03:32:21! So excited.
By the way, Could you help me to understand this command?
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
From what I understand, if I apply this code for recon-all, this command assigns the total threads of recon-all task = number of the cpu used in the recon-all task? I know a cpu can handle multiples threads, is this mean that only one thread to one cpu for the recon-all task?
Best, Wei
On Fri, Jan 29, 2021 at 10:53 AM Douglas N. Greve dgreve@mgh.harvard.edu wrote:
No, all the results are fine. By "broken" I mean that it might crash in places that it would not have crashed in v6 or that it just would not have been running in parallel when you think it was.
On 1/28/2021 3:48 PM, Michael Krause wrote:
External Email - Use CautionOn 1/28/21 6:00 PM, Douglas N. Greve wrote:
No, use of -threads or -openmp will always trigger the openmp code. The -parallel may introduce other parallel. I did not write this part of the code so I'm not sure, and I think I might have broken the -parallel implementation when I did the last recon-all rewrite
Ah, that's good to know. I will remove the redundancy in our scripts
then.
What do you mean by "may have broken"? Broken as in we have to worry
about
results? We are currently running a couple of thousand subjects with
7.1.1
and they appear to be running fine so far *with* -parallel and -openmp 4 with an average core utilization of around 2.4 out of the possible 4.
cheers Michael Krause
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://secure-web.cisco.com/1x1FDZItu_iknOi5S4MQ7eWCXCrNKoHyIpbIzQsfN4GnSDC...
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://secure-web.cisco.com/1x1FDZItu_iknOi5S4MQ7eWCXCrNKoHyIpbIzQsfN4GnSDC...
If you are using slurm, then you have to allocate the number of CPUs when you run sbatch, otherwise, you will get only one CPU regardless of how many are tell recon-all to use.
On 1/29/2021 1:16 PM, Wei Shao wrote:
External Email - Use Caution
Hi, Freesufer experts
Sorry for the late reply, I have tried again with openmp 8 flag and added the cpus-per-task=16 for a subject. I have cut the recon-all processing time from 06:17:14 to 03:32:21! So excited.
By the way, Could you help me to understand this command? export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK From what I understand, if I apply this code for recon-all, this command assigns the total threads of recon-all task = number of the cpu used in the recon-all task? I know a cpu can handle multiples threads, is this mean that only one thread to one cpu for the recon-all task?
Best, Wei
On Fri, Jan 29, 2021 at 10:53 AM Douglas N. Greve <dgreve@mgh.harvard.edu mailto:dgreve@mgh.harvard.edu> wrote:
No, all the results are fine. By "broken" I mean that it might crash in places that it would not have crashed in v6 or that it just would not have been running in parallel when you think it was. On 1/28/2021 3:48 PM, Michael Krause wrote: > External Email - Use Caution > > On 1/28/21 6:00 PM, Douglas N. Greve wrote: >> No, use of -threads or -openmp will always trigger the openmp code. The >> -parallel may introduce other parallel. I did not write this part of the >> code so I'm not sure, and I think I might have broken the -parallel >> implementation when I did the last recon-all rewrite > Ah, that's good to know. I will remove the redundancy in our scripts then. > > What do you mean by "may have broken"? Broken as in we have to worry about > results? We are currently running a couple of thousand subjects with 7.1.1 > and they appear to be running fine so far *with* -parallel and -openmp 4 > with an average core utilization of around 2.4 out of the possible 4. > > cheers > Michael Krause > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> > *MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer <https://secure-web.cisco.com/1x1FDZItu_iknOi5S4MQ7eWCXCrNKoHyIpbIzQsfN4GnSDCjOyXy3kp8OfLt-7YyV7WgFwH2QpOtwJt1SZ6vVVUWqbIfTAHGEyxJSpkXF-mNroJRELpRhpqgQDCDweSaiQkKLfBBGKf33Q9KFyzLzJc_1DLl3wdcGXR0CDJ882WuvH1YyrZdsHjWX_-69ehG3CMWgIX3hM04i_TR4dgU_Pp2zEZIbjVphSRB3QORhMrmnlC97s7_FjaoyxdZVturIgMFwdGpKnNp2L93i64F8aA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer> _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> *MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer <https://secure-web.cisco.com/1x1FDZItu_iknOi5S4MQ7eWCXCrNKoHyIpbIzQsfN4GnSDCjOyXy3kp8OfLt-7YyV7WgFwH2QpOtwJt1SZ6vVVUWqbIfTAHGEyxJSpkXF-mNroJRELpRhpqgQDCDweSaiQkKLfBBGKf33Q9KFyzLzJc_1DLl3wdcGXR0CDJ882WuvH1YyrZdsHjWX_-69ehG3CMWgIX3hM04i_TR4dgU_Pp2zEZIbjVphSRB3QORhMrmnlC97s7_FjaoyxdZVturIgMFwdGpKnNp2L93i64F8aA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer>
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
freesurfer@nmr.mgh.harvard.edu
-
Douglas N. Greve -
Michael Krause -
Wei Shao