Hi Experts,
If I have 100s of subjects to recon and have access to an HPC with both GPU and CPU options what is the recommended flags to maximize speed performance? Is … -parallel -openmp 4 … still the best approach? And if so, is 2GB memory/core best (keeping it 8GB memory/subject) or is there some better arrangement?
Best, Mitch
External Email - Use Caution
Hi Mitch,
In our experience, -openmp 8 gives faster recons than -openmp 4, but the gains after -openmp 8 diminish quickly and aren't worth it.
-Paul
On Tue, May 3, 2022 at 12:51 PM Horn, Mitchell Jacob MJHORN@mgh.harvard.edu wrote:
Hi Experts,
If I have 100s of subjects to recon and have access to an HPC with both GPU and CPU options what is the recommended flags to maximize speed performance? Is … -parallel -openmp 4 … still the best approach? And if so, is 2GB memory/core best (keeping it 8GB memory/subject) or is there some better arrangement?
Best,
Mitch
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://secure-web.cisco.com/1KwX_rLAvesnrj3sVBEmrKlInc5WVsWyRLeOP9S9eAg0Osh... The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://secure-web.cisco.com/1WxqT2k67AYeX_6TtvedmLH3c-AlZH7neY7yJ3tCTkhZZWq... < https://secure-web.cisco.com/1WxqT2k67AYeX_6TtvedmLH3c-AlZH7neY7yJ3tCTkhZZWq... . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
Hi Paul,
Thanks for your speedy response! In that case, I assume 1GB/core is the memory allocation? Is it helpful to increase that?
Best, Mitch ________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Paul Wighton paul@corticometrics.com Sent: Tuesday, May 3, 2022 1:34 PM To: Freesurfer support list freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Scalable recon workflow
External Email - Use Caution
Hi Mitch,
In our experience, -openmp 8 gives faster recons than -openmp 4, but the gains after -openmp 8 diminish quickly and aren't worth it.
-Paul
On Tue, May 3, 2022 at 12:51 PM Horn, Mitchell Jacob <MJHORN@mgh.harvard.edumailto:MJHORN@mgh.harvard.edu> wrote: Hi Experts,
If I have 100s of subjects to recon and have access to an HPC with both GPU and CPU options what is the recommended flags to maximize speed performance? Is … -parallel -openmp 4 … still the best approach? And if so, is 2GB memory/core best (keeping it 8GB memory/subject) or is there some better arrangement?
Best, Mitch
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurferhttps://secure-web.cisco.com/1KwX_rLAvesnrj3sVBEmrKlInc5WVsWyRLeOP9S9eAg0OshRNiuN-Ah-YsWHXgPswXDNOuknm3WuIwW2uqT8Ifphx_C-TOwxjBeBqNgwbFZz87zgrJ2-2iGbZaVfaggfExMtQGY_r0NGX52-85cFulaPJfe_HIgruLaiqtXMlPtbqKZe782JlfYvS2yGDPs30UpkiKPUV6SKWlUrolMumbSHe3PrM-9pJfBMrYc3wieA6jitXpU3JvIYTTUhCaL2GK3ewzON3kmKGx-FgUbmkZj79FToVMWEy276qcNpNygxJFYl1L9xzBT7cdjTibKb-/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://www.massgeneralbrigham.org/compliancelinehttps://secure-web.cisco.com/1WxqT2k67AYeX_6TtvedmLH3c-AlZH7neY7yJ3tCTkhZZWqH_9u9jXF3UGKrGRC6ybXdqizqBFcN0wVMw2y9SijUuXXQI0bIUE9OhJAAqbEzmgtkwgPO-bvo6Oacen2R7AiLlg9E32BiOpwvVHdqYBRCmu0Pnf8YfBv_7q_yYLDJrJ-AMhmbsDFGN6NmQGdaTpHxj8Cz9KmZ3a5l5F__k7NJXxt1eF_CjRNQwG7l_XipuoMQPvv66uR28cK2S2Qju5ODWi6_oGcnkt22cqStgroiVnovgtnxZXFRtnbKwjUXySXWwrjmXyeNn4hYV4yJg/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline <MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://www.massgeneralbrigham.org/compliancelinehttps://secure-web.cisco.com/1WxqT2k67AYeX_6TtvedmLH3c-AlZH7neY7yJ3tCTkhZZWqH_9u9jXF3UGKrGRC6ybXdqizqBFcN0wVMw2y9SijUuXXQI0bIUE9OhJAAqbEzmgtkwgPO-bvo6Oacen2R7AiLlg9E32BiOpwvVHdqYBRCmu0Pnf8YfBv_7q_yYLDJrJ-AMhmbsDFGN6NmQGdaTpHxj8Cz9KmZ3a5l5F__k7NJXxt1eF_CjRNQwG7l_XipuoMQPvv66uR28cK2S2Qju5ODWi6_oGcnkt22cqStgroiVnovgtnxZXFRtnbKwjUXySXWwrjmXyeNn4hYV4yJg/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
There's a small percentage of subjects that need more than the 8Gb of memory (for mris_fix_topology if my memory is correct).
You could increase the memory allocation for all subjects, or you could run your entire dataset with 8Gb per subject, then re-run the ones that failed with increased memory.
-Paul ________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Horn, Mitchell Jacob MJHORN@mgh.harvard.edu Sent: Tuesday, May 3, 2022 2:26 PM To: Freesurfer support list freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Scalable recon workflow
Hi Paul,
Thanks for your speedy response! In that case, I assume 1GB/core is the memory allocation? Is it helpful to increase that?
Best, Mitch ________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Paul Wighton paul@corticometrics.com Sent: Tuesday, May 3, 2022 1:34 PM To: Freesurfer support list freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Scalable recon workflow
External Email - Use Caution
Hi Mitch,
In our experience, -openmp 8 gives faster recons than -openmp 4, but the gains after -openmp 8 diminish quickly and aren't worth it.
-Paul
On Tue, May 3, 2022 at 12:51 PM Horn, Mitchell Jacob <MJHORN@mgh.harvard.edumailto:MJHORN@mgh.harvard.edu> wrote:
Hi Experts,
If I have 100s of subjects to recon and have access to an HPC with both GPU and CPU options what is the recommended flags to maximize speed performance? Is … -parallel -openmp 4 … still the best approach? And if so, is 2GB memory/core best (keeping it 8GB memory/subject) or is there some better arrangement?
Best,
Mitch
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurferhttps://secure-web.cisco.com/1KwX_rLAvesnrj3sVBEmrKlInc5WVsWyRLeOP9S9eAg0OshRNiuN-Ah-YsWHXgPswXDNOuknm3WuIwW2uqT8Ifphx_C-TOwxjBeBqNgwbFZz87zgrJ2-2iGbZaVfaggfExMtQGY_r0NGX52-85cFulaPJfe_HIgruLaiqtXMlPtbqKZe782JlfYvS2yGDPs30UpkiKPUV6SKWlUrolMumbSHe3PrM-9pJfBMrYc3wieA6jitXpU3JvIYTTUhCaL2GK3ewzON3kmKGx-FgUbmkZj79FToVMWEy276qcNpNygxJFYl1L9xzBT7cdjTibKb-/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://www.massgeneralbrigham.org/compliancelinehttps://secure-web.cisco.com/1WxqT2k67AYeX_6TtvedmLH3c-AlZH7neY7yJ3tCTkhZZWqH_9u9jXF3UGKrGRC6ybXdqizqBFcN0wVMw2y9SijUuXXQI0bIUE9OhJAAqbEzmgtkwgPO-bvo6Oacen2R7AiLlg9E32BiOpwvVHdqYBRCmu0Pnf8YfBv_7q_yYLDJrJ-AMhmbsDFGN6NmQGdaTpHxj8Cz9KmZ3a5l5F__k7NJXxt1eF_CjRNQwG7l_XipuoMQPvv66uR28cK2S2Qju5ODWi6_oGcnkt22cqStgroiVnovgtnxZXFRtnbKwjUXySXWwrjmXyeNn4hYV4yJg/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline <MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://www.massgeneralbrigham.org/compliancelinehttps://secure-web.cisco.com/1WxqT2k67AYeX_6TtvedmLH3c-AlZH7neY7yJ3tCTkhZZWqH_9u9jXF3UGKrGRC6ybXdqizqBFcN0wVMw2y9SijUuXXQI0bIUE9OhJAAqbEzmgtkwgPO-bvo6Oacen2R7AiLlg9E32BiOpwvVHdqYBRCmu0Pnf8YfBv_7q_yYLDJrJ-AMhmbsDFGN6NmQGdaTpHxj8Cz9KmZ3a5l5F__k7NJXxt1eF_CjRNQwG7l_XipuoMQPvv66uR28cK2S2Qju5ODWi6_oGcnkt22cqStgroiVnovgtnxZXFRtnbKwjUXySXWwrjmXyeNn4hYV4yJg/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
btw, it will almost certainly be more efficient to run each subject with 1 thread and run them all at the same time rather than running a few of them with 4 or 8. If you can run them all with multiple threads, then that might be more efficient (depends on your scheduler, for slurm it would be).
On 5/3/2022 3:57 PM, Wighton, Paul wrote:
There's a small percentage of subjects that need more than the 8Gb of memory (for mris_fix_topology if my memory is correct).
You could increase the memory allocation for all subjects, or you could run your entire dataset with 8Gb per subject, then re-run the ones that failed with increased memory.
-Paul
*From:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Horn, Mitchell Jacob MJHORN@mgh.harvard.edu *Sent:* Tuesday, May 3, 2022 2:26 PM *To:* Freesurfer support list freesurfer@nmr.mgh.harvard.edu *Subject:* Re: [Freesurfer] Scalable recon workflow Hi Paul,
Thanks for your speedy response! In that case, I assume 1GB/core is the memory allocation? Is it helpful to increase that?
Best, Mitch
*From:* freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Paul Wighton paul@corticometrics.com *Sent:* Tuesday, May 3, 2022 1:34 PM *To:* Freesurfer support list freesurfer@nmr.mgh.harvard.edu *Subject:* Re: [Freesurfer] Scalable recon workflow
External Email - Use Caution
Hi Mitch,
In our experience, -openmp 8 gives faster recons than -openmp 4, but the gains after -openmp 8 diminish quickly and aren't worth it.
-Paul
On Tue, May 3, 2022 at 12:51 PM Horn, Mitchell Jacob MJHORN@mgh.harvard.edu wrote:
Hi Experts, If I have 100s of subjects to recon and have access to an HPC with both GPU and CPU options what is the recommended flags to maximize speed performance? Is … -parallel -openmp 4 … still the best approach? And if so, is 2GB memory/core best (keeping it 8GB memory/subject) or is there some better arrangement? Best, Mitch _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu *MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer <https://secure-web.cisco.com/1KwX_rLAvesnrj3sVBEmrKlInc5WVsWyRLeOP9S9eAg0OshRNiuN-Ah-YsWHXgPswXDNOuknm3WuIwW2uqT8Ifphx_C-TOwxjBeBqNgwbFZz87zgrJ2-2iGbZaVfaggfExMtQGY_r0NGX52-85cFulaPJfe_HIgruLaiqtXMlPtbqKZe782JlfYvS2yGDPs30UpkiKPUV6SKWlUrolMumbSHe3PrM-9pJfBMrYc3wieA6jitXpU3JvIYTTUhCaL2GK3ewzON3kmKGx-FgUbmkZj79FToVMWEy276qcNpNygxJFYl1L9xzBT7cdjTibKb-/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer> The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at *MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://www.massgeneralbrigham.org/complianceline <https://secure-web.cisco.com/1WxqT2k67AYeX_6TtvedmLH3c-AlZH7neY7yJ3tCTkhZZWqH_9u9jXF3UGKrGRC6ybXdqizqBFcN0wVMw2y9SijUuXXQI0bIUE9OhJAAqbEzmgtkwgPO-bvo6Oacen2R7AiLlg9E32BiOpwvVHdqYBRCmu0Pnf8YfBv_7q_yYLDJrJ-AMhmbsDFGN6NmQGdaTpHxj8Cz9KmZ3a5l5F__k7NJXxt1eF_CjRNQwG7l_XipuoMQPvv66uR28cK2S2Qju5ODWi6_oGcnkt22cqStgroiVnovgtnxZXFRtnbKwjUXySXWwrjmXyeNn4hYV4yJg/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline> <*MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://www.massgeneralbrigham.org/complianceline <https://secure-web.cisco.com/1WxqT2k67AYeX_6TtvedmLH3c-AlZH7neY7yJ3tCTkhZZWqH_9u9jXF3UGKrGRC6ybXdqizqBFcN0wVMw2y9SijUuXXQI0bIUE9OhJAAqbEzmgtkwgPO-bvo6Oacen2R7AiLlg9E32BiOpwvVHdqYBRCmu0Pnf8YfBv_7q_yYLDJrJ-AMhmbsDFGN6NmQGdaTpHxj8Cz9KmZ3a5l5F__k7NJXxt1eF_CjRNQwG7l_XipuoMQPvv66uR28cK2S2Qju5ODWi6_oGcnkt22cqStgroiVnovgtnxZXFRtnbKwjUXySXWwrjmXyeNn4hYV4yJg/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline>> . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
freesurfer@nmr.mgh.harvard.edu
-
Douglas N. Greve -
Horn, Mitchell Jacob -
Paul Wighton -
Wighton, Paul