I have recently upgraded our server from v5.0.0 to v5.3.0 running on CentOS 5.10 and was a little surprised that there wasnt a version of Freesurfer built for this platform but rather 4 and 6. I guess my main question is, was the version for centos4 built with openmpi at all? THe server we are running Freesurfer on has 48 cores and we are trying to figure out how to make sure that while running Freesurfer we take advantage of all these cores. Is there any documentation that i can look at to help me in this endeavor or does anyone have any tips they can share?
Thank you in advance Michael Weiner Lead Analyst/UNIX Systems Administrator Lerner Research Institute Computing Services Lerner Research Institute Cleveland Clinic
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by U.S.News & World Report (2013). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations.
Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy.
Thank you.
Hello Michael,
Use the Freesurfer Centos4 build for your Centos5 platform. People who want to take advantage of of multiple cores typically use the "-openmp <num cores>" flag when running recon-all. Theoretically the more cores you use the faster the runtime, but we have found that that there is little gain after 8 and when using that many we typically only see a 25% decrease in the time it takes to run the entire recon-all stream.
You could try and get fancy by writing up a script and breaking up the reson-all stream into parts and parrallelizing some of the processing of the left and right hemispere... something along the lines of:
recon-all -s subj -autorecon1 -openmp 8 recon-all -s subj -autorecon2-volonly -openmp 8 recon-all -s subj -autorecon2-perhemi -hemi rh -log recon-all-hemi rh.log -notify rh-done.log -openmp 8 & recon-all -s subj -autorecon2-perhemi -hemi lh -log recon-all-hemi-lh.log -notify lh-done.log -openmp 8 & while (rh-done.log AND lh-done.log do not both exist) do sleep 1 end loop recon-all -s subj -autorecon3 -openmp 8
-Zeke
On 03/03/2014 10:21 AM, Weiner, Michael wrote:
I have recently upgraded our server from v5.0.0 to v5.3.0 running on CentOS 5.10 and was a little surprised that there wasnt a version of Freesurfer built for this platform but rather 4 and 6. I guess my main question is, was the version for centos4 built with openmpi at all? THe server we are running Freesurfer on has 48 cores and we are trying to figure out how to make sure that while running Freesurfer we take advantage of all these cores. Is there any documentation that i can look at to help me in this endeavor or does anyone have any tips they can share?
Thank you in advance Michael Weiner Lead Analyst/UNIX Systems Administrator Lerner Research Institute Computing Services Lerner Research Institute Cleveland Clinic
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by U.S.News & World Report (2013). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations. Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Thank you.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Thank you for the information, I wasn't sure if the centos4 build had mp built into it. I will try this and forward the information on to the researcher to see how their runs can be improved.
Regards Michael weiner
-----Original Message----- From: Z K [mailto:zkaufman@nmr.mgh.harvard.edu] Sent: Tuesday, March 04, 2014 1:17 PM To: Weiner, Michael; freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Freesurfer 5.3 and CentOS 5.10
Hello Michael,
Use the Freesurfer Centos4 build for your Centos5 platform. People who want to take advantage of of multiple cores typically use the "-openmp <num cores>" flag when running recon-all. Theoretically the more cores you use the faster the runtime, but we have found that that there is little gain after 8 and when using that many we typically only see a 25% decrease in the time it takes to run the entire recon-all stream.
You could try and get fancy by writing up a script and breaking up the reson-all stream into parts and parrallelizing some of the processing of the left and right hemispere... something along the lines of:
recon-all -s subj -autorecon1 -openmp 8 recon-all -s subj -autorecon2-volonly -openmp 8 recon-all -s subj -autorecon2-perhemi -hemi rh -log recon-all-hemi rh.log -notify rh-done.log -openmp 8 & recon-all -s subj -autorecon2-perhemi -hemi lh -log recon-all-hemi-lh.log -notify lh-done.log -openmp 8 & while (rh-done.log AND lh-done.log do not both exist) do sleep 1 end loop recon-all -s subj -autorecon3 -openmp 8
-Zeke
On 03/03/2014 10:21 AM, Weiner, Michael wrote:
I have recently upgraded our server from v5.0.0 to v5.3.0 running on CentOS 5.10 and was a little surprised that there wasnt a version of Freesurfer built for this platform but rather 4 and 6. I guess my main
question is, was the version for centos4 built with openmpi at all? THe server we are running Freesurfer on has 48 cores and we are trying
to figure out how to make sure that while running Freesurfer we take advantage of all these cores. Is there any documentation that i can look at to help me in this endeavor or does anyone have any tips they
can share?
Thank you in advance Michael Weiner Lead Analyst/UNIX Systems Administrator Lerner Research Institute Computing Services Lerner Research Institute Cleveland Clinic
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by U.S.News & World Report (2013). Visit us online at http://www.clevelandclinic.org for a complete listing of our services,
staff and locations. Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is
not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Thank you.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by U.S.News & World Report (2013). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations.
Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy.
Thank you.
Micheal,
Im sorry but I sent you conflicting information.
The Centos4 build does NOT have openmp built into it. Our Centos4 builds use gcc version 3.4.6. It wasnt till gcc 4.2 that the gcc compiler implement openmp (v2.5)
Our CentOS6 build uses gcc 4.4.7 and does have openmp built into it. Sorry for the confusion.
-Zeke
On 03/05/2014 10:15 AM, Weiner, Michael wrote:
Thank you for the information, I wasn't sure if the centos4 build had mp built into it. I will try this and forward the information on to the researcher to see how their runs can be improved.
Regards Michael weiner
-----Original Message----- From: Z K [mailto:zkaufman@nmr.mgh.harvard.edu] Sent: Tuesday, March 04, 2014 1:17 PM To: Weiner, Michael; freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Freesurfer 5.3 and CentOS 5.10
Hello Michael,
Use the Freesurfer Centos4 build for your Centos5 platform. People who want to take advantage of of multiple cores typically use the "-openmp <num cores>" flag when running recon-all. Theoretically the more cores you use the faster the runtime, but we have found that that there is little gain after 8 and when using that many we typically only see a 25% decrease in the time it takes to run the entire recon-all stream.
You could try and get fancy by writing up a script and breaking up the reson-all stream into parts and parrallelizing some of the processing of the left and right hemispere... something along the lines of:
recon-all -s subj -autorecon1 -openmp 8 recon-all -s subj -autorecon2-volonly -openmp 8 recon-all -s subj -autorecon2-perhemi -hemi rh -log recon-all-hemi rh.log -notify rh-done.log -openmp 8 & recon-all -s subj -autorecon2-perhemi -hemi lh -log recon-all-hemi-lh.log -notify lh-done.log -openmp 8 & while (rh-done.log AND lh-done.log do not both exist) do sleep 1 end loop recon-all -s subj -autorecon3 -openmp 8
-Zeke
On 03/03/2014 10:21 AM, Weiner, Michael wrote:
I have recently upgraded our server from v5.0.0 to v5.3.0 running on CentOS 5.10 and was a little surprised that there wasnt a version of Freesurfer built for this platform but rather 4 and 6. I guess my main
question is, was the version for centos4 built with openmpi at all? THe server we are running Freesurfer on has 48 cores and we are trying
to figure out how to make sure that while running Freesurfer we take advantage of all these cores. Is there any documentation that i can look at to help me in this endeavor or does anyone have any tips they
can share?
Thank you in advance Michael Weiner Lead Analyst/UNIX Systems Administrator Lerner Research Institute Computing Services Lerner Research Institute Cleveland Clinic
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by U.S.News & World Report (2013). Visit us online at http://www.clevelandclinic.org for a complete listing of our services,
staff and locations. Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is
not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. Thank you.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by U.S.News & World Report (2013). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations.
Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy.
Thank you.
Zeke -
Thank you for getting back in touch with me. I have since grabbed the source distribution to see if I can compile this beast on my own ... quite the arduous task indeed. It would be great to see the configure options that were used to create the public stable 5 release, still looking around for as much information as I can. I want to mimic the centos6 build on a CentOS 5 server as we haven't migrated all our servers to 6 yet.
Thanks again Michael
-----Original Message----- From: Z K [mailto:zkaufman@nmr.mgh.harvard.edu] Sent: Wednesday, March 05, 2014 11:14 AM To: Weiner, Michael; freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Freesurfer 5.3 and CentOS 5.10
Micheal,
Im sorry but I sent you conflicting information.
The Centos4 build does NOT have openmp built into it. Our Centos4 builds use gcc version 3.4.6. It wasnt till gcc 4.2 that the gcc compiler implement openmp (v2.5)
Our CentOS6 build uses gcc 4.4.7 and does have openmp built into it. Sorry for the confusion.
-Zeke
On 03/05/2014 10:15 AM, Weiner, Michael wrote:
Thank you for the information, I wasn't sure if the centos4 build had mp built into it. I will try this and forward the information on to the researcher to see how their runs can be improved.
Regards Michael weiner
-----Original Message----- From: Z K [mailto:zkaufman@nmr.mgh.harvard.edu] Sent: Tuesday, March 04, 2014 1:17 PM To: Weiner, Michael; freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Freesurfer 5.3 and CentOS 5.10
Hello Michael,
Use the Freesurfer Centos4 build for your Centos5 platform. People who
want to take advantage of of multiple cores typically use the "-openmp
<num cores>" flag when running recon-all. Theoretically the more cores
you use the faster the runtime, but we have found that that there is little gain after 8 and when using that many we typically only see a 25% decrease in the time it takes to run the entire recon-all stream.
You could try and get fancy by writing up a script and breaking up the
reson-all stream into parts and parrallelizing some of the processing of the left and right hemispere... something along the lines of:
recon-all -s subj -autorecon1 -openmp 8 recon-all -s subj -autorecon2-volonly -openmp 8 recon-all -s subj -autorecon2-perhemi -hemi rh -log recon-all-hemi rh.log -notify rh-done.log -openmp 8 & recon-all -s subj -autorecon2-perhemi -hemi lh -log recon-all-hemi-lh.log -notify lh-done.log -openmp 8 & while (rh-done.log AND lh-done.log do not both exist) do sleep 1 end loop recon-all -s subj -autorecon3 -openmp 8
-Zeke
On 03/03/2014 10:21 AM, Weiner, Michael wrote:
I have recently upgraded our server from v5.0.0 to v5.3.0 running on CentOS 5.10 and was a little surprised that there wasnt a version of Freesurfer built for this platform but rather 4 and 6. I guess my main
question is, was the version for centos4 built with openmpi at all? THe server we are running Freesurfer on has 48 cores and we are trying
to figure out how to make sure that while running Freesurfer we take advantage of all these cores. Is there any documentation that i can look at to help me in this endeavor or does anyone have any tips they
can share?
Thank you in advance Michael Weiner Lead Analyst/UNIX Systems Administrator Lerner Research Institute Computing Services Lerner Research Institute Cleveland Clinic
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by U.S.News & World Report (2013). Visit us online at http://www.clevelandclinic.org for a complete listing of our services,
staff and locations. Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and
may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is
not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard
copy.
Thank you.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom
it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline
. If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by
U.S.News & World Report (2013).
Visit us online at http://www.clevelandclinic.org for a complete
listing of our services, staff and locations.
Confidentiality Note: This message is intended for use only by the
individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy.
Thank you.
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by U.S.News & World Report (2013). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations.
Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy.
Thank you.
freesurfer@nmr.mgh.harvard.edu
-
Weiner, Michael -
Z K