Hi FSers,
I have an Ubuntu machine (14.04 LTS) with a quad-core i7 4770k/32gb RAM and recon-all is taking me approximately 6 hours per subject without using openmp (haven't tried this yet). According to the terminal, it finishes without errors. I've read (e.g., https://surfer.nmr.mgh.harvard.edu/fswiki/FsQuizAnswers) and heard that recon-all should take 20-24 hours per participant to complete. So, should I be celebrating my 6 hour run time or should I be very suspicious of it?
Also, how does freesurfer go with hyper threaded CPUs? (as is the case with the 4770k; 4 cores, 8 threads). I've read that you should restrict parallel processing to one subject per core. Can I take advantage of the hyper threading and run 8 subjects simultaneously or should I be more conservative and only run 4?
Cheers, Linden
When in doubt, check the tissue and surface segmentations (https://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/OutputData_freeview) but ultimately celebrations seem to be in order...
__________________________ Peter Goodin, BSc (Hons), Ph.D Candidate.
Brain and Psychological Sciences Research Centre (BPsych) Swinburne University, Hawthorn, Vic, 3122 http://www.swinburne.edu.au/swinburneresearchers/index.php?fuseaction=profil...
Monash Alfred Psychiatry Research Centre (MAPrc) Level 4, 607 St Kilda Road, Melbourne 3004 ________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of Linden Parkes [linden.parkes@monash.edu] Sent: Tuesday, 17 June 2014 8:50 PM To: freesurfer@nmr.mgh.harvard.edu Subject: [Freesurfer] Version 5.3 recon-all run time
Hi FSers,
I have an Ubuntu machine (14.04 LTS) with a quad-core i7 4770k/32gb RAM and recon-all is taking me approximately 6 hours per subject without using openmp (haven't tried this yet). According to the terminal, it finishes without errors. I've read (e.g., https://surfer.nmr.mgh.harvard.edu/fswiki/FsQuizAnswers) and heard that recon-all should take 20-24 hours per participant to complete. So, should I be celebrating my 6 hour run time or should I be very suspicious of it?
Also, how does freesurfer go with hyper threaded CPUs? (as is the case with the 4770k; 4 cores, 8 threads). I've read that you should restrict parallel processing to one subject per core. Can I take advantage of the hyper threading and run 8 subjects simultaneously or should I be more conservative and only run 4?
Cheers, Linden
yes, the 20-24 hour numbers are several years old at this point, and we have made some improvements to speed so 6 hours is not unheard of. That said, definitely visually inspect the results! I'll leave the hyperthreading question for Zeke or Nick
cheers Bruce On Tue, 17 Jun 2014, Peter Goodin wrote:
When in doubt, check the tissue and surface segmentations (https://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/OutputData_freeview) but ultimately celebrations seem to be in order...
Peter Goodin, BSc (Hons), Ph.D Candidate.
Brain and Psychological Sciences Research Centre (BPsych) Swinburne University, Hawthorn, Vic, 3122 http://www.swinburne.edu.au/swinburneresearchers/index.php?fuseaction=profi le&pid=4149
Monash Alfred Psychiatry Research Centre (MAPrc) Level 4, 607 St Kilda Road, Melbourne 3004
From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of Linden Parkes [linden.parkes@monash.edu] Sent: Tuesday, 17 June 2014 8:50 PM To: freesurfer@nmr.mgh.harvard.edu Subject: [Freesurfer] Version 5.3 recon-all run time
Hi FSers,
I have an Ubuntu machine (14.04 LTS) with a quad-core i7 4770k/32gb RAM and recon-all is taking me approximately 6 hours per subject without using openmp (haven't tried this yet). According to the terminal, it finishes without errors. I've read (e.g., https://surfer.nmr.mgh.harvard.edu/fswiki/FsQuizAnswers) and heard that recon-all should take 20-24 hours per participant to complete. So, should I be celebrating my 6 hour run time or should I be very suspicious of it?
Also, how does freesurfer go with hyper threaded CPUs? (as is the case with the 4770k; 4 cores, 8 threads). I've read that you should restrict parallel processing to one subject per core. Can I take advantage of the hyper threading and run 8 subjects simultaneously or should I be more conservative and only run 4?
Cheers, Linden
Linden,
Its my understanding that that with hyperthreading enabled, you can run a subject for each processor the computer "thinks" it has. For instance, on CentOS (Im not sure the equivalent command on Ubuntu) if I 'cat /proc/cpuinfo' I see 4 processors listed. In this case I would be able to process 4 subjects using the --openmp 4 flag. Though it is sometimes recommended to process n-1 subjects in order to keep one processor available to the user.
Something to keep in mind is the available RAM. We recommend 4gigs of RAM per subject so in the case of 4 subjects, the machine should have 16gig of RAM available. And if you are going to process 8 subjects than you would want 32gigs.
-Zeke
On 06/17/2014 06:50 AM, Linden Parkes wrote:
Hi FSers,
I have an Ubuntu machine (14.04 LTS) with a quad-core i7 4770k/32gb RAM and recon-all is taking me approximately 6 hours per subject without using openmp (haven't tried this yet). According to the terminal, it finishes without errors. I've read (e.g., https://surfer.nmr.mgh.harvard.edu/fswiki/FsQuizAnswers) and heard that recon-all should take 20-24 hours per participant to complete. So, should I be celebrating my 6 hour run time or should I be very suspicious of it?
Also, how does freesurfer go with hyper threaded CPUs? (as is the case with the 4770k; 4 cores, 8 threads). I've read that you should restrict parallel processing to one subject per core. Can I take advantage of the hyper threading and run 8 subjects simultaneously or should I be more conservative and only run 4?
Cheers, Linden
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Hi Peter, Bruce, and Zeke,
Thank you all for the various input. According to the tutorial Peter posted, the handful of subjects I've done look good. I ran another subject with -openmp 4 flag and it shaved the processing down to <4 hours, so that is a very nice saving! The System Monitor app in Ubuntu indeed lists 8 CPUs so the hyper threading makes Ubuntu "think" it has 8 cores. Given the 32gb of ram I have, I'll try -openmp 8 flag and see how it goes.
Does anyone have some tips on how to setup parallel processing? Is it better to set up, say, 4 participants all with the -openmp 2 flag in 4 separate tcsh sessions in order to saturate the 8 cores and 32gb ram? Or is there a more sophisticated way of doing it that I should consider?
Cheers, Linden
On 18 June 2014 07:27, Z K zkaufman@nmr.mgh.harvard.edu wrote:
Linden,
Its my understanding that that with hyperthreading enabled, you can run a subject for each processor the computer "thinks" it has. For instance, on CentOS (Im not sure the equivalent command on Ubuntu) if I 'cat /proc/cpuinfo' I see 4 processors listed. In this case I would be able to process 4 subjects using the --openmp 4 flag. Though it is sometimes recommended to process n-1 subjects in order to keep one processor available to the user.
Something to keep in mind is the available RAM. We recommend 4gigs of RAM per subject so in the case of 4 subjects, the machine should have 16gig of RAM available. And if you are going to process 8 subjects than you would want 32gigs.
-Zeke
On 06/17/2014 06:50 AM, Linden Parkes wrote:
Hi FSers,
I have an Ubuntu machine (14.04 LTS) with a quad-core i7 4770k/32gb RAM and recon-all is taking me approximately 6 hours per subject without using openmp (haven't tried this yet). According to the terminal, it finishes without errors. I've read (e.g., https://surfer.nmr.mgh.harvard.edu/fswiki/FsQuizAnswers) and heard that recon-all should take 20-24 hours per participant to complete. So, should I be celebrating my 6 hour run time or should I be very suspicious of it?
Also, how does freesurfer go with hyper threaded CPUs? (as is the case with the 4770k; 4 cores, 8 threads). I've read that you should restrict parallel processing to one subject per core. Can I take advantage of the hyper threading and run 8 subjects simultaneously or should I be more conservative and only run 4?
Cheers, Linden
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Linden,
The fastest runtime that I have seen is just under 3.5 hours using a new and very fast system such as yours, using the -openmp 8 flag.
Processing four participants in separate tcsh sessions is the recommended way of 'parallel' processing. There are flags in recon-all to allow splitting the pipeline into two when the hemisphere processing starts, but thats more hassle than its worth, given that people usually have multiple subjects, so 'spending' a cpu on each subject is better than speeding-up any one subject.
I'd recommend staggering the start times of your subjects a bit, by say 20 minutes.
Nick
On Wed, 2014-06-18 at 09:55 +1000, Linden Parkes wrote:
Hi Peter, Bruce, and Zeke,
Thank you all for the various input.
According to the tutorial Peter posted, the handful of subjects I've done look good. I ran another subject with -openmp 4 flag and it shaved the processing down to <4 hours, so that is a very nice saving!
The System Monitor app in Ubuntu indeed lists 8 CPUs so the hyper threading makes Ubuntu "think" it has 8 cores. Given the 32gb of ram I have, I'll try -openmp 8 flag and see how it goes.
Does anyone have some tips on how to setup parallel processing? Is it better to set up, say, 4 participants all with the -openmp 2 flag in 4 separate tcsh sessions in order to saturate the 8 cores and 32gb ram? Or is there a more sophisticated way of doing it that I should consider?
Cheers, Linden
On 18 June 2014 07:27, Z K zkaufman@nmr.mgh.harvard.edu wrote: Linden,
Its my understanding that that with hyperthreading enabled, you can run a subject for each processor the computer "thinks" it has. For instance, on CentOS (Im not sure the equivalent command on Ubuntu) if I 'cat /proc/cpuinfo' I see 4 processors listed. In this case I would be able to process 4 subjects using the --openmp 4 flag. Though it is sometimes recommended to process n-1 subjects in order to keep one processor available to the user. Something to keep in mind is the available RAM. We recommend 4gigs of RAM per subject so in the case of 4 subjects, the machine should have 16gig of RAM available. And if you are going to process 8 subjects than you would want 32gigs. -Zeke On 06/17/2014 06:50 AM, Linden Parkes wrote: > Hi FSers, > > I have an Ubuntu machine (14.04 LTS) with a quad-core i7 4770k/32gb RAM > and recon-all is taking me approximately 6 hours per subject without > using openmp (haven't tried this yet). According to the terminal, it > finishes without errors. > I've read (e.g., > https://surfer.nmr.mgh.harvard.edu/fswiki/FsQuizAnswers) and heard that > recon-all should take 20-24 hours per participant to complete. So, > should I be celebrating my 6 hour run time or should I be very > suspicious of it? > > Also, how does freesurfer go with hyper threaded CPUs? (as is the case > with the 4770k; 4 cores, 8 threads). I've read that you should restrict > parallel processing to one subject per core. Can I take advantage of the > hyper threading and run 8 subjects simultaneously or should I be more > conservative and only run 4? > > Cheers, > Linden > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
freesurfer@nmr.mgh.harvard.edu
-
Bruce Fischl -
Linden Parkes -
Nick Schmansky, MGH -
Peter Goodin -
Z K