Hello,
I'd like to suggest an alternative, for anyone interested in going faster than 4 hours. I can't verify compatibility from a technical standpoint, but GNU Parallel (https://www.gnu.org/software/parallel/) has worked fast and run to completion reliably when batching multiple recon-all subjects. You'll just need to determine your batch size based on your RAM (4GB per subject has worked ok for me). It takes about 20 hours on a good computer, which is equivalent to running recon-all on a single subject without parallelization. So, for example, with 32 subjects run together on 32 cores with 128GB of RAM available, it averages out to just over half an hour per subject.
First, you would just need to give parallel a text file of your raw data (your nifti files) and their directory. You could make one like so:
freesurferRawData=/path/to/your/raw/data ls $freesurferRawData | grep .nii > $freesurferRawData/subjects.txt
Next, you could decide a max number of subjects that you want to run simultaneously. Note that the number of subjects in your list can be larger than this number, or vice versa. You can set your maximum batch size based on your available RAM like so:
#max subjects that can be run with >= 4GB RAM available per subject maxSubjects=$( free -t -m | grep ^Total: | tr -s ' ' | cut -d ' ' -f 4 ) maxSubjects=$(( maxSubjects/4096 )) #trucated float < 1 [ $maxSubjects != 0 ] || maxSubjects=1
And then run parallel once you've set $maxSubjects, $freesurferRawData, and subjects.txt:
parallel -a $freesurferRawData/subjects.txt -j$maxSubjects --joblog $freesurferRawData/joblog.txt --progress --group recon-all -all -subjid {1.} -i $freesurferRawData/{1}
I've included some useful parallel flags:
a. joblog.txt will list all your completed subjects as they finish. b. a progress indicator in terminal lets you know how many are currently running and how many have completed c. rather than mixing up the text in real time, the entire recon-all terminal output will be display in the correct order once a subject has completed (--group) d. The subject ID in your subjects folder will be the name of the raw file without the extension
Regards, Chris
-----Original Message----- From: Z K [mailto:zkaufman@nmr.mgh.harvard.edu] Sent: Tuesday, February 14, 2017 9:09 AM To: freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Parallelization not working as implemented
I will have to inspect our Mac test results for parallel processing, but generally speaking we do not see much performance increase in terms of run times after using 4 threads.
On 02/13/2017 06:13 PM, Francis Tyson Thomas wrote:
I did provide the execution times in my first email. I have copied it once more for quick reference,
no parallelization : 7 hrs and 58 mins -parallel (coarse parallelization) : 4 hrs and 39 mins -parallel -openmp 8 (fine parallelization) : 4 hrs and 34 mins -parallel -openmp 14 (fine parallelization ) : 4 hrs and 37 mins
I have attached the logs to this email. The naming convention is as follows,
serial_recon-all.log : serial execution with no parallelization parallel_recon-all.log : parallelization using -parallel openmp8_recon-all.log : parallelization using -parallel -openmp 8 openmp14_recon-all.log : parallelization using -parallel -openmp 14
I'll rerun this with/-time/ later and see what I get. It is gonna take some time to get those logs.
Thanks, Tyson
On Mon, Feb 13, 2017 at 3:19 PM, Douglas N Greve <greve@nmr.mgh.harvard.edu mailto:greve@nmr.mgh.harvard.edu> wrote:
What are your execution times? Can you send the recon-all files for your parallel and non-parallel runs? If you run it with -time, then lots of information will be printed to the log for each command. This can be helpful for debugging these types of things. On 02/13/2017 05:13 PM, Francis Tyson Thomas wrote: > Hi, > > I have run recon-all with both -parallel enabled (including different > openmp threads) and disabled and I'm not able to get processing times > same as the information provided in recon-all help. The CPU used is > *Dual Intel **Xeon E5-2623 v3* paired with 32GB DDR4 RAM. Is the > configuration presented in recon-all help a dual cpu configuration? > > Following are the runtimes I have obtained, > > no parallelization : 7 hrs and 58 mins > -parallel (coarse parallelization) : 4 hrs and 39 mins > -parallel (fine parallelization; -openmp 8) : 4 hrs and 34 mins > -parallel (fine parallelization; -openmp 14) : 4 hrs and 37 mins > > Somehow varying the number of threads is having no effect in the > execution time of recon-all as can been seen from the processing > times. If you can explain what could be possibly going wrong here, it > will help me in speeding it up further. > > Thanks, > Tyson > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer <https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer> -- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu <mailto:greve@nmr.mgh.harvard.edu> Phone Number: 617-724-2358 <tel:617-724-2358> Fax: 617-726-7422 <tel:617-726-7422> Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting <http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting> FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 <https://gate.nmr.mgh.harvard.edu/filedrop2> www.nmr.mgh.harvard.edu/facility/filedrop/index.html <http://www.nmr.mgh.harvard.edu/facility/filedrop/index.html> Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/ <ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/> _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer <https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer> The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline <http://www.partners.org/complianceline> . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
===================================
Please consider the environment before printing this e-mail
Cleveland Clinic is ranked as one of the top hospitals in America by U.S.News & World Report (2015). Visit us online at http://www.clevelandclinic.org for a complete listing of our services, staff and locations.
Confidentiality Note: This message is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy.
Thank you.