Hi Joost -
Here's my understanding which comes from limited experience with PBS and SLURM. Your master process is a bash, tcsh, or other shell script with PBS directives embedded in it. Each recon-all command within your script should include a -openmp N switch which specifies the number of the threads you want to use for that freesurfer run. It looks like you get a total of 32 threads per core. Since a typical recon-all -all run will use multiple threads about 30% of the time, you must over-subscribe the number of threads to which you have access to use them efficiently. You have two key parameters to choose:
(1) #threads for each freesurfer run: I suggest 8.
(2) #simultaneous freesurfer instances running at any one time: I suggest starting with 10, i.e. 10 x 8 = 80 threads requested. You also have the problem of sensing when one of your freesurfer runs completes and starting another to keep 10 recon-all -all instances running. You can use the recon-all switch -notify to create a "completed" file for each freesurfer run and poll and count those files to decide when to spawn additional instances of recon-all.
I hope this is helpful. I think it would be interesting and valuable to hear what your experience is with this in whatever detail you can provide.
Best - Don
From: freesurfer-bounces@nmr.mgh.harvard.edu [mailto:freesurfer-bounces@nmr.mgh.harvard.edu] On Behalf Of Riphagen Joost (NP) Sent: Wednesday, December 16, 2015 6:01 AM To: freesurfer@nmr.mgh.harvard.edu Subject: [Freesurfer] example #PBS Torque script for FreeSurfer on a HPC cluster & multithreading question
Dear all,
I'm learning how to work with Freesurfer on a #PBS/Torque HPC cluster. There is enough documentation about using #PBS, but I have not been able to find anything on the net about how to process multiple subjects in FS in this environment.
If somebody is willing to share an example script that would be most helpful.
Am I right in thinking that by using the "-itkthreads" flag FS can make use of multiple threads but not use multiple cores ?
The nodes use two Xeon processors with 8 cores and 16 threads per CPU and have 64 GB per node. I have to book per node so the more I can use it the better.
Regards
Joost Riphagen