External Email - Use Caution
Hi Alex,
I am also running FreeSurfer 6.0.0 on SLURM and recently did some tests on how to parallelize it, how much ressources to ask for… This is what I learned:
1) FreeSurfer 6.0.0 is able to use OpenMP parallelization (I guess for the commands mri_em_register, mri_ca_register and mri_ca label used in the steps Skull Stripping, EM registration, CA reg and Subcortical registration). 2) From Version 6.0.0 FreeSurfer is also able to process the two hemispheres in parallel instead of sequentially (steps 16 – 30).
Taking advantage of these changes reduces processing time dramatically. If I use 4 CPU per task on SLURM instead of 1 the processing time drops form around 8 hours to around 4 or 4 ½ hours. Increasing the CPU per task did not improve processing time much though, so to save a bit CPU time now use 4 CPU regularly. I don’t know if you have to submit your jobs exactly the same way but a typical job file for me looks like this:
------------------------------------------------------------------------------------ #!/bin/bash #SBATCH --time=10:00:00 #SBATCH --output=/your_working_directory/job-%j.out #SBATCH --error=/your_working_directory/job-%j.err #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --tasks-per-node=1 #SBATCH --cpus-per-task=4 #SBATCH --partition=haswell #SBATCH --mem-per-cpu=2500 #SBATCH -A your_project_name
module load modenv/eb module load FreeSurfer source $FREESURFER_HOME/SetUpFreeSurfer.sh
export SUBJECTS_DIR=/your_subjects_directory
recon-all -s your_subject_ID -all -parallel -openmp 4 --------------------------------------------------------------------------------------
Of course you can just add the –hippocampal-subfields-T1 tag flag your command. I found that asking for 2500 MB works just fine for 4CPU per task (as the 2500 is multiplied times 4 in this case). The maximum RAM was about 7000 KB for my jobs. Just make sure you add the” -parallel -openmp 4” flag to your command line AND “#SBATCH --cpus-per-task=4” to your job file, otherwise the parallelization will not work.
Hope this helps! Get back to me if you have any more questions!
Best, Lea
------------------------------ Lea Backhausen Research Assistant
Department of Child and Adolescent Psychiatry, Faculty of Medicine of the TU Dresden, Germany http://www.uniklinikum-dresden.de
------------------------------
Message: 5 Date: Thu, 19 Apr 2018 17:05:46 +0200 From: Alexandre Obert obert.alexandre@gmail.com Subject: [Freesurfer] SLURM - speed processing To: freesurfer@nmr.mgh.harvard.edu Message-ID: CAGTEUh_XypWu4h+4ban62BfH8idf96HEquUL+OMMaWT_GEk4Cw@mail.gmail.com Content-Type: text/plain; charset="utf-8"
External Email - Use Caution
Hi all,
I'm definitively a noob with both freesurfer and SLURM processes and I wonder if anyone could give me some advice. I tried processing a subject (including hippocampal segmentation) using the following command (freesurfer 6.0.0):
recon-all -all -s s03 -hippocampal-subfields-T1
It took nearly 12h to process. I then submit this job to a supercomputer (using slurm) and I asked for nearly 30Go RAM It took nearly 7h to process. That seems good but I wonder if it's possible to speed it up a little more?
Regards,
Alex