SLURM - speed processing - Freesurfer

20 Apr 2018

      External Email - Use Caution
Hi Lea,
Thanks for the tips!
I think I did something similar for a previous test without hipo
segmentation (took nearly 3 to 4h).
I will check the way I've got to set the number of cpu per tasks, just in
case.
Thanks
Alex
...
Message: 20
Date: Fri, 20 Apr 2018 09:19:12 +0000
From: "Backhausen, Lea" Lea.Backhausen@uniklinikum-dresden.de
Subject: Re: [Freesurfer] SLURM - speed processing (Alexandre Obert)
To: "'freesurfer@nmr.mgh.harvard.edu'"
        freesurfer@nmr.mgh.harvard.edu
Message-ID:
        <1D0E7A9ED60D3E4399550C3A60C0BADC394606B6@G06EDBN1.med.tu-dr
esden.de>
Content-Type: text/plain; charset="utf-8"
    External Email - Use Caution

Hi Alex,
I am also running FreeSurfer 6.0.0 on SLURM and recently did some tests on
how to parallelize it, how much ressources to ask for? This is what I
learned:

 FreeSurfer 6.0.0 is able to use OpenMP parallelization (I guess

for the commands mri_em_register, mri_ca_register and mri_ca label used in
the steps Skull Stripping, EM registration, CA reg and Subcortical
registration).
2)      From Version 6.0.0 FreeSurfer is also able to process the two
hemispheres in parallel instead of sequentially (steps 16 ? 30).
Taking advantage of these changes reduces processing time dramatically. If
I use 4 CPU per task on SLURM instead of 1 the processing time drops form
around 8 hours to around 4 or 4 ? hours. Increasing the CPU per task did
not improve processing time much though, so to save a bit CPU time now use
4 CPU regularly.
I don?t know if you have to submit your jobs exactly the same way but a
typical job file for me looks like this:

#!/bin/bash
#SBATCH --time=10:00:00
#SBATCH --output=/your_working_directory/job-%j.out
#SBATCH --error=/your_working_directory/job-%j.err
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=4
#SBATCH --partition=haswell
#SBATCH --mem-per-cpu=2500
#SBATCH -A your_project_name
module load modenv/eb
module load FreeSurfer
source $FREESURFER_HOME/SetUpFreeSurfer.sh
export SUBJECTS_DIR=/your_subjects_directory
recon-all -s your_subject_ID -all -parallel -openmp 4

Of course you can just add the ?hippocampal-subfields-T1 tag flag your
command.
I found that asking for 2500 MB works just fine for 4CPU per task (as the
2500 is multiplied times 4 in this case). The maximum RAM was about 7000 KB
for my jobs.
Just make sure you add the? -parallel -openmp 4? flag to your command line
AND ?#SBATCH --cpus-per-task=4? to your job file, otherwise the
parallelization will not work.
Hope this helps! Get back to me if you have any more questions!
Best,
Lea

Lea Backhausen
Research Assistant
Department of Child and Adolescent Psychiatry, Faculty of Medicine of the
TU Dresden, Germany
http://www.uniklinikum-dresden.de

Message: 5
Date: Thu, 19 Apr 2018 17:05:46 +0200
From: Alexandre Obert obert.alexandre@gmail.com
Subject: [Freesurfer] SLURM - speed processing
To: freesurfer@nmr.mgh.harvard.edu
Message-ID:
        <CAGTEUh_XypWu4h+4ban62BfH8idf96HEquUL+OMMaWT_GEk4Cw@mail.
gmail.com>
Content-Type: text/plain; charset="utf-8"
    External Email - Use Caution

Hi all,
I'm definitively a noob with both freesurfer and SLURM processes and I
wonder if anyone could give me some advice.
I tried processing a subject (including hippocampal segmentation) using
the following command (freesurfer 6.0.0):
recon-all -all -s s03 -hippocampal-subfields-T1
It took nearly 12h to process.
I then submit this job to a supercomputer (using slurm) and I asked for
nearly 30Go RAM It took nearly 7h to process. That seems good but I wonder
if it's possible to speed it up a little more?
Regards,
Alex