External Email - Use Caution        

Hi John,

I know others (Satra?) might have more recent experience. I used to be able to get recon-all down to about 4 hours. OpenMP made the most difference but the GPU code always helped if you have it running. It really depends on whether you can get the admin to let you use the version of CUDA that FreeSurfer uses, etc. I was also always doing TRACULA too and the FSL GPU code is (IMO) essential. There are flags you add to recon-all to enable these. If you are having trouble with CUDA, OpenMP definitely speeds things up and try and stay on nodes with the most cores. 

It’s not the supercomputing that you need really. Check what the batch/queue system is there. Hopefully it’s SGE (or whatever its called now) or something that can mimic SGE. I know I will always have to modify the fsl_sub included in FreeSurfer to make sure this works properly but if its something different don’t worry and just disable it. Again, this problem normally comes up in TRACULA more than FS recon-all but you want to write a small shell script that starts all your subjects and then wait for it to be done. Sometimes the batch system will be where you also select the nodes/num of cores. 

Again, super-common:) Visualization is done on a local workstation unless you have an interactive node that, well, usually is very site specific. Can be useful as with a NVIDIA docker node on AWS if you want to see how it is sometimes done with GPU for visualization with something like ParaView. 

Hope that helps. 

Cheers,

-Morgan

On Fri, Dec 7, 2018 at 6:42 AM John Absher <JAbsher@ghs.org> wrote:

        External Email - Use Caution        

Hi,

 

I’m planning a freesurfer analysis of a large MRI dataset, and want to use the “380 GPU nodes” (and other cores/nodes) on the Palmetto Cluster (https://www.palmetto.clemson.edu/palmetto/userguide_palmetto_overview.html) to speed up the process. Since I am not a programmer, I’m hoping someone can give me a quick tutorial:

 

a)      Is this going to speed up recon-all and the data analysis?

b)      How much programming/expertise is required to enable freesurfer to take advantage of a supercomputer’s resources?

c)       Has anyone done this already?

d)      The Palmetto Cluster is more or less limited to command-line. As long as I visualize the data on another system, I assume this will not be a problem, right?

 

Thanks,

 

John R. Absher, MD

 

jabsher@ghs.org

GHS Neuroscience Associates

University of South Carolina School of Medicine Greenville

864-350-6655 (mobile)

 

_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer