New to Freesurfer here, but I was wondering if there is any viability to the idea of parallelizing freesurfer for running on multiple servers so as to reduce the time required to run recon-all on a single subject. Is there an aspect of the algorithm that makes this inherently infeasible?
Spencer
Hi Spencer,
the easy answer is that we have a GPU implementation of several binaries that speed things up. The parallelization ranges from easy to impossible, but we are working on it. We don't have any generic (e.g. MPI) parallization at the moment, just some CUDA code.
cheers Bruce
On Mon, 3 Oct 2011, Spencer Boucher wrote:
New to Freesurfer here, but I was wondering if there is any viability to the idea of parallelizing freesurfer for running on multiple servers so as to reduce the time required to run recon-all on a single subject. Is there an aspect of the algorithm that makes this inherently infeasible?
Spencer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
You can also parallelize by hemisphere by running a single job up to the tessellation stage (about 8 hours), then two jobs (one for each hemi). In serial these 2nd steps take about 8 hours, so you could in theory save 4 hours. doug
Bruce Fischl wrote:
Hi Spencer,
the easy answer is that we have a GPU implementation of several binaries that speed things up. The parallelization ranges from easy to impossible, but we are working on it. We don't have any generic (e.g. MPI) parallization at the moment, just some CUDA code.
cheers Bruce
On Mon, 3 Oct 2011, Spencer Boucher wrote:
New to Freesurfer here, but I was wondering if there is any viability to the idea of parallelizing freesurfer for running on multiple servers so as to reduce the time required to run recon-all on a single subject. Is there an aspect of the algorithm that makes this inherently infeasible?
Spencer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
On Mon, Oct 3, 2011 at 12:22 PM, Spencer Boucher sboucher@stanford.edu wrote:
New to Freesurfer here, but I was wondering if there is any viability to the idea of parallelizing freesurfer for running on multiple servers so as to reduce the time required to run recon-all on a single subject. Is there an aspect of the algorithm that makes this inherently infeasible?
Given that essentially everything these days is multicore, running on multiple servers wouldn't even be necessary - you could do it with OpenMP (Multiple servers running MPI would only be necessary once the new 1000T MRI machines start producing 8192^3 data sets with good SNR). Indeed a lot of algorithms within Freesurfer are well suited to such parallelisation, since they are basically independent iterations over all the voxels in an image (this is how the GPU version works). Unfortunately, the detailed implementation of these algorithms in the current codebase is not always optimal for this. It might be worthwhile experimenting with a couple of carefully placed OpenMP pragmas, though - the convolution routines would be an ideal starting point, since they're already separated out into a unit test (under cudatest) with timers attached.
Richard
freesurfer@nmr.mgh.harvard.edu