At The Uni. of Chicago, we've been playing around with parallelizing freesurfer on a 128 node grid (256 processors). Developers have parallelized procedures (scripts) by unpacking "for" loops that rotate across left and right hemispheres [i.e., they fork the independent processing of left and right hemispheres to different nodes running in parallel, whenever possible].
The main point of this work is to acquire provenance records and therefore Freesurfer scripts are "wrapped" or expressed using a virtual data system language (VDS/VDL). The freesurfer implementation, AFAIK, is in its baby steps, but the general workflow model is pretty well established http://www.ci.uchicago.edu/wiki/bin/view/VDS/VDSWeb/WebMain
Best, Uri
On 12/6/06, Uri Hasson uhasson@gmail.com wrote:
Here (Uni. of Chicago), we've been playing around with parallelizing freesurfer on a 128 node grid (256 processors), and developers have parallelized procedures (scripts) by unpacking "for" loops that rotate across left and right hemispheres [i.e., they fork the independent processing of left and right hemispheres to different nodes running in parallel, whenever possible].
The main point of this work is to acquire provenance records and therefore Freesurfer scripts are "wrapped" or expressed using a virtual data system language (VDS/VDL). The freesurfer implementation, AFAIK, is in its baby steps, but the general workflow model is pretty well established http://www.ci.uchicago.edu/wiki/bin/view/VDS/VDSWeb/WebMain
Best, Uri
On 12/6/06, Nick Schmansky nicks@nmr.mgh.harvard.edu wrote:
Andries,
I am not aware of usage of Freesurfer in a (Sun) Grid Engine environment (such as that used by the Cohen group at UCLA).
However, here at the MGH/MIT/HMS Martinos Center we use a cluster of some 100+ nodes configured with Linux Centos 4, and governed by PBS (Portable Batch System). Researchers here often conduct studies with dozens to hundreds of brains, and for each subject, an instance of Freesurfer's 'recon-all -s <subject> -all' script is submitted to the batch system, which, under the hood, gets submitted to one computing node. Thus, several dozen brains can be processed in a day (and a half).
Freesurfer does not currently support fine-grain parallelism. Some coarse-grain parallelism, whereby each brain hemisphere is processed independently (benefiting multiprocessor nodes) is possible, but not currently implemented in our 'recon-all' script, as the error handling and logging for doing so is somewhat tricky (and so this feature is in- the-works-but-not-anytime-soon).
In short, if you plan on using Freesurfer in studies with large numbers of subjects, I would recommend some kind of computing cluster, and some fairly simple batch software (like PBS) should be sufficient. For instance, I know of one group that has successfully run Freesurfer on their Altix Itanium Linux cluster.
Groetjes,
Nick
On Wed, 2006-12-06 at 18:44 +0100, Andries van der Leij wrote:
From: Andries van der Leij Sent: Wednesday, December 06, 2006 5:59 PM To: 'freesurfer@nmr.mgh.harvard.edu' Subject: Freesurfer and Grid computing
Dear Freesurfer community,
I'm a PHD student at the university of Amsterdam and I'm currently investigating the possibilities to streamline our MRI data processing stream. Next summer we'll obtain a research-only scanner. I'm trying to push the group to also invest in computing power and am currently investigating the applications that researchers will most probably use.
I came across a project of the group of Cohen at UCLA. They have configured a Apple (unix) grid and have proposed a more or less standard setup specially designed for MRI analyses:
http://airto.bmap.ucla.edu/mt- static/NICluster/archives/2005/06/welcome.html
It is my understanding that one of the members has rewritten the FSL code which allow distributed parallel processing in a Grid. See the benchmarks here:
http://airto.bmap.ucla.edu/bmcweb/bmc_bios/MarkCohen/Apple/Benchmarks.htm
My question is fairly simple: Are similar steps taken in the Freesurfer community? I have no experience with this app myself, but it is my understanding that Freesurfer consumes a lot of resources.
Thank you very much in advance,
Andries van der Leij
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer