Dear Freesurfers!
I am a new user of freesurfer and I intend to install it on a HPC system. There are two different systems available (see end of message). Could you please give me a hint which one is best suited wrt hardware and software.
In order to use these systems one also has to state some estimations of the needed resources. In detail, I have to estimate:
* CPU time * maximal number of parallel used CPUs * size of used RAM * size of permanent HDD usage * size of temporary HDD usage
Could you share some experience how many resources are usually used for the segmentation (recon-all) of a single subject?
Thank you & best wishes from Dresden, Daniel
--- 1.SGI Altix 4700 (Mars)
* For highly parallel and memory-intensive applications * 2048 cores Intel Itanium II Montecito 1.6 GHz * 6,5 TB memory * One partition for interactive operation with 128 cores * 68 TB SAN disk storage * 1 PB SAN tape archive * SuSE Linux Enterprise Server 10, SGI ProPack 5, batch system LSF * 13,1 TFlop/s peak performance
2. Linux Networx Evolocity II PC-Farm (Deimos)
* For capacity computing * 2576 cores AMD Opteron 2,6 GHz * 5,4 TB memory * 724 nodes with 1, 2, or 4 dual-core CPUs each * 68 TB SAN disk storage * SuSE Linux Enterprise Server 10, batch system LSF * 13,9 TFlop/s peak performance
For software specification please refer to: http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/hpc/hochleist...
the second option is best because it uses the x86 processor (amd opteron), which is our common distribution. the first option uses the Itanium processor which we dont actively support because we no longer have access to a build machine.
the recon-all stream is not parallelized so you wouldnt benefit from multi-cores or any special parallelization hardware. runtime on the amd cluster would be about 25 hours per subject, and consume a peak of about 2.7GB of ram. each subject consumes around 400MB of hard disk space (budget 500MB per subject). negligible temporary hard disk space is used.
n.
On Wed, 2010-09-22 at 19:17 +0200, daniel geisler wrote:
Dear Freesurfers!
I am a new user of freesurfer and I intend to install it on a HPC system. There are two different systems available (see end of message). Could you please give me a hint which one is best suited wrt hardware and software.
In order to use these systems one also has to state some estimations of the needed resources. In detail, I have to estimate:
- CPU time
- maximal number of parallel used CPUs
- size of used RAM
- size of permanent HDD usage
- size of temporary HDD usage
Could you share some experience how many resources are usually used for the segmentation (recon-all) of a single subject?
Thank you & best wishes from Dresden, Daniel
1.SGI Altix 4700 (Mars)
- For highly parallel and memory-intensive applications
- 2048 cores Intel Itanium II Montecito 1.6 GHz
- 6,5 TB memory
- One partition for interactive operation with 128 cores
- 68 TB SAN disk storage
- 1 PB SAN tape archive
- SuSE Linux Enterprise Server 10, SGI ProPack 5, batch system LSF
- 13,1 TFlop/s peak performance
- Linux Networx Evolocity II PC-Farm (Deimos)
- For capacity computing
- 2576 cores AMD Opteron 2,6 GHz
- 5,4 TB memory
- 724 nodes with 1, 2, or 4 dual-core CPUs each
- 68 TB SAN disk storage
- SuSE Linux Enterprise Server 10, batch system LSF
- 13,9 TFlop/s peak performance
For software specification please refer to: http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/hpc/hochleist... _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Is the 2.7GB RAM estimate new for v5? For v3 and v4, 2GB seems to have been sufficient (using a hard limit on a cluster). We haven't switched to v5 yet (waiting for 5.1 or 5.2), so would like to know what to expect when we do.
From: nicks@nmr.mgh.harvard.edu To: daniel.geisler@gmail.com Date: Wed, 22 Sep 2010 16:50:59 -0400 CC: freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] System requirements
the second option is best because it uses the x86 processor (amd opteron), which is our common distribution. the first option uses the Itanium processor which we dont actively support because we no longer have access to a build machine.
the recon-all stream is not parallelized so you wouldnt benefit from multi-cores or any special parallelization hardware. runtime on the amd cluster would be about 25 hours per subject, and consume a peak of about 2.7GB of ram. each subject consumes around 400MB of hard disk space (budget 500MB per subject). negligible temporary hard disk space is used.
n.
On Wed, 2010-09-22 at 19:17 +0200, daniel geisler wrote:
Dear Freesurfers!
I am a new user of freesurfer and I intend to install it on a HPC system. There are two different systems available (see end of message). Could you please give me a hint which one is best suited wrt hardware and software.
In order to use these systems one also has to state some estimations of the needed resources. In detail, I have to estimate:
- CPU time
- maximal number of parallel used CPUs
- size of used RAM
- size of permanent HDD usage
- size of temporary HDD usage
Could you share some experience how many resources are usually used for the segmentation (recon-all) of a single subject?
Thank you & best wishes from Dresden, Daniel
1.SGI Altix 4700 (Mars)
- For highly parallel and memory-intensive applications
- 2048 cores Intel Itanium II Montecito 1.6 GHz
- 6,5 TB memory
- One partition for interactive operation with 128 cores
- 68 TB SAN disk storage
- 1 PB SAN tape archive
- SuSE Linux Enterprise Server 10, SGI ProPack 5, batch system LSF
- 13,1 TFlop/s peak performance
- Linux Networx Evolocity II PC-Farm (Deimos)
- For capacity computing
- 2576 cores AMD Opteron 2,6 GHz
- 5,4 TB memory
- 724 nodes with 1, 2, or 4 dual-core CPUs each
- 68 TB SAN disk storage
- SuSE Linux Enterprise Server 10, batch system LSF
- 13,9 TFlop/s peak performance
For software specification please refer to: http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/hpc/hochleist... _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
the 2.7GB ram estimate is based on actual usage by Mehul Sampat on a cluster, from a posting on the list yesterday by him. it's difficult to give a clear estimate because it depends on the subject. a poorly scanned subject, or one with lots of lesions, generally creates problems for the algorithms. the algorithms have not changed much in v5 from v4, so i wouldnt expect a greater ram need.
n.
On Thu, 2010-09-23 at 08:53 -0700, Don Hagler wrote:
Is the 2.7GB RAM estimate new for v5? For v3 and v4, 2GB seems to have been sufficient (using a hard limit on a cluster). We haven't switched to v5 yet (waiting for 5.1 or 5.2), so would like to know what to expect when we do.
From: nicks@nmr.mgh.harvard.edu To: daniel.geisler@gmail.com Date: Wed, 22 Sep 2010 16:50:59 -0400 CC: freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] System requirements
the second option is best because it uses the x86 processor (amd opteron), which is our common distribution. the first option uses
the
Itanium processor which we dont actively support because we no
longer
have access to a build machine.
the recon-all stream is not parallelized so you wouldnt benefit from multi-cores or any special parallelization hardware. runtime on the
amd
cluster would be about 25 hours per subject, and consume a peak of
about
2.7GB of ram. each subject consumes around 400MB of hard disk space (budget 500MB per subject). negligible temporary hard disk space is used.
n.
On Wed, 2010-09-22 at 19:17 +0200, daniel geisler wrote:
Dear Freesurfers!
I am a new user of freesurfer and I intend to install it on a HPC system. There are two different systems available (see end of message). Could you please give me a hint which one is best suited
wrt
hardware and software.
In order to use these systems one also has to state some
estimations
of the needed resources. In detail, I have to estimate:
- CPU time
- maximal number of parallel used CPUs
- size of used RAM
- size of permanent HDD usage
- size of temporary HDD usage
Could you share some experience how many resources are usually
used
for the segmentation (recon-all) of a single subject?
Thank you & best wishes from Dresden, Daniel
1.SGI Altix 4700 (Mars)
- For highly parallel and memory-intensive applications
- 2048 cores Intel Itanium II Montecito 1.6 GHz
- 6,5 TB memory
- One partition for interactive operation with 128 cores
- 68 TB SAN disk storage
- 1 PB SAN tape archive
- SuSE Linux Enterprise Server 10, SGI ProPack 5, batch system LSF
- 13,1 TFlop/s peak performance
- Linux Networx Evolocity II PC-Farm (Deimos)
- For capacity computing
- 2576 cores AMD Opteron 2,6 GHz
- 5,4 TB memory
- 724 nodes with 1, 2, or 4 dual-core CPUs each
- 68 TB SAN disk storage
- SuSE Linux Enterprise Server 10, batch system LSF
- 13,9 TFlop/s peak performance
For software specification please refer to:
http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/hpc/hochleist...
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to
whom it is
addressed. If you believe this e-mail was sent to you in error and
the e-mail
contains patient information, please contact the Partners Compliance
HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to
you in error
but does not contain patient information, please contact the sender
and properly
dispose of the e-mail.
freesurfer@nmr.mgh.harvard.edu
-
daniel geisler -
Don Hagler -
Nick Schmansky