New subject: reprocessing on FS4.0 takes much longer for some scans

12 Nov 2007

      I'm sure Bruce has much more experience on the subject than me. Also I'm not part of Martinos development team.  I'll just add my personal experience:
1) Your running times seens very high. I never saw 72 hours in my studies. Notice that Itanium (and Itanium 2) cpu are far distinct than x86 and x86-64, and even running correctly x86 code its performance in fpu falls in my experience to 50%. Since FS code is not compiled to Itanium arch I think the stated above can explain the problem.
2) Regarding memory my experience is that 1GB usually does fine if you have swap space.
3) about 3.0.5 and 4.0.1 differences in running time my data is: 22 hrs vs 31 hrs average. It depends on the "extended" topological fix 4.0.1 does.
It would be nice to know your cluster configuration and a sample of your data to give you a better advice.
best regards,
Pedro Paulo Oliveira Jr.
-- mens. original --
Assunto:	Re: [Freesurfer] reprocessing on FS4.0 takes much longer for some scans
De:	Bruce Fischl fischl@nmr.mgh.harvard.edu
Data:		11/11/2007 21:42
Hi Nicolas,
1. No, sorry, the amount of RAM is sometimes dependent on the 
individual anatomy, and in any case can't be predefine.
2. Not sure about the itaniums. We have no real experience. 72 hours does 
sound pretty long. That's I think about what it used to take on our old 
athlons. Can you extend the time limit?
As for the random stopping of recon-all, we have seen that sometimes as 
well, and are trying to track it down. It seems pretty mysterious, as a 
binary will exit with a nonzero exit code according to the shell even 
though the last printf in the code has been executed and the next statement 
is an exit(0).
Bruce
On 
Mon, 12 Nov 2007, Nicolas Cherbuin wrote:
...
Hi,
At the beginning of the year I processed 400 scans on a linux cluster. It had a reported ram limit of 1 Gig but it could cope with the slight excess of FS3 and a processing time limit of 48 hours (which was fine for 99% of the scans). Most scans went through autorecon-all without problem.
I am now trying to reprocess the scans with the new version but I am running into a number of problems. On the same linux cluser processing the same scans, some scans (~30%) run without problem. The rest either fail because they exceed the memory limit or because they take much more than 48 hours (the jobs are being killed and the logs report only the left and sometimes part of the right hemisphere being processed).
Since the documentation makes clear that FS works best with 2 Gig of ram, I have switched to an Itinium cluster with 2 Gig ram limit and 48 hours processing time limit. When I compare the logs of the same scans processed on both systems the Itinium cluster seems to take longer and although I am still running tests it appears that for at least some scans autoreconall might take 70+ hours.
Here are my questions:

On the linux cluster can I tell freesurfer not to exceed a certain ram allocation? and if yes how?

Do the problems I have on the Itinium cluster suggest that FS is badly configured on this system? and if yes where should we look? (I don't have access to this system's configuration and I have to feedback to the system managers to fix eventual problems.)

Thank you very much for your help and for sharing these great tools with us.
Nic

Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer