Hi Nick,
Thanks for the reply. I will look into the memory limit issue some more (we recently instituted per job memory limits), but I have done a test run with a 16GB memory limit (running one job per node).
Meanwhile, thanks for the novtk suggestion. Our fallback option is to just disable the ribbon generation. I don't think we ever use them for anything.
Don
Subject: Re: [Freesurfer] vtkMultiThreader error From: nicks@nmr.mgh.harvard.edu To: martin.kavec@gmail.com; dhaglerjr@hotmail.com Date: Mon, 8 Jun 2009 11:15:51 -0400 CC: freesurfer@nmr.mgh.harvard.edu
Don and Martin,
I dont have a good solution for you to this problem. The error code indicates an 'out-of-memory' error returned by pthread_create, and I do know the mris_vol_mask uses a lot of memory (at least 1GB).
I have not been able to find any clues to the source of this problem on the web either. The only curious thing to it is that in the source code, that section is conditionally compiled to ignore the return code if 'VTK_HP_PTHREADS' is defined, which implies some sort of odd behavior on some special HP platform build.
mris_vol_mask has been a real thorn in the side of the recon-all stream, but its not easy to replace because it does a subjectively better job of finding the voxels which intersect the pial and white surfaces. For comparison, or even to solve your problem, you can use mris_vol_mask_novtk to create the ribbon.mgz files. The output is not the same as that produced by mris_vol_mask though. Perhaps you can run it, and compare the results side-by-side and give us your opinion on which is the better ribbon (as there is no objective measure on what a 'correct' ribbon.mgz should like like, ie, should it include voxels that intersect more than half way through a surface, or not at all?).
Nick
On Mon, 2009-06-08 at 09:16 +0200, Martin Kavec wrote:
Hi Don,
I am in a similar situation. I am on a IBM BladeCenter cluster (though with one blade only) with 2 Quad core Intels, and I experience intermittent crashes of mri_volmask as well. Similarly as you, when I run the mri_volmask on the crashed subject on a desktop, this passes perfectly fine and sometimes also on the cluster as well.
I am wonder whether how do you obtain the debug information about the cause of the crash (being in the vtkMultiThreader)?
I also wonder, whether the crashes could be due to different compilation environments of the VTK libraries used in teh FS distribution (centos) and your redhat. I went on the bug-report site of VTK and there really isn't anythink major reported on this class.
In my experience the mri_volmask runs fine for quite a while and then crashes.
From the debug messages you show it would thus could be that the process may
hit some limit of the threads which could be allocated to a process.
What do you think?
Martin
On Saturday 06 June 2009 04:19:51 Don Hagler wrote:
When we run v430 on our cluster, running rocks redhat 4.3, we get the error message below when mris_volmask is run. From a google search, I see that someone had the same error message back in Oct 2007. Was a solution found for this? It runs fine on our desktop machines as well as when we log in directly to a node and run the job interactively.
mris_volmask --label_left_white 2 --label_left_ribbon 3 --label_right_white 41 --label_right_ribbon 42 --save_ribbon --save_distance FREESURFERRECON_NTNU_PRE_095v2_20071126.173245.968000_1
SUBJECTS_DIR is /home/dhagler/data_test ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx, line 372 vtkMultiThreader (0x1306ec80): Unable to create a thread. pthread_create() returned 12
ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx, line 372 vtkMultiThreader (0x1306ec80): Unable to create a thread. pthread_create() returned 12
ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx, line 372 vtkMultiThreader (0x1306ec80): Unable to create a thread. pthread_create() returned 12
ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx, line 372 vtkMultiThreader (0x1306ec80): Unable to create a thread. pthread_create() returned 12
ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx, line 372 vtkMultiThreader (0x1306ec80): Unable to create a thread. pthread_create() returned 12
ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx, line 372 vtkMultiThreader (0x1306ec80): Unable to create a thread. pthread_create() returned 12
ERROR: In /usr/pubsw/packages/vtk/5.2.0/src/Common/vtkMultiThreader.cxx, line 372 vtkMultiThreader (0x1306ec80): Unable to create a thread. pthread_create() returned 12
Segmentation fault Linux compute-0-12.local 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:14:47 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
recon-all exited with ERRORS at Fri Jun 5 18:58:09 PDT 2009
Hotmail® has ever-growing storage! Don’t worry about storage limits. http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutori al_Storage_062009
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
_________________________________________________________________ Insert movie times and more without leaving Hotmail®. http://windowslive.com/Tutorial/Hotmail/QuickAdd?ocid=TXT_TAGLM_WL_HM_Tutori...