External Email - Use Caution
I do not have access to a computer with more RAM. I can run recon-all with 1mm T1 images without any problems. How much RAM would recon-all-clinical be expected to need? Does it require less RAM if it uses the GPU instead of CPU?
On Thu, Jul 13, 2023 at 1:58 PM Fischl, Bruce R.,PHD < BFISCHL@mgh.harvard.edu> wrote:
External Email - Use CautionCan you try it on a computer with more ram?
*From:* freesurfer-bounces@nmr.mgh.harvard.edu < freesurfer-bounces@nmr.mgh.harvard.edu> *On Behalf Of *Matthew Lynch *Sent:* Thursday, July 13, 2023 1:55 PM *To:* freesurfer@nmr.mgh.harvard.edu *Subject:* [Freesurfer] recon-all-clinical out of memory error
External Email - Use Caution *I am having trouble running the new recon-all-clinical module. I input 5mm T1 axial DICOM images and it fails in SynthSeg with what appears to be an out of memory error. I am running FreeSurfer 7.4.0 in WSL with Ubuntu 22 on a computer with 16 Gb RAM. Can anyone provide any thoughts on how to correct this? Thanks in advance!
Here is the command:
recon-all-clinical.sh ./T1AX5MM/119 fsclin 1
here is the error output:
SynthSeg-robust 2.0 using CPU, hiding all CUDA_VISIBLE_DEVICES using 1 thread predicting 1/1
the following problem occured with image /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz : Traceback (most recent call last): File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input]) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler raise e.with_traceback(filtered_tb) from None File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:
Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most recent call last): File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 2642, in <module> main() File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 162, in main predict( File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input]) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2382, in predict tmp_batch_outputs = self.predict_function(iterator) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2169, in predict_function return step_function(self, iterator) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2155, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2143, in run_step outputs = model.predict_step(data) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2111, in predict_step return self(x, training=False) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 558, in __call__ return super().__call__(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 512, in call return self._run_internal_graph(inputs, training=training, mask=mask) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph outputs = node.layer(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py", line 196, in call return self._merge_function(inputs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py", line 134, in _merge_function return backend.concatenate(inputs, axis=self.axis) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py", line 3581, in concatenate return tf.concat([to_dense(x) for x in tensors], axis) Node: 'model_5/unet_parc_merge_8/concat' OOM when allocating tensor with shape[1,256,256,160,72] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu [[{{node model_5/unet_parc_merge_8/concat}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [Op:__inference_predict_function_10473]
resuming program execution
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://secure-web.cisco.com/1cgiw0QpeuIZHhfIZEvHRuSB2y_kwffJua3EEuK4AZSxIkP... The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://secure-web.cisco.com/1OHxlgWD90YT7mlV-NqvsHcbQvZrB7oYQInUUJVdImP_oVX... < https://secure-web.cisco.com/1OHxlgWD90YT7mlV-NqvsHcbQvZrB7oYQInUUJVdImP_oVX... .