recon-all-clinical out of memory error - Freesurfer

13 Jul 2023


      External Email - Use Caution
I am having trouble running the new recon-all-clinical module.  I input 5mm
T1 axial DICOM images and it fails in SynthSeg with what appears to be an
out of memory error.  I am running FreeSurfer 7.4.0 in WSL with Ubuntu 22
on a computer with 16 Gb RAM.   Can anyone provide any thoughts on how to
correct this?  Thanks in advance!
Here is the command:
recon-all-clinical.sh ./T1AX5MM/119 fsclin 1
here is the error output:
SynthSeg-robust 2.0
using CPU, hiding all CUDA_VISIBLE_DEVICES
using 1 thread
predicting 1/1
the following problem occured with image
/mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz :
Traceback (most recent call last):
  File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315,
in predict
    post_patch_segmentation, post_patch_parcellation, qc_score =
net.predict([image, shape_input])
  File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py",
line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph
execution error:
Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most recent
call last):
    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
2642, in <module>
      main()
    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
162, in main
      predict(
    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
315, in predict
      post_patch_segmentation, post_patch_parcellation, qc_score =
net.predict([image, shape_input])
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
      return fn(*args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2382, in predict
      tmp_batch_outputs = self.predict_function(iterator)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2169, in predict_function
      return step_function(self, iterator)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2155, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2143, in run_step
      outputs = model.predict_step(data)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2111, in predict_step
      return self(x, training=False)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
      return fn(*args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 558, in __call__
      return super().__call__(*args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
      return fn(*args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
      return fn(*args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py",
line 512, in call
      return self._run_internal_graph(inputs, training=training, mask=mask)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py",
line 669, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
      return fn(*args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
      return fn(*args, **kwargs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py",
line 196, in call
      return self._merge_function(inputs)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py",
line 134, in _merge_function
      return backend.concatenate(inputs, axis=self.axis)
    File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py",
line 3581, in concatenate
      return tf.concat([to_dense(x) for x in tensors], axis)
Node: 'model_5/unet_parc_merge_8/concat'
OOM when allocating tensor with shape[1,256,256,160,72] and type float on
/job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu
         [[{{node model_5/unet_parc_merge_8/concat}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add
report_tensor_allocations_upon_oom to RunOptions for current allocation
info. This isn't available when running in Eager mode.
 [Op:__inference_predict_function_10473]
resuming program execution