External Email - Use Caution
I am having trouble running the new recon-all-clinical module. I input 5mm
T1 axial DICOM images and it fails in SynthSeg with what appears to be an
out of memory error. I am running FreeSurfer 7.4.0 in WSL with Ubuntu 22
on a computer with 16 Gb RAM. Can anyone provide any thoughts on how to
correct this? Thanks in advance!
Here is the command:
recon-all-clinical.sh ./T1AX5MM/119 fsclin 1
here is the error output:
SynthSeg-robust 2.0
using CPU, hiding all CUDA_VISIBLE_DEVICES
using 1 thread
predicting 1/1
the following problem occured with image
/mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz :
Traceback (most recent call last):
File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315,
in predict
post_patch_segmentation, post_patch_parcellation, qc_score =
net.predict([image, shape_input])
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py",
line 52, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph
execution error:
Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most recent
call last):
File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
2642, in <module>
main()
File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
162, in main
predict(
File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
315, in predict
post_patch_segmentation, post_patch_parcellation, qc_score =
net.predict([image, shape_input])
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2382, in predict
tmp_batch_outputs = self.predict_function(iterator)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2169, in predict_function
return step_function(self, iterator)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2155, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2143, in run_step
outputs = model.predict_step(data)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2111, in predict_step
return self(x, training=False)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 558, in __call__
return super().__call__(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
outputs = call_fn(inputs, *args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py",
line 512, in call
return self._run_internal_graph(inputs, training=training, mask=mask)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py",
line 669, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
outputs = call_fn(inputs, *args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py",
line 196, in call
return self._merge_function(inputs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py",
line 134, in _merge_function
return backend.concatenate(inputs, axis=self.axis)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py",
line 3581, in concatenate
return tf.concat([to_dense(x) for x in tensors], axis)
Node: 'model_5/unet_parc_merge_8/concat'
OOM when allocating tensor with shape[1,256,256,160,72] and type float on
/job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu
[[{{node model_5/unet_parc_merge_8/concat}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add
report_tensor_allocations_upon_oom to RunOptions for current allocation
info. This isn't available when running in Eager mode.
[Op:__inference_predict_function_10473]
resuming program execution