External Email - Use Caution
I am having trouble running the new recon-all-clinical module. I input 5mm T1 axial DICOM images and it fails in SynthSeg with what appears to be an out of memory error. I am running FreeSurfer 7.4.0 in WSL with Ubuntu 22 on a computer with 16 Gb RAM. Can anyone provide any thoughts on how to correct this? Thanks in advance!
Here is the command:
recon-all-clinical.sh ./T1AX5MM/119 fsclin 1
here is the error output:
SynthSeg-robust 2.0 using CPU, hiding all CUDA_VISIBLE_DEVICES using 1 thread predicting 1/1
the following problem occured with image /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz : Traceback (most recent call last): File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input]) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler raise e.with_traceback(filtered_tb) from None File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:
Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most recent call last): File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 2642, in <module> main() File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 162, in main predict( File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input]) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2382, in predict tmp_batch_outputs = self.predict_function(iterator) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2169, in predict_function return step_function(self, iterator) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2155, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2143, in run_step outputs = model.predict_step(data) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2111, in predict_step return self(x, training=False) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 558, in __call__ return super().__call__(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 512, in call return self._run_internal_graph(inputs, training=training, mask=mask) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph outputs = node.layer(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py", line 196, in call return self._merge_function(inputs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py", line 134, in _merge_function return backend.concatenate(inputs, axis=self.axis) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py", line 3581, in concatenate return tf.concat([to_dense(x) for x in tensors], axis) Node: 'model_5/unet_parc_merge_8/concat' OOM when allocating tensor with shape[1,256,256,160,72] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu [[{{node model_5/unet_parc_merge_8/concat}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [Op:__inference_predict_function_10473]
resuming program execution