External Email - Use Caution        

I do not have access to a computer with more RAM.  I can run recon-all with 1mm T1 images without any problems.  How much RAM would recon-all-clinical be expected to need?  Does it require less RAM if it uses the GPU instead of CPU?  

On Thu, Jul 13, 2023 at 1:58 PM Fischl, Bruce R.,PHD <BFISCHL@mgh.harvard.edu> wrote:

        External Email - Use Caution        

Can you try it on a computer with more ram?

 

From: freesurfer-bounces@nmr.mgh.harvard.edu <freesurfer-bounces@nmr.mgh.harvard.edu> On Behalf Of Matthew Lynch
Sent: Thursday, July 13, 2023 1:55 PM
To: freesurfer@nmr.mgh.harvard.edu
Subject: [Freesurfer] recon-all-clinical out of memory error

 

        External Email - Use Caution        

I am having trouble running the new recon-all-clinical module.  I input 5mm T1 axial DICOM images and it fails in SynthSeg with what appears to be an out of memory error.  I am running FreeSurfer 7.4.0 in WSL with Ubuntu 22 on a computer with 16 Gb RAM.   Can anyone provide any thoughts on how to correct this?  Thanks in advance!

 

 Here is the command:

 

recon-all-clinical.sh ./T1AX5MM/119 fsclin 1

 

 

here is the error output:

 

SynthSeg-robust 2.0
using CPU, hiding all CUDA_VISIBLE_DEVICES
using 1 thread
predicting 1/1

the following problem occured with image /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz :
Traceback (most recent call last):
  File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict
    post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input])
  File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:

Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most recent call last):
    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 2642, in <module>
      main()
    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 162, in main
      predict(
    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict
      post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input])
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2382, in predict
      tmp_batch_outputs = self.predict_function(iterator)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2169, in predict_function
      return step_function(self, iterator)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2155, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2143, in run_step
      outputs = model.predict_step(data)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2111, in predict_step
      return self(x, training=False)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 558, in __call__
      return super().__call__(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 512, in call
      return self._run_internal_graph(inputs, training=training, mask=mask)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py", line 196, in call
      return self._merge_function(inputs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py", line 134, in _merge_function
      return backend.concatenate(inputs, axis=self.axis)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py", line 3581, in concatenate
      return tf.concat([to_dense(x) for x in tensors], axis)
Node: 'model_5/unet_parc_merge_8/concat'
OOM when allocating tensor with shape[1,256,256,160,72] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu
         [[{{node model_5/unet_parc_merge_8/concat}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
 [Op:__inference_predict_function_10473]

resuming program execution

 

_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed.  If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://www.massgeneralbrigham.org/complianceline <MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://www.massgeneralbrigham.org/complianceline> .