External Email - Use Caution        

Here is some additional follow up.  I was able to get recon-all-clinical working in Ubuntu in WSL on a 16 Gb machine by increasing the swap and memory allowed in WSL.  This is done through the .wslconfig file which is located in the %USERPROFILE% folder (e.g. C:\Users\username\).  I had to specify a swap size of 32 Gb and also specify that WSL is allowed to use 32 Gb (even though the system only has 16 Gb -- it will use swap space).

Here is the .wslconfig file I used:
-----
[wsl2]

# Limits VM memory to use no more than x GB, this can be set as whole numbers using GB or MB
# This must be set to 32 Gb or so to accomodate recon-all-clinical
# If you set a 32 Gb swap but not 32 Gb memory allowance, WSL will stop at the memory limit and not use swap
memory=32GB

# Sets amount of swap storage space to 8GB, default is 25% of available RAM
swap=32GB

# Sets swapfile path location, default is %USERPROFILE%\AppData\Local\Temp\swap.vhdx
# I believe the double backslash is necessary
swapfile=P:\\Users\\username\\WSL\\wsl-swap.vhdx
------

For information on the .wslconfig file, see: MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://learn.microsoft.com/en-us/windows/wsl/wsl-config


On Sat, Jul 29, 2023 at 6:27 AM Matthew Lynch <matthewl077@gmail.com> wrote:

I was able to get recon-all-clinical working with 16 Gb computer.

I thought the problem might be Windows hogging memory, so I created a live persistent USB install of lubuntu, using lubuntu 22 as a low resource distro in order to minimize memory usage, and installed freesurfer 7.4.0.  Even with initial memory usage on 1Gb on lubuntu, synthseg still failed with out of memory error with 8 Gb swap file.  I watched the swap memory fill up and then process was killed.

I added 16Gb swap using the following instructions, for a total swap space of 24 Gb + 16 Gb memory:

MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://askubuntu.com/questions/178712/how-to-increase-swap-space

This worked.





On Thu, Jul 13, 2023 at 3:28 PM Matthew Lynch <matthewl077@gmail.com> wrote:
I do not have access to a computer with more RAM.  I can run recon-all with 1mm T1 images without any problems.  How much RAM would recon-all-clinical be expected to need?  Does it require less RAM if it uses the GPU instead of CPU?  

On Thu, Jul 13, 2023 at 1:58 PM Fischl, Bruce R.,PHD <BFISCHL@mgh.harvard.edu> wrote:

        External Email - Use Caution        

Can you try it on a computer with more ram?

 

From: freesurfer-bounces@nmr.mgh.harvard.edu <freesurfer-bounces@nmr.mgh.harvard.edu> On Behalf Of Matthew Lynch
Sent: Thursday, July 13, 2023 1:55 PM
To: freesurfer@nmr.mgh.harvard.edu
Subject: [Freesurfer] recon-all-clinical out of memory error

 

        External Email - Use Caution        

I am having trouble running the new recon-all-clinical module.  I input 5mm T1 axial DICOM images and it fails in SynthSeg with what appears to be an out of memory error.  I am running FreeSurfer 7.4.0 in WSL with Ubuntu 22 on a computer with 16 Gb RAM.   Can anyone provide any thoughts on how to correct this?  Thanks in advance!

 

 Here is the command:

 

recon-all-clinical.sh ./T1AX5MM/119 fsclin 1

 

 

here is the error output:

 

SynthSeg-robust 2.0
using CPU, hiding all CUDA_VISIBLE_DEVICES
using 1 thread
predicting 1/1

the following problem occured with image /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz :
Traceback (most recent call last):
  File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict
    post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input])
  File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:

Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most recent call last):
    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 2642, in <module>
      main()
    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 162, in main
      predict(
    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict
      post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input])
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2382, in predict
      tmp_batch_outputs = self.predict_function(iterator)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2169, in predict_function
      return step_function(self, iterator)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2155, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2143, in run_step
      outputs = model.predict_step(data)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2111, in predict_step
      return self(x, training=False)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 558, in __call__
      return super().__call__(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 512, in call
      return self._run_internal_graph(inputs, training=training, mask=mask)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py", line 196, in call
      return self._merge_function(inputs)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py", line 134, in _merge_function
      return backend.concatenate(inputs, axis=self.axis)
    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py", line 3581, in concatenate
      return tf.concat([to_dense(x) for x in tensors], axis)
Node: 'model_5/unet_parc_merge_8/concat'
OOM when allocating tensor with shape[1,256,256,160,72] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu
         [[{{node model_5/unet_parc_merge_8/concat}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
 [Op:__inference_predict_function_10473]

resuming program execution

 

_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed.  If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://www.massgeneralbrigham.org/complianceline <MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://www.massgeneralbrigham.org/complianceline> .