External Email - Use Caution

 Thanks for the updates.  Not that it will likely help with the memory underrun, but it should be straightforward to update Freesurfer from 7.2.0 to 7.4.1 in the VM from the command line. You can use the Firefox browser in the VM to download the 7.4.1 installer with the link,https://secure-web.cisco.com/1NY_o7fEYkraJG0kPriDNOoRkAxLu_PnGwgYcRK1s8senK4... In a terminal window, remove the existing freesurfer 7.2.0 installation,$ sudo apt remove freesurfer Then go to where the 7.4.1 installer was downloaded and run,$ sudo apt install ./freesurfer_ubuntu18-7.4.1_amd64.deb There was likely a soft link to the previous freesurfer distribution in developer's home directory, so to update that, $ cd $HOME$ rm -f  freesurfer$ ln -s  /usr/local/freesurfer/7.4.1  freesurfer$ ls freesurfer<output deleted) .... check you have 7.4.1 ... $ cat  freesurfer/build-stamp.txtfreesurfer-linux-ubuntu18_x86_64-7.4.1-20230614-7eb8460

- R. On Aug 19, 2023, at 08:00, Matthew Lynch <matthewl077@gmail.com> wrote:        External Email - Use Caution        Thank you fsbuild for the suggestions.  Unfortunately it still does not work.  I did not try the virtual machine, because I could only find version 7.2.0 ova file on the freesurfer website.  It appears the 7.4.0 version does not yet have a virtual machine?  I am trying to use recon-all-clinical, which did not appear until 7.4.0.  The standard recon-all works fine on 1mm scans on my 16 Gb machine under WSL.  But I am trying to explore use of recon-all-clinical.It appeared as if WSL was not using any of the available swap space, so I tried a few different swap and memory configurations in .wslconfig.  Then I tried to create a cgroup that would limit physical memory and force freesurfer to use swap.  This did cause freesurfer to use swap space, but it still crashed with out of memory error when the combined swap and physical memory got to about 23 Gb (out of a total of 100 Gb virtual memory).I then created a total Linux version using Porteus Linux live USB and unpacking the freesurfer 7.4.0 tar.gz and finding a few minor dependencies.  This is a fairly lightweight Linux distro that runs from USB so I didn't have to create a dual boot..  The base memory usage was about 1.3 Gb before running recon-all-clinical.  I also activated a 16 Gb swap file with swapon.  It didn't work either, even in a pure Linux environment.  At maximum memory usage, it used only about 7-9% of the 16 Gb swap space and I think it also crashed at the same place around 21 Gb total memory usage with an OOM error.So it appears to be complaining of out of memory even when there is ample memory available in the form of swap space.  I have tried various options for vm.overcommit and vm.swappiness.This seems to occur only for a few selected scans.  For most of them it works.  There is a long error chain but this seems to be the relevant line:OOM when allocating tensor with shape[1,288,288,192,72] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu[[{{node model_5/unet_parc_merge_8/concat}}]]It fails there every time. I bit the bullet and bought an additional 16 Gb physical RAM for a total of 32 Gb.  I'll let you know if that works.On Sun, Aug 13, 2023 at 10:11 PM fsbuild <fsbuild@contbay.com> wrote:You might be better off with swapping inside the Linux VM instead of on the windows host.  Or I don’t see anything preventing the Windows host from trying to swap out the entire VM process - which I would not expect to be efficient.  You only need the application running *inside*the*Linux*VM* to be swapped out if necessary (using the Linux OS to do that).This should be straightforward to setup using VirtualBox to run the Ubuntu Linux VM we provide with Freesurfer already installed, i.e., download fromMailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/1oFOJz1tXGspfsKNscw0jmPq8nuxBXOxbqTB1n-bwDnNM5m... the screenshot below, the FreeSurfer VM starts with 4G allocated to it.  So correspondingly, inside the VM the “free” command reports there is ~4G available of which only 1G is being used since nothing consequential is running.  If you stopped the VM, increased the allocated memory in the VirtualBox application, and re-started the VM, you would see the total available memory scale up accordingly.For available swap space, you would look at the boot image on /dev/sda1.  That image is made on purpose as small as possible so it can grow and/or space can otherwise be used.  The “df” commands shows that out of the box, the VM takes up 22G of the max allowed size of 170G.  That leaves potentially 170-22=148G to be used for other purposes.  You certainly would not want to allocate all that for swap space, but there is space to use for a good chunk of swap space apart from space you will consume doing work that adds files into the VM.   Here is 1 of many articles about setting up swapping in VirtualBox (which we do not setup by default),MailScanner has detected a possible fraud attempt from "secure-web.cisco..com" claiming to be https://secure-web.cisco.com/1Zqb8y3NbSCxfL4B31cjKDD_lN8KL6yjtwEypwgDMnjLewl... efficiency of swapping will be limited by your system, e.g.., how fast the disks are, their bus interface, etc.  There’s no guarantee even the most efficient swap configuration won’t cause the machine to slow down to a crawl depending upon what you are processing.  It’s not a substitute for having RAM to allocate to the VM.   We have discussed increasing the published memory system requirements for Freesurfer.- R.On Aug 13, 2023, at 06:39, Matthew Lynch <matthewl077@gmail.com> wrote:        External Email - Use Caution        The method I mentioned before does not always work.  It worked for all but about 3 scans out of approximately 30.  There are some scans for which the "committed memory" caps at about 23 Gb in the task manager and then either the out of memory error occurs or the WSL session is killed.  Increasing the .wslconfig swap and memory beyond 32 Gb didn't work.  Increasing the Windows pagefile size to 8 Gb didn't work.  The output of free -h during the synthseg run was 0Gb used, 40Gb free, which seems to suggest it is not using the swap.  I suspect this is a matter of somehow getting WSL to properly use large amounts of swap space.On Thu, Aug 10, 2023 at 5:52 PM fsbuild <fsbuild@contbay.com> wrote: Hello Matthew,Thanks for this info.If it is not too much work, just wondering if you could make a rough estimate or compute an average for the *total*memory* in use = (size of physical RAM in use by synthseg + size of all swap files resident on disk used by synthseg).- R.On Aug 10, 2023, at 16:08, Matthew Lynch <matthewl077@gmail.com> wrote:        External Email - Use Caution        Here is some additional follow up.  I was able to get recon-all-clinical working in Ubuntu in WSL on a 16 Gb machine by increasing the swap and memory allowed in WSL..  This is done through the .wslconfig file which is located in the %USERPROFILE% folder (e.g. C:\Users\username).  I had to specify a swap size of 32 Gb and also specify that WSL is allowed to use 32 Gb (even though the system only has 16 Gb -- it will use swap space).Here is the .wslconfig file I used:-----[wsl2]# Limits VM memory to use no more than x GB, this can be set as whole numbers using GB or MB# This must be set to 32 Gb or so to accomodate recon-all-clinical# If you set a 32 Gb swap but not 32 Gb memory allowance, WSL will stop at the memory limit and not use swapmemory=32GB # Sets amount of swap storage space to 8GB, default is 25% of available RAMswap=32GB# Sets swapfile path location, default is %USERPROFILE%\AppData\Local\Temp\swap.vhdx# I believe the double backslash is necessaryswapfile=P:\Users\username\WSL\wsl-swap.vhdx------For information on the .wslconfig file, see: MailScanner has detected a possible fraud attempt from "secure-web..cisco.com" claiming to be https://secure-web.cisco.com/19DCKWC0AnWOY-AFXDoqJLBE2tTUpB9JfKhrDVvmJ9NTI22... Sat, Jul 29, 2023 at 6:27 AM Matthew Lynch <matthewl077@gmail.com> wrote:I was able to get recon-all-clinical working with 16 Gb computer.I thought the problem might be Windows hogging memory, so I created a live persistent USB install of lubuntu, using lubuntu 22 as a low resource distro in order to minimize memory usage, and installed freesurfer 7.4.0.  Even with initial memory usage on 1Gb on lubuntu, synthseg still failed with out of memory error with 8 Gb swap file.  I watched the swap memory fill up and then process was killed.I added 16Gb swap using the following instructions, for a total swap space of 24 Gb + 16 Gb memory:MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/1kRX3xVbTiZ9JT54ixRH2s2lX2HsDjAcrgZSnsH701UOmFP... worked.On Thu, Jul 13, 2023 at 3:28 PM Matthew Lynch <matthewl077@gmail.com> wrote:I do not have access to a computer with more RAM.  I can run recon-all with 1mm T1 images without any problems.  How much RAM would recon-all-clinical be expected to need?  Does it require less RAM if it uses the GPU instead of CPU?  On Thu, Jul 13, 2023 at 1:58 PM Fischl, Bruce R.,PHD <BFISCHL@mgh.harvard.edu> wrote:        External Email - Use Caution        Can you try it on a computer with more ram?  From: freesurfer-bounces@nmr..mgh.harvard.edu <freesurfer-bounces@nmr.mgh.harvard.edu> On Behalf Of Matthew LynchSent: Thursday, July 13, 2023 1:55 PMTo: freesurfer@nmr.mgh.harvard.eduSubject: [Freesurfer] recon-all-clinical out of memory error         External Email - Use Caution        I am having trouble running the new recon-all-clinical module.  I input 5mm T1 axial DICOM images and it fails in SynthSeg with what appears to be an out of memory error.  I am running FreeSurfer 7.4.0 in WSL with Ubuntu 22 on a computer with 16 Gb RAM.   Can anyone provide any thoughts on how to correct this?  Thanks in advance!  Here is the command: recon-all-clinical.sh ./T1AX5MM/119 fsclin 1  here is the error output: SynthSeg-robust 2.0using CPU, hiding all CUDA_VISIBLE_DEVICESusing 1 threadpredicting 1/1the following problem occured with image /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz :Traceback (most recent call last):  File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict    post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input])  File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler    raise e.with_traceback(filtered_tb) from None  File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most recent call last):    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 2642, in <module>      main()    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 162, in main      predict(    File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict      post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input])    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler      return fn(*args, **kwargs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2382, in predict      tmp_batch_outputs = self.predict_function(iterator)    File "/usr/local/freesurfer/7.4.0/python/lib/python3..8/site-packages/keras/engine/training.py", line 2169, in predict_function      return step_function(self, iterator)    File "/usr/local/freesurfer/7.4..0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2155, in step_function      outputs = model.distribute_strategy..run(run_step, args=(data,))    File "/usr/local/freesurfer/7.4..0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2143, in run_step      outputs = model.predict_step(data)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2111, in predict_step      return self(x, training=False)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler      return fn(*args, **kwargs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 558, in __call__      return super().__call__(*args, **kwargs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler      return fn(*args, **kwargs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__      outputs = call_fn(inputs, *args, **kwargs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler      return fn(*args, **kwargs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3..8/site-packages/keras/engine/functional..py", line 512, in call      return self._run_internal_graph(inputs, training=training, mask=mask)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph      outputs = node.layer(*args, **kwargs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler      return fn(*args, **kwargs)    File "/usr/local/freesurfer/7.4..0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__      outputs = call_fn(inputs, *args, **kwargs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler      return fn(*args, **kwargs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py", line 196, in call      return self._merge_function(inputs)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py", line 134, in _merge_function      return backend.concatenate(inputs, axis=self..axis)    File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py", line 3581, in concatenate      return tf.concat([to_dense(x) for x in tensors], axis)Node: 'model_5/unet_parc_merge_8/concat'OOM when allocating tensor with shape[1,256,256,160,72] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu         [[{{node model_5/unet_parc_merge_8/concat}}]]Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.. This isn't available when running in Eager mode. [Op:__inference_predict_function_10473]resuming program execution _______________________________________________Freesurfer mailing listFreesurfer@nmr.mgh.harvard.eduMailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/19kJTXmqHxdv9FUBUNscNtwPP-oousx8-y3qBfrFrP6SiU3... information in this e-mail is intended only for the person to whom it is addressed.  If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/1d3qoA8jkiEujxcb_3JFzMd8QkU88j8PgCllABvBDL3zU7w... has detected a possible fraud attempt from "secure-web..cisco.com" claiming to be https://secure-web.cisco.com/1vjDVAlnQcZSScJlFw_X3CoNDMlcbgzI5NuM-LQqQrYPqEE...; ._______________________________________________Freesurfer mailing listFreesurfer@nmr.mgh.harvard.eduMailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to behttps://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer________________... mailing listFreesurfer@nmr.mgh.harvard.eduMailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/1BbIXKRtCoHVre8VbJJhEr8xJ7iyvoJdAeHrQyG3d6rpZ52... mailing listFreesurfer@nmr.mgh.harvard.eduhttps://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer