External Email - Use Caution        

Thank you for the suggestions. It appears now to be seeing the DNN libraries.  However, now I get:

OOM when allocating tensor with shape[1,256,256,160,24] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

My GPU has 4Gb RAM and the CPU has 32 Gb RAM.  Is this error occurring because of insufficient GPU RAM?  What are the requirements of the GPU for mri_synthseg?  Is there a workaround for this?


For those who may be having the same trouble configuring the GPU for FreeSurfer 7.4.0 in WSL, I installed an old Nvidia Game Ready Driver 522 (for my GPU) in Windows, which installed CUDA 11.8 support.   In WSL, I installed CUDA 11.8 toolkit WSL-Ubuntu version using instructions from Nvidia website and CuDNN 8.6.0.  I then had to ensure the CuDNN libraries were in the LD_LIBRARY_PATH using:

export LD_LIBRARY_PATH="/usr/lib/wsl/lib:/lib/x86_64-linux-gnu"

This seemed to work, except I received the above mentioned out of memory error.

On Tue, Sep 10, 2024 at 6:04 PM fsbuild <fsbuild@contbay.com> wrote:

        External Email - Use Caution        

 
The 7.4.0 release contains the python distribution all the python scripts (including mri_synthseg) were tested with (on CentOS 8, 7; Ubuntu 22, 20 18).  That python includes tensorflow 2.12.0 which is documented to support up thru Cuda 11.8.   If you look at all the cuda libraries you will see their major release numbers range from 8-11.  Since older libraries may be backwards compatible, I suggest you try installing Cuda 11 on your system and see if it is recognized.

- R.


On Sep 9, 2024, at 22:31, Matthew Lynch <matthewl077@gmail.com> wrote:

        External Email - Use Caution        



I run this command: 

mri_synthseg --i /mnt/p/test/test-t2-3mm.nii --parc --robust --o /mnt/p/test

and it fails with the following (partial) error message:

Node: 'model_3/unet_conv_downarm_0_0/Conv3D'
DNN library is not found.

which I assume means it cannot access the appropriate the CuDNN library.

I have installed

GPU: nvidia geforce RTX 3050 Ti (4Gb)
WSL2
Linux kernel: 5.15.153.1-microsoft-standard-WSL2 (after recent wsl --update)
CUDA Version 12.6
Windows Nvidia Driver version 560.94
CuDNN Version 9.4.0 installed in WSL, with libraries located in /lib/x86_64-linux-gnu

However, freesurfer's tensorflow seems to point to its own cudnn libraries in:
/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/nvidia/cudnn/lib

including

libcudnn.so.8 

which I assume is CuDNN version 8, and may not be compatible with CUDA 12.6?

Is there a way I can get freesurfer's script to use the CuDNN 9.4.0 that are installed globally on the WSL system?  Any other suggestions on what might be causing this error and how to fix it?

_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed.  If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> .