External Email - Use Caution
The 7.4.0 release contains the python distribution all the python scripts (including mri_synthseg) were tested with (on CentOS 8, 7; Ubuntu 22, 20 18). That python includes tensorflow 2.12.0 which is documented to support up thru Cuda 11.8. If you look at all the cuda libraries you will see their major release numbers range from 8-11. Since older libraries may be backwards compatible, I suggest you try installing Cuda 11 on your system and see if it is recognized. - R.
On Sep 9, 2024, at 22:31, Matthew Lynch <matthewl077@gmail.com> wrote: External Email - Use Caution I run this command:
mri_synthseg --i /mnt/p/test/test-t2-3mm.nii --parc --robust --o /mnt/p/test
and it fails with the following (partial) error message:
Node: 'model_3/unet_conv_downarm_0_0/Conv3D' DNN library is not found.
which I assume means it cannot access the appropriate the CuDNN library.
I have installed
GPU: nvidia geforce RTX 3050 Ti (4Gb) WSL2 Linux kernel: 5.15.153.1-microsoft-standard-WSL2 (after recent wsl --update) CUDA Version 12.6 Windows Nvidia Driver version 560.94 CuDNN Version 9.4.0 installed in WSL, with libraries located in /lib/x86_64-linux-gnu
However, freesurfer's tensorflow seems to point to its own cudnn libraries in: /usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/nvidia/cudnn/lib
including
libcudnn.so.8
which I assume is CuDNN version 8, and may not be compatible with CUDA 12.6?
Is there a way I can get freesurfer's script to use the CuDNN 9.4.0 that are installed globally on the WSL system? Any other suggestions on what might be causing this error and how to fix it?
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
External Email - Use Caution
Thank you for the suggestions. It appears now to be seeing the DNN libraries. However, now I get:
OOM when allocating tensor with shape[1,256,256,160,24] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
My GPU has 4Gb RAM and the CPU has 32 Gb RAM. Is this error occurring because of insufficient GPU RAM? What are the requirements of the GPU for mri_synthseg? Is there a workaround for this?
For those who may be having the same trouble configuring the GPU for FreeSurfer 7.4.0 in WSL, I installed an old Nvidia Game Ready Driver 522 https://www.nvidia.com/download/driverResults.aspx/193713/en-us/ (for my GPU) in Windows, which installed CUDA 11.8 support. In WSL, I installed CUDA 11.8 toolkit https://developer.nvidia.com/cuda-11-8-0-download-archive WSL-Ubuntu version using instructions from Nvidia website and CuDNN 8.6.0 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/libcudnn8_8.6.0.163-1+cuda11.8_amd64.deb. I then had to ensure the CuDNN libraries were in the LD_LIBRARY_PATH using:
export LD_LIBRARY_PATH="/usr/lib/wsl/lib:/lib/x86_64-linux-gnu"
This seemed to work, except I received the above mentioned out of memory error.
On Tue, Sep 10, 2024 at 6:04 PM fsbuild fsbuild@contbay.com wrote:
External Email - Use CautionThe 7.4.0 release contains the python distribution all the python scripts (including mri_synthseg) were tested with (on CentOS 8, 7; Ubuntu 22, 20 18). That python includes tensorflow 2.12.0 which is documented to support up thru Cuda 11.8. If you look at all the cuda libraries you will see their major release numbers range from 8-11. Since older libraries may be backwards compatible, I suggest you try installing Cuda 11 on your system and see if it is recognized.
- R.
On Sep 9, 2024, at 22:31, Matthew Lynch matthewl077@gmail.com wrote:
External Email - Use CautionI run this command:
mri_synthseg --i /mnt/p/test/test-t2-3mm.nii --parc --robust --o /mnt/p/test
and it fails with the following (partial) error message:
Node: 'model_3/unet_conv_downarm_0_0/Conv3D' DNN library is not found.
which I assume means it cannot access the appropriate the CuDNN library.
I have installed
GPU: nvidia geforce RTX 3050 Ti (4Gb) WSL2 Linux kernel: 5.15.153.1-microsoft-standard-WSL2 (after recent wsl --update) CUDA Version 12.6 Windows Nvidia Driver version 560.94 CuDNN Version 9.4.0 installed in WSL, with libraries located in /lib/x86_64-linux-gnu
However, freesurfer's tensorflow seems to point to its own cudnn libraries in:
/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/nvidia/cudnn/lib
including
libcudnn.so.8
which I assume is CuDNN version 8, and may not be compatible with CUDA 12.6?
Is there a way I can get freesurfer's script to use the CuDNN 9.4.0 that are installed globally on the WSL system? Any other suggestions on what might be causing this error and how to fix it?
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline < https://www.massgeneralbrigham.org/complianceline%3E .
freesurfer@nmr.mgh.harvard.edu
-
fsbuild -
Matthew Lynch