Just wanted to report back a success. I had to do some fixes so hope this helps anybody.
My gpus are GTX1080Ti and the CPU involved is E5-2620 dual 10 core on this server.
The vanilla cpu recon-all finished in 7 hours, the gpu run finished in 4.3 hours.
Q: I seem to not have any errors or indications why mri_glmfit or mri_robust_register cuda binaries did not compile. Any ideas?
-Henk
In configure file (from previous post) comment out the with_cuda="" line https://mail.nmr.mgh.harvard.edu/pipermail//freesurfer/2016-June/046236.html CUDA_DIR="" with_cuda=""
In utils/mrifilter.c uncomment line defining 'int width, weight, height' above the test if (width <=1 || height <=1 || depth <=1) on line 3005
then find Makefiles with incorrect gpu arch for the gpu packages change architecture from sm_?? to sm_61 (multiple times per file)
which nvcc
export PATH=/usr/local/cuda/bin:$PATH locate libcuda.so export LD_LIBRARY_PATH=/usr/lib64:/usr/local/cuda/lib64:$LD_LIBRARY_PATH
./configure --disable-Werror --with-pkgs-dir=/usr/local/centos6-x86_64-packages --prefix=/usr/local/freesurfer-6.0.0 --with-cuda=/usr/local/cuda make clean make -j 4 make install
yields
./bin/mris_flatten_cuda ./bin/mris_sphere_cuda ./bin/mris_fix_topology_cuda ./bin/mri_segstats_cuda ./bin/mri_em_register_cuda ./bin/mri_vol2vol_cuda ./bin/mris_inflate_cuda ./bin/mri_ca_register_cuda
Well, it looks the *_cuda files for mri_glmfit and mri_robust_register and some others were dropped from the repository. Downloaded fresh copy today but the files are not there. So no errors.
-Henk
________________________________ From: Meij, Henk Sent: Monday, November 27, 2017 11:02:44 AM To: freesurfer@nmr.mgh.harvard.edu Subject: GPU compile cuda8/centos7
Just wanted to report back a success. I had to do some fixes so hope this helps anybody.
My gpus are GTX1080Ti and the CPU involved is E5-2620 dual 10 core on this server.
The vanilla cpu recon-all finished in 7 hours, the gpu run finished in 4.3 hours.
Q: I seem to not have any errors or indications why mri_glmfit or mri_robust_register cuda binaries did not compile. Any ideas?
-Henk
In configure file (from previous post) comment out the with_cuda="" line https://mail.nmr.mgh.harvard.edu/pipermail//freesurfer/2016-June/046236.html CUDA_DIR="" with_cuda=""
In utils/mrifilter.c uncomment line defining 'int width, weight, height' above the test if (width <=1 || height <=1 || depth <=1) on line 3005
then find Makefiles with incorrect gpu arch for the gpu packages change architecture from sm_?? to sm_61 (multiple times per file)
which nvcc
export PATH=/usr/local/cuda/bin:$PATH locate libcuda.so export LD_LIBRARY_PATH=/usr/lib64:/usr/local/cuda/lib64:$LD_LIBRARY_PATH
./configure --disable-Werror --with-pkgs-dir=/usr/local/centos6-x86_64-packages --prefix=/usr/local/freesurfer-6.0.0 --with-cuda=/usr/local/cuda make clean make -j 4 make install
yields
./bin/mris_flatten_cuda ./bin/mris_sphere_cuda ./bin/mris_fix_topology_cuda ./bin/mri_segstats_cuda ./bin/mri_em_register_cuda ./bin/mri_vol2vol_cuda ./bin/mris_inflate_cuda ./bin/mri_ca_register_cuda
Hi,
Has all of the GPU code also been updated in the new FS6.0 version or the code is still from the old 5.x but can be compiled with new cuda toolkits?
Best regards,
--
Shahid.
On Mon, Nov 27, 2017 at 9:29 PM, Meij, Henk hmeij@wesleyan.edu wrote:
Well, it looks the *_cuda files for mri_glmfit and mri_robust_register and some others were dropped from the repository. Downloaded fresh copy today but the files are not there. So no errors.
-Henk
*From:* Meij, Henk *Sent:* Monday, November 27, 2017 11:02:44 AM *To:* freesurfer@nmr.mgh.harvard.edu *Subject:* GPU compile cuda8/centos7
Just wanted to report back a success. I had to do some fixes so hope this helps anybody.
My gpus are GTX1080Ti and the CPU involved is E5-2620 dual 10 core on this server.
The vanilla cpu recon-all finished in 7 hours, the gpu run finished in 4.3 hours.
Q: I seem to not have any errors or indications why mri_glmfit or mri_robust_register cuda binaries did not compile. Any ideas?
-Henk
In configure file (from previous post) comment out the with_cuda="" line https://mail.nmr.mgh.harvard.edu/pipermail//freesurfer/ 2016-June/046236.html CUDA_DIR="" with_cuda=""
In utils/mrifilter.c uncomment line defining 'int width, weight, height' above the test if (width <=1 || height <=1 || depth <=1) on line 3005
then find Makefiles with incorrect gpu arch for the gpu packages change architecture from sm_?? to sm_61 (multiple times per file)
which nvcc
export PATH=/usr/local/cuda/bin:$PATH locate libcuda.so export LD_LIBRARY_PATH=/usr/lib64:/usr/local/cuda/lib64:$LD_LIBRARY_PATH
./configure --disable-Werror --with-pkgs-dir=/usr/local/centos6-x86_64-packages --prefix=/usr/local/freesurfer-6.0.0 --with-cuda=/usr/local/cuda make clean make -j 4 make install
yields
./bin/mris_flatten_cuda ./bin/mris_sphere_cuda ./bin/mris_fix_topology_cuda ./bin/mri_segstats_cuda ./bin/mri_em_register_cuda ./bin/mri_vol2vol_cuda ./bin/mris_inflate_cuda ./bin/mri_ca_register_cuda
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
On 28 November 2017 at 09:12, Dr. M. Shahid mohammad.shahid@gmail.com wrote:
Has all of the GPU code also been updated in the new FS6.0 version or the code is still from the old 5.x but can be compiled with new cuda toolkits?
I made a small tweak for GCAMorph (primarily ca_register) which might have made it in to v6; I'm not aware of any other changes. However, being compiled with a new compiler for the new architectures should give a small speed boost.
Richard
On Wed, Nov 29, 2017 at 5:27 AM, R Edgar freesurfer.rge@gmail.com wrote:
On 28 November 2017 at 09:12, Dr. M. Shahid mohammad.shahid@gmail.com wrote:
Has all of the GPU code also been updated in the new FS6.0 version or the code is still from the old 5.x but can be compiled with new cuda
toolkits?
I made a small tweak for GCAMorph (primarily ca_register) which might have made it in to v6; I'm not aware of any other changes. However, being compiled with a new compiler for the new architectures should give a small speed boost.
Thanks for the info.
However if the cpu code in FS6 is changed a lot, but the cuda code not, then the results would be different even if we get speed boost. Then how would we take the different results?
Best regards,
-- Shahid.
Richard _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
On 30 November 2017 at 09:42, Dr. M. Shahid mohammad.shahid@gmail.com wrote:
Has all of the GPU code also been updated in the new FS6.0 version or the code is still from the old 5.x but can be compiled with new cuda toolkits?
I made a small tweak for GCAMorph (primarily ca_register) which might have made it in to v6; I'm not aware of any other changes. However, being compiled with a new compiler for the new architectures should give a small speed boost.
Thanks for the info.
However if the cpu code in FS6 is changed a lot, but the cuda code not, then the results would be different even if we get speed boost. Then how would we take the different results?
I don't know if there have been changes in the relevant portions of the CPU code. And unfortunately, there aren't any automated comparisons. If there have, then there will obviously be larger deviations between the CPU and GPU streams (they will never be identical anyway).
Richard
On 27 November 2017 at 11:02, Meij, Henk hmeij@wesleyan.edu wrote:
Just wanted to report back a success. I had to do some fixes so hope this helps anybody.
My gpus are GTX1080Ti and the CPU involved is E5-2620 dual 10 core on this server.
The vanilla cpu recon-all finished in 7 hours, the gpu run finished in 4.3 hours.
Thanks for the update. Can you give the CPU vs GPU times for individual programs in the pipeline? I'd be interested to know those (especially for a machine with as many cores as yours).
TIA,
Richard
freesurfer@nmr.mgh.harvard.edu
-
Dr. M. Shahid -
Meij, Henk -
R Edgar