Anyone manage to successfully configure recon-all to run with -use-gpu for recent cuda toolkit versions? I realize cuda support has ceased over the past few years for Freesurfer, but have there been any successful builds for CUDA toolkit versions 6 and up? I am currently looking to configure cuda support on a Tesla K80 GPU, which is part of a small beowulf cluster, which is working wonderfully with FSL's cuda support. Many moons ago, when gpu support was working, I found the -use-gpu option with recon-all to be significantly faster and more practical (for the user) than the -openmp option (which even on a 150 core cluster has provided limited speedup during batch runs). Curious to hear what others have been doing.
-Derek
On 22 June 2016 at 18:17, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Anyone manage to successfully configure recon-all to run with -use-gpu for recent cuda toolkit versions? I realize cuda support has ceased over the past few years for Freesurfer, but have there been any successful builds for CUDA toolkit versions 6 and up? I am currently looking to configure cuda support on a Tesla K80 GPU, which is part of a small beowulf cluster, which is working wonderfully with FSL's cuda support. Many moons ago, when gpu support was working, I found the -use-gpu option with recon-all to be significantly faster and more practical (for the user) than the -openmp option (which even on a 150 core cluster has provided limited speedup during batch runs). Curious to hear what others have been doing.
I've successfully compiled and run mri_ca_register_cuda with CUDA SDK 7.5 (and a K1200 Quadro). I've also made a few performance tweaks, which have shaved another minute or so off mri_ca_register_cuda.
If you're having trouble compiling from source, I can try to help, if you let me know where you're hitting trouble.
Regards,
Richard
Hi Richards,
Thanks for the very quick reply. Good to know that someone has successfully compiled GPU binaries for newer CUDA versions. I took a quick look at the developer's guide for building the cuda binaries from source, but it seems like a very involved process? Do your compiled binaries only work for CUDA SDK 7.5 or have you tested them with other CUDA versions?
Also, would you mind sending me an example binary (e.g. mri_ca_register_cuda) just so I cant test it myself and rule out whether this is just an issue with the way our GPU is configured? (I can't even get cudadetect to acknowledge my GPU as a cuda-enabled device (it just says 0 devices found), despite it successfully acknowledging that the GPU "exists") Please let me know and I cna send you a private email address.
Thanks again for your help! And hopefully this thread will help others interested in updating their gpu-capabilities with Freesurfer.
-Derek ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Wednesday, June 22, 2016 3:46 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 22 June 2016 at 18:17, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Anyone manage to successfully configure recon-all to run with -use-gpu for recent cuda toolkit versions? I realize cuda support has ceased over the past few years for Freesurfer, but have there been any successful builds for CUDA toolkit versions 6 and up? I am currently looking to configure cuda support on a Tesla K80 GPU, which is part of a small beowulf cluster, which is working wonderfully with FSL's cuda support. Many moons ago, when gpu support was working, I found the -use-gpu option with recon-all to be significantly faster and more practical (for the user) than the -openmp option (which even on a 150 core cluster has provided limited speedup during batch runs). Curious to hear what others have been doing.
I've successfully compiled and run mri_ca_register_cuda with CUDA SDK 7.5 (and a K1200 Quadro). I've also made a few performance tweaks, which have shaved another minute or so off mri_ca_register_cuda.
If you're having trouble compiling from source, I can try to help, if you let me know where you're hitting trouble.
Regards,
Richard
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
On 22 June 2016 at 19:32, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Thanks for the very quick reply. Good to know that someone has successfully compiled GPU binaries for newer CUDA versions. I took a quick look at the developer's guide for building the cuda binaries from source, but it seems like a very involved process? Do your compiled binaries only work for CUDA SDK 7.5 or have you tested them with other CUDA versions?
I only have one Linux box, running CentOS 7 (which isn't officially supported by Freesurfer - I had to tweak CFLAGS quite a bit too) with CUDA 7.5.
I wouldn't say the process is very involved to compile from source, but I did have to tweak things a bit. Specifically, after running ./setup_configure, I had to open up the configure script, and find the section which looks like: ############################################################# # Nvidia CUDA enabling ############################################################ CUDA_DIR="" with_cuda=""
And comment out the with_cuda="" line. Then I run configure with --enable-fermi-gpu --with-cuda="/usr/local/cuda" (plus some other options for my machine). Note that for me: $ which nvcc /usr/local/cuda-7.5/bin/nvcc
Also, would you mind sending me an example binary (e.g. mri_ca_register_cuda) just so I cant test it myself and rule out whether this is just an issue with the way our GPU is configured? (I can't even get cudadetect to acknowledge my GPU as a cuda-enabled device (it just says 0 devices found), despite it successfully acknowledging that the GPU "exists") Please let me know and I cna send you a private email address.
I can do, but there's a lot of things which could go wrong.
Are you able to compile and run the sample CUDA programs which NVIDIA supplies as part of the SDK? If you haven't tried that yet, it would be a better place to start.
Thanks again for your help! And hopefully this thread will help others interested in updating their gpu-capabilities with Freesurfer.
Indeed - one Francis Thomas @arizona.edu was asking similar questions a few months ago.
I'm afraid I'm not doing any extra GPU porting right now - I've been getting concerned about potential drift between the CPU and GPU pipelines, and trying to come up with a better way around the problem.
HTH,
Richard
Hi Richard,
Thanks again for the quick response. I am able to compile CUDA programs from the SDK, so I will try to compile from the Fs source per the advice that you provided and see what happens. Will let you know how it goes!
-Derek ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Wednesday, June 22, 2016 6:14 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 22 June 2016 at 19:32, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Thanks for the very quick reply. Good to know that someone has successfully compiled GPU binaries for newer CUDA versions. I took a quick look at the developer's guide for building the cuda binaries from source, but it seems like a very involved process? Do your compiled binaries only work for CUDA SDK 7.5 or have you tested them with other CUDA versions?
I only have one Linux box, running CentOS 7 (which isn't officially supported by Freesurfer - I had to tweak CFLAGS quite a bit too) with CUDA 7.5.
I wouldn't say the process is very involved to compile from source, but I did have to tweak things a bit. Specifically, after running ./setup_configure, I had to open up the configure script, and find the section which looks like: ############################################################# # Nvidia CUDA enabling ############################################################ CUDA_DIR="" with_cuda=""
And comment out the with_cuda="" line. Then I run configure with --enable-fermi-gpu --with-cuda="/usr/local/cuda" (plus some other options for my machine). Note that for me: $ which nvcc /usr/local/cuda-7.5/bin/nvcc
Also, would you mind sending me an example binary (e.g. mri_ca_register_cuda) just so I cant test it myself and rule out whether this is just an issue with the way our GPU is configured? (I can't even get cudadetect to acknowledge my GPU as a cuda-enabled device (it just says 0 devices found), despite it successfully acknowledging that the GPU "exists") Please let me know and I cna send you a private email address.
I can do, but there's a lot of things which could go wrong.
Are you able to compile and run the sample CUDA programs which NVIDIA supplies as part of the SDK? If you haven't tried that yet, it would be a better place to start.
Thanks again for your help! And hopefully this thread will help others interested in updating their gpu-capabilities with Freesurfer.
Indeed - one Francis Thomas @arizona.edu was asking similar questions a few months ago.
I'm afraid I'm not doing any extra GPU porting right now - I've been getting concerned about potential drift between the CPU and GPU pipelines, and trying to come up with a better way around the problem.
HTH,
Richard
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Hi Richard,
Hope this finds you well. I followed your instructions and compiled from source (you were right-- compiling Fs from source isn't too bad at all!) I also made your tweak to configure.in and then ran ./configure like so:
./configure --disable-Werror --enable-fermi-gpu --with-cuda="/usr/local/cuda" --with-pkgs-dir=/tmp/centos6-x86_64-packages --prefix=/usr/local/freesurfer
That ran successfully. However, when I then ran 'make -j4', I get the following error and it exits:
/usr/bin/ld: skipping incompatible /usr/local/cuda/lib/libcudart.so when searching for -lcudart /usr/bin/ld: cannot find -lcudart collect2: error: ld returned 1 exit status Makefile:568: recipe for target 'cudadetect' failed make[2]: *** [cudadetect] Error 1 make[2]: Leaving directory '/tmp/freesurfer/gpu_utils' Makefile:1053: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/tmp/freesurfer' Makefile:962: recipe for target 'all' failed make: *** [all] Error 2
It is very strange, because I don't think this is an issue with not being able to find the cuda libraries (i.e. libcudart*) since they are detected by FSL's CUDA wrappers and are exported in my ~/.bashrc:
export PATH=$PATH:/usr/local/cuda/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/lib
Curious to hear your thoughts on this. Also, if it helps to know, I am using cuda toolkit 6.5 and compiling on Debian 8.0.
Thanks in advance, Derek ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Wednesday, June 22, 2016 6:14 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 22 June 2016 at 19:32, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Thanks for the very quick reply. Good to know that someone has successfully compiled GPU binaries for newer CUDA versions. I took a quick look at the developer's guide for building the cuda binaries from source, but it seems like a very involved process? Do your compiled binaries only work for CUDA SDK 7.5 or have you tested them with other CUDA versions?
I only have one Linux box, running CentOS 7 (which isn't officially supported by Freesurfer - I had to tweak CFLAGS quite a bit too) with CUDA 7.5.
I wouldn't say the process is very involved to compile from source, but I did have to tweak things a bit. Specifically, after running ./setup_configure, I had to open up the configure script, and find the section which looks like: ############################################################# # Nvidia CUDA enabling ############################################################ CUDA_DIR="" with_cuda=""
And comment out the with_cuda="" line. Then I run configure with --enable-fermi-gpu --with-cuda="/usr/local/cuda" (plus some other options for my machine). Note that for me: $ which nvcc /usr/local/cuda-7.5/bin/nvcc
Also, would you mind sending me an example binary (e.g. mri_ca_register_cuda) just so I cant test it myself and rule out whether this is just an issue with the way our GPU is configured? (I can't even get cudadetect to acknowledge my GPU as a cuda-enabled device (it just says 0 devices found), despite it successfully acknowledging that the GPU "exists") Please let me know and I cna send you a private email address.
I can do, but there's a lot of things which could go wrong.
Are you able to compile and run the sample CUDA programs which NVIDIA supplies as part of the SDK? If you haven't tried that yet, it would be a better place to start.
Thanks again for your help! And hopefully this thread will help others interested in updating their gpu-capabilities with Freesurfer.
Indeed - one Francis Thomas @arizona.edu was asking similar questions a few months ago.
I'm afraid I'm not doing any extra GPU porting right now - I've been getting concerned about potential drift between the CPU and GPU pipelines, and trying to come up with a better way around the problem.
HTH,
Richard
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Hi Richard,
I resolved the make error. Just needed to point my libcudart.so in /usr/local/cuda/lib to the 64-bit equivalent in /usr/local/cuda/lib64. Running into other cuda-related issues with gcc now, but that should be much easier to resolve.
-Derek ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of Derek Pisner [dpisner@psychiatry.arizona.edu] Sent: Thursday, June 23, 2016 1:26 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
Hi Richard,
Hope this finds you well. I followed your instructions and compiled from source (you were right-- compiling Fs from source isn't too bad at all!) I also made your tweak to configure.in and then ran ./configure like so:
./configure --disable-Werror --enable-fermi-gpu --with-cuda="/usr/local/cuda" --with-pkgs-dir=/tmp/centos6-x86_64-packages --prefix=/usr/local/freesurfer
That ran successfully. However, when I then ran 'make -j4', I get the following error and it exits:
/usr/bin/ld: skipping incompatible /usr/local/cuda/lib/libcudart.so when searching for -lcudart /usr/bin/ld: cannot find -lcudart collect2: error: ld returned 1 exit status Makefile:568: recipe for target 'cudadetect' failed make[2]: *** [cudadetect] Error 1 make[2]: Leaving directory '/tmp/freesurfer/gpu_utils' Makefile:1053: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/tmp/freesurfer' Makefile:962: recipe for target 'all' failed make: *** [all] Error 2
It is very strange, because I don't think this is an issue with not being able to find the cuda libraries (i.e. libcudart*) since they are detected by FSL's CUDA wrappers and are exported in my ~/.bashrc:
export PATH=$PATH:/usr/local/cuda/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/lib
Curious to hear your thoughts on this. Also, if it helps to know, I am using cuda toolkit 6.5 and compiling on Debian 8.0.
Thanks in advance, Derek ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Wednesday, June 22, 2016 6:14 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 22 June 2016 at 19:32, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Thanks for the very quick reply. Good to know that someone has successfully compiled GPU binaries for newer CUDA versions. I took a quick look at the developer's guide for building the cuda binaries from source, but it seems like a very involved process? Do your compiled binaries only work for CUDA SDK 7.5 or have you tested them with other CUDA versions?
I only have one Linux box, running CentOS 7 (which isn't officially supported by Freesurfer - I had to tweak CFLAGS quite a bit too) with CUDA 7.5.
I wouldn't say the process is very involved to compile from source, but I did have to tweak things a bit. Specifically, after running ./setup_configure, I had to open up the configure script, and find the section which looks like: ############################################################# # Nvidia CUDA enabling ############################################################ CUDA_DIR="" with_cuda=""
And comment out the with_cuda="" line. Then I run configure with --enable-fermi-gpu --with-cuda="/usr/local/cuda" (plus some other options for my machine). Note that for me: $ which nvcc /usr/local/cuda-7.5/bin/nvcc
Also, would you mind sending me an example binary (e.g. mri_ca_register_cuda) just so I cant test it myself and rule out whether this is just an issue with the way our GPU is configured? (I can't even get cudadetect to acknowledge my GPU as a cuda-enabled device (it just says 0 devices found), despite it successfully acknowledging that the GPU "exists") Please let me know and I cna send you a private email address.
I can do, but there's a lot of things which could go wrong.
Are you able to compile and run the sample CUDA programs which NVIDIA supplies as part of the SDK? If you haven't tried that yet, it would be a better place to start.
Thanks again for your help! And hopefully this thread will help others interested in updating their gpu-capabilities with Freesurfer.
Indeed - one Francis Thomas @arizona.edu was asking similar questions a few months ago.
I'm afraid I'm not doing any extra GPU porting right now - I've been getting concerned about potential drift between the CPU and GPU pipelines, and trying to come up with a better way around the problem.
HTH,
Richard
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
On 23 June 2016 at 16:26, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Hi Richard,
Hope this finds you well. I followed your instructions and compiled from source (you were right-- compiling Fs from source isn't too bad at all!) I also made your tweak to configure.in and then ran ./configure like so:
./configure --disable-Werror --enable-fermi-gpu --with-cuda="/usr/local/cuda" --with-pkgs-dir=/tmp/centos6-x86_64-packages --prefix=/usr/local/freesurfer
That ran successfully. However, when I then ran 'make -j4', I get the following error and it exits:
/usr/bin/ld: skipping incompatible /usr/local/cuda/lib/libcudart.so when searching for -lcudart /usr/bin/ld: cannot find -lcudart collect2: error: ld returned 1 exit status Makefile:568: recipe for target 'cudadetect' failed make[2]: *** [cudadetect] Error 1 make[2]: Leaving directory '/tmp/freesurfer/gpu_utils' Makefile:1053: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/tmp/freesurfer' Makefile:962: recipe for target 'all' failed make: *** [all] Error 2
It is very strange, because I don't think this is an issue with not being able to find the cuda libraries (i.e. libcudart*) since they are detected by FSL's CUDA wrappers and are exported in my ~/.bashrc:
export PATH=$PATH:/usr/local/cuda/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/lib
The "skipping incompatible" line looks suspicious - surely it should have been looking for libcudart in /us/local/cuda/lib64 ?
Can you do another, plain "make" and look at the command which is ultimately giving the error (it'll be a gcc with a bunch of object files, to trigger the linking of the executable). That might help us find out what it should be doing.
Richard
Hi Richard,
Fixed the gcc problems as well (this required a tweak to host_config.h in /usr/local/cuda/include). Now, I've hit another roadblock:
Makefile:4542: recipe for target 'libutils_cuda_a-mrisurf.o' failed make[3]: *** [libutils_cuda_a-mrisurf.o] Error 1 make[3]: *** Waiting for unfinished jobs.... mv -f .deps/libutils_cuda_a-mriVolume.Tpo .deps/libutils_cuda_a-mriVolume.Po mv -f .deps/libutils_cuda_a-mri_topology.Tpo .deps/libutils_cuda_a-mri_topology.Po make[3]: Leaving directory '/tmp/freesurfer/utils' Makefile:6185: recipe for target 'all-recursive' failed make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory '/tmp/freesurfer/utils' Makefile:1053: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/tmp/freesurfer' Makefile:962: recipe for target 'all' failed make: *** [all] Error 2
Seems like we are getting really close, but is there some deeper cuda incompatibility here with some of Fs's cuda-enabled tools?
-Derek ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Thursday, June 23, 2016 5:23 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 23 June 2016 at 16:26, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Hi Richard,
Hope this finds you well. I followed your instructions and compiled from source (you were right-- compiling Fs from source isn't too bad at all!) I also made your tweak to configure.in and then ran ./configure like so:
./configure --disable-Werror --enable-fermi-gpu --with-cuda="/usr/local/cuda" --with-pkgs-dir=/tmp/centos6-x86_64-packages --prefix=/usr/local/freesurfer
That ran successfully. However, when I then ran 'make -j4', I get the following error and it exits:
/usr/bin/ld: skipping incompatible /usr/local/cuda/lib/libcudart.so when searching for -lcudart /usr/bin/ld: cannot find -lcudart collect2: error: ld returned 1 exit status Makefile:568: recipe for target 'cudadetect' failed make[2]: *** [cudadetect] Error 1 make[2]: Leaving directory '/tmp/freesurfer/gpu_utils' Makefile:1053: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/tmp/freesurfer' Makefile:962: recipe for target 'all' failed make: *** [all] Error 2
It is very strange, because I don't think this is an issue with not being able to find the cuda libraries (i.e. libcudart*) since they are detected by FSL's CUDA wrappers and are exported in my ~/.bashrc:
export PATH=$PATH:/usr/local/cuda/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/lib
The "skipping incompatible" line looks suspicious - surely it should have been looking for libcudart in /us/local/cuda/lib64 ?
Can you do another, plain "make" and look at the command which is ultimately giving the error (it'll be a gcc with a bunch of object files, to trigger the linking of the executable). That might help us find out what it should be doing.
Richard _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
On 23 June 2016 at 22:44, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Fixed the gcc problems as well (this required a tweak to host_config.h in /usr/local/cuda/include). Now, I've hit another roadblock:
Makefile:4542: recipe for target 'libutils_cuda_a-mrisurf.o' failed make[3]: *** [libutils_cuda_a-mrisurf.o] Error 1 make[3]: *** Waiting for unfinished jobs....
What was the error leading to this failure?
Richard
On 24 June 2016 at 05:34, R Edgar freesurfer.rge@gmail.com wrote:
On 23 June 2016 at 22:44, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Fixed the gcc problems as well (this required a tweak to host_config.h in /usr/local/cuda/include). Now, I've hit another roadblock:
Makefile:4542: recipe for target 'libutils_cuda_a-mrisurf.o' failed make[3]: *** [libutils_cuda_a-mrisurf.o] Error 1 make[3]: *** Waiting for unfinished jobs....
What was the error leading to this failure?
Were you able to resolve this?
Richard
Hi Richard:
Was not able to resolve this quite yet. I figured out that I had to backtrack a bit because I was having trouble getting the complete set of data files from the Fs ftp server using "git annex ." And now it appears that freesurfer's entire ftp server is down and has been all day. I will try again tomorrow and let you know how it goes.
Thanks for sticking it out with me! -Derek ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Sunday, June 26, 2016 1:51 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 24 June 2016 at 05:34, R Edgar freesurfer.rge@gmail.com wrote:
On 23 June 2016 at 22:44, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Fixed the gcc problems as well (this required a tweak to host_config.h in /usr/local/cuda/include). Now, I've hit another roadblock:
Makefile:4542: recipe for target 'libutils_cuda_a-mrisurf.o' failed make[3]: *** [libutils_cuda_a-mrisurf.o] Error 1 make[3]: *** Waiting for unfinished jobs....
What was the error leading to this failure?
Were you able to resolve this?
Richard _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
On 27 June 2016 at 01:27, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Was not able to resolve this quite yet. I figured out that I had to backtrack a bit because I was having trouble getting the complete set of data files from the Fs ftp server using "git annex ." And now it appears that freesurfer's entire ftp server is down and has been all day. I will try again tomorrow and let you know how it goes.
Did it work out?
Richard
Hi,
I have one question here. If the code compiles using sdk 7.5 or latest, will the resulting binary be FS v6? I wonder if the mri_ca_register_cuda code is also changed in v6 tree...and if mixing of binaries from different FS versions would not affect the results that much...
Best regards,
-- Shahid.
On Fri, Jul 1, 2016 at 2:46 AM, R Edgar freesurfer.rge@gmail.com wrote:
On 27 June 2016 at 01:27, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Was not able to resolve this quite yet. I figured out that I had to
backtrack a bit because I was having trouble getting the complete set of data files from the Fs ftp server using "git annex ." And now it appears that freesurfer's entire ftp server is down and has been all day. I will try again tomorrow and let you know how it goes.
Did it work out?
Richard
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
On 1 July 2016 at 04:30, Dr. M. Shahid mohammad.shahid@gmail.com wrote:
I have one question here. If the code compiles using sdk 7.5 or latest, will the resulting binary be FS v6? I wonder if the mri_ca_register_cuda code is also changed in v6 tree...and if mixing of binaries from different FS versions would not affect the results that much...
The binary will be from whichever version of the source tree you checked out. The GPU results are inevitably going to be slightly different to the CPU ones, though. If, however, you find significant differences, I'd like to know about them :-)
HTH,
Richard
Hi Richard,
Sorry for the delay again. I had to get in touch with Zeke at Fs to get the repo working (it was broken for a week)
Despite a fresh source download and following your instructions carefully, I am still experiencing the same set of libcuda util errors upon running make.
See attached pastebin text file for the complete output. I cannot figure out what the issue is. Also, since our GPU is a Tesla K80 and we are using cuda toolkit 6.5, would the --enable-fermi-gpu option with ./configure still make sense?
Many thanks, Derek ________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Thursday, June 30, 2016 5:46 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 27 June 2016 at 01:27, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Was not able to resolve this quite yet. I figured out that I had to backtrack a bit because I was having trouble getting the complete set of data files from the Fs ftp server using "git annex ." And now it appears that freesurfer's entire ftp server is down and has been all day. I will try again tomorrow and let you know how it goes.
Did it work out?
Richard
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Here is the output attachment. ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of Derek Pisner [dpisner@psychiatry.arizona.edu] Sent: Friday, July 01, 2016 2:47 PM To: Freesurfer support list Cc: Matthew Allbright Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
Hi Richard,
Sorry for the delay again. I had to get in touch with Zeke at Fs to get the repo working (it was broken for a week)
Despite a fresh source download and following your instructions carefully, I am still experiencing the same set of libcuda util errors upon running make.
See attached pastebin text file for the complete output. I cannot figure out what the issue is. Also, since our GPU is a Tesla K80 and we are using cuda toolkit 6.5, would the --enable-fermi-gpu option with ./configure still make sense?
Many thanks, Derek ________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Thursday, June 30, 2016 5:46 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 27 June 2016 at 01:27, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Was not able to resolve this quite yet. I figured out that I had to backtrack a bit because I was having trouble getting the complete set of data files from the Fs ftp server using "git annex ." And now it appears that freesurfer's entire ftp server is down and has been all day. I will try again tomorrow and let you know how it goes.
Did it work out?
Richard
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
On 1 July 2016 at 17:48, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Here is the output attachment.
Hmmm.... this is the actual error: ====== gcc -DHAVE_CONFIG_H -I. -I.. -I../include -I../include/dicom -I../xml2 -I../include/NrrdIO -DTEEM_DIO=0 -DTEEM_32BIT=0 -DTEEM_QNANHIBIT=1 -DTEEM_ENDIAN=1234 -I../jpeg -I../tiff -I../expat -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Algorithms -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/BasicFilters -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Common -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/IO -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Numerics -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Numerics/Statistics -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Review -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Review/Statistics -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/SpatialObject -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Utilities -DHAVE_ITK_LIBS -I../glut -DHAVE_GLUT_LIBS -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/vxl/current/include/vxl/core -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/vxl/current/include/vxl/vcl -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/vxl/current/include/vxl/v3p/netlib -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/vxl/current/include/vxl/v3p/netlib/opt -I/usr/local/cuda/include -DHAVE_ZLIB -g -O3 -msse2 -mfpmath=sse -Wall -DUSE_SSE_MATHFUN -fopenmp -DHAVE_OPENMP -m64 -DLinux -DANSI -fdata-sections -ffunction-sections -Wl,--gc-sections -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/mni/current/include -DFS_CUDA -I/usr/local/cuda/include -DGCAMORPH_ON_GPU -MT libutils_cuda_a-mrisurf.o -MD -MP -MF .deps/libutils_cuda_a-mrisurf.Tpo -c -o libutils_cuda_a-mrisurf.o `test -f 'mrisurf.c' || echo './'`mrisurf.c In file included from /usr/local/cuda/include/cuda_runtime.h:59:0, from ../include/mrisurf_cuda.h:28, from mrisurf.c:664: /usr/local/cuda/include/host_config.h:159:2: error: #endif without #if #endif /* !__HOST_CONFIG_H__ */ ======
I've just tried doing a git pull, and restarting from ./setup_configure, and I can't reproduce this, I'm afraid.
You do absolutely want to have --enable-fermi-gpu set, though. Fermi is five years old, and that flag enables most of the acceleration of mri_ca_register.
Richard
Hi Richard,
Thanks anyway for taking a look. Despite a number of further tweaks to our cuda libraries, we have made no further progress with this. Any chance you could pass along your cuda binaries so that I can see whether they might just "work"? Very curious.
-Derek ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Friday, July 01, 2016 5:47 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 1 July 2016 at 17:48, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Here is the output attachment.
Hmmm.... this is the actual error: ====== gcc -DHAVE_CONFIG_H -I. -I.. -I../include -I../include/dicom -I../xml2 -I../include/NrrdIO -DTEEM_DIO=0 -DTEEM_32BIT=0 -DTEEM_QNANHIBIT=1 -DTEEM_ENDIAN=1234 -I../jpeg -I../tiff -I../expat -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Algorithms -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/BasicFilters -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Common -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/IO -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Numerics -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Numerics/Statistics -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Review -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Review/Statistics -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/SpatialObject -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/itk/current/include/InsightToolkit/Utilities -DHAVE_ITK_LIBS -I../glut -DHAVE_GLUT_LIBS -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/vxl/current/include/vxl/core -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/vxl/current/include/vxl/vcl -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/vxl/current/include/vxl/v3p/netlib -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/vxl/current/include/vxl/v3p/netlib/opt -I/usr/local/cuda/include -DHAVE_ZLIB -g -O3 -msse2 -mfpmath=sse -Wall -DUSE_SSE_MATHFUN -fopenmp -DHAVE_OPENMP -m64 -DLinux -DANSI -fdata-sections -ffunction-sections -Wl,--gc-sections -I/home/scanlab/Desktop/Derek/centos6-x86_64-packages/mni/current/include -DFS_CUDA -I/usr/local/cuda/include -DGCAMORPH_ON_GPU -MT libutils_cuda_a-mrisurf.o -MD -MP -MF .deps/libutils_cuda_a-mrisurf.Tpo -c -o libutils_cuda_a-mrisurf.o `test -f 'mrisurf.c' || echo './'`mrisurf.c In file included from /usr/local/cuda/include/cuda_runtime.h:59:0, from ../include/mrisurf_cuda.h:28, from mrisurf.c:664: /usr/local/cuda/include/host_config.h:159:2: error: #endif without #if #endif /* !__HOST_CONFIG_H__ */ ======
I've just tried doing a git pull, and restarting from ./setup_configure, and I can't reproduce this, I'm afraid.
You do absolutely want to have --enable-fermi-gpu set, though. Fermi is five years old, and that flag enables most of the acceleration of mri_ca_register.
Richard _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
On 2 July 2016 at 19:06, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Thanks anyway for taking a look. Despite a number of further tweaks to our cuda libraries, we have made no further progress with this. Any chance you could pass along your cuda binaries so that I can see whether they might just "work"? Very curious.
I've sent a tarball in a separate email - let me know if it gets through. I wouldn't be too hopeful, though - I don't know that libcuda is forward compatible.
Regards,
Richard
Hi Richard,
Many thanks for your effort and help. The binaries did not work unfortunately seemingly because of libcuda from 6.5 not being forward compatible with libcuda from 7.5, as you noted.
-Derek ________________________________________ From: freesurfer-bounces@nmr.mgh.harvard.edu [freesurfer-bounces@nmr.mgh.harvard.edu] on behalf of R Edgar [freesurfer.rge@gmail.com] Sent: Sunday, July 03, 2016 3:06 PM To: Freesurfer support list Subject: Re: [Freesurfer] Any updates on recon-all GPU support?
On 2 July 2016 at 19:06, Derek Pisner dpisner@psychiatry.arizona.edu wrote:
Thanks anyway for taking a look. Despite a number of further tweaks to our cuda libraries, we have made no further progress with this. Any chance you could pass along your cuda binaries so that I can see whether they might just "work"? Very curious.
I've sent a tarball in a separate email - let me know if it gets through. I wouldn't be too hopeful, though - I don't know that libcuda is forward compatible.
Regards,
Richard _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
freesurfer@nmr.mgh.harvard.edu
-
Derek Pisner -
Dr. M. Shahid -
R Edgar