Malcolm,
To verify your build, our 'gold standard' is to inspect the segmentations and surfaces manually. Of course, this requires skill both in neuroanatomy, and what to look for in a typical freesurfer recon. We do this with our 'buckner40' (40 subject) and 'adni60' (60 subject) datasets prior to release. We also compare group analysis runs between versions to look for statistical differences.
For you purposes though, it should be sufficient to run asegstatsdiff and aparcstatsdiff comparing the results of a single subject run on your build and the release build. you should not see hippocampal volume differences greater than 2%. you can also send me the results of these stats diff files.
Also, I added a check to the libg2c make check (dev/talairach_avi/test_libg2c) to skip it if SKIP_LIBG2C_TEST is set in the enviro. unfortunately, make check doesnt allow continuing if a test fails (or least i dont know of flag to do so).
N.
On Thu, 2012-01-19 at 09:55 -0600, Malcolm Tobias wrote:
My foray into custom-built binaries looks promising. Using the distributed (CentOS4) build of 5.1, I'm running the Bert/Ernie benchmark in ~12 hours. If I build this myself (using 4.1.2 / Red Hat 4.1.2-51) I can run in ~10 hours. If I build the same code using the Intel compilers (12.0.0) with pretty generic options I can get this down to ~9 hours, which is the 10% speed-up that Nick suggested.
Now the hard part: What steps should I take to validate these custom-built binaries before users start using them for production runs?
It seems like at a minimum I should run and pass all of 'make check'. These seem to stop after the first error. I wonder if it would be better to have them run to completion to get an idea of what all passes/fails?
-For gcc, it fails on the test for libg2c. I can understand why you need to test for this before releasing to the world, but is there some way for me to disable this particular test so it doesn't stop the rest of the tests from running?
-For the Intel compilers, the environment doesn't seem to inherit my LD_LIBRARY_PATH so most of the tests fail early on because of missing libraries. Any thoughts on how to fix this?
The 'make check' seems to test for basic functionality, not necessarily the consistency of the results (please correct me if I'm wrong). To test the validity of the processing, it seems the Bert/Ernie sample data is used. I'm not sure what parts of the results I should check? It's been suggested that I compare the results of the aseg.stats and aparc.stats files, possibly using asegstatsdiff and aparcstatsdiff. Does this seem reasonable? If so, what levels of differences would be considered acceptable?
To put this all another way, what do you guys normally do to test out a build before releasing it?
Cheers, Malcolm