External Email - Use Caution
We are trying to run recon_all inside an fmriprep singularity container, but are running into an issue with nu_correct. It says it cannot write to disk, however there is plenty of space on the tmpdir and output dir. Does this script try to write to any other locations? Because this would fail inside the container, where each writable location has to be 'mounted'.
Error message:
ncendef: ncid 5: No space left on device Error outputting volume: possibly disk full? nu_evaluate: crashed while running evaluate_field (termination status=139) nu_correct: crashed while running nu_evaluate (termination status=65280) ERROR: nu_correct
Command ran:
nu_correct -clobber ./tmp.mri_nu_correct.mni.1121/nu0.mnc ./tmp.mri_nu_correct.mni.1121/nu1.mnc -tmpdir ./tmp.mri_nu_correct.mni.1121/0/ -iterations 1000 -distance 50 [xxx@yyyy.cluster:/data/BIDS/derivatives/freesurfer/sub-1/mri/] [2018-10-04 14:59:29] running: /opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.1121/0/ ./tmp.mri_nu_correct.mni.1121/nu0.mnc ./tmp.mri_nu_correct.mni.1121/nu1.imp
this issue at the fmriprep repo: https://github.com/poldracklab/fmriprep/issues/1308 possibly related: https://github.com/freesurfer/freesurfer/issues/462
Many thanks for any advice, Jasper van den Bosch
Not that I know of, but this is not a native FS script, it comes from the MNI, so it could be doing something we don't understand. Can you do an experiment for us? Can you run it outside of the container using reprozip (or something equivalent) to see all the files it touches?
On 10/5/18 5:31 AM, Jasper van den Bosch wrote:
External Email - Use Caution
We are trying to run recon_all inside an fmriprep singularity container, but are running into an issue with nu_correct. It says it cannot write to disk, however there is plenty of space on the tmpdir and output dir. Does this script try to write to any other locations? Because this would fail inside the container, where each writable location has to be 'mounted'.
Error message:
ncendef: ncid 5: No space left on device Error outputting volume: possibly disk full? nu_evaluate: crashed while running evaluate_field (termination status=139) nu_correct: crashed while running nu_evaluate (termination status=65280) ERROR: nu_correct
Command ran:
nu_correct -clobber ./tmp.mri_nu_correct.mni.1121/nu0.mnc ./tmp.mri_nu_correct.mni.1121/nu1.mnc -tmpdir ./tmp.mri_nu_correct.mni.1121/0/ -iterations 1000 -distance 50 [xxx@yyyy.cluster:/data/BIDS/derivatives/freesurfer/sub-1/mri/mailto:xxx@yyyy.cluster:/data/BIDS/derivatives/freesurfer/sub-1/mri/] [2018-10-04 14:59:29] running: /opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.1121/0/ ./tmp.mri_nu_correct.mni.1121/nu0.mnc ./tmp.mri_nu_correct.mni.1121/nu1.imp
this issue at the fmriprep repo: https://github.com/poldracklab/fmriprep/issues/1308 possibly related: https://github.com/freesurfer/freesurfer/issues/462
Many thanks for any advice, Jasper van den Bosch
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Dear *,
we also run Freesurfer on a cluster and are starting to use singularity, so naturally I'm curious about this issue.
I was able to reproduce the error with a clean singularity run (-c -e). Then, I rebuilt a container to start nu_correct with "strace -f -e open,write" and I believe I found the problem:
[pid 35994] open("/subs/testsub/mri/tmp.mri_nu_correct.mni.35410/nu1.imp", O_RDONLY) = 3 [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_EXCL, 0600) = 5 [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_TRUNC, 0666) = 5 [pid 35994] write(5, "\0\0\0\0\0\0\0\0", 8) = 8 [pid 35994] write(5, "CDF\1\0\0\0\0\0\0\0\n\0\0\0\3\0\0\0\6xspace\0\0\0\0\1\0"..., 8192) = 8192 [pid 35994] write(5, "|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"..., 8192) = 8192 [...] [pid 35994] write(5, "|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"..., 8192) = -1 ENOSPC (No space left on device) [pid 35994] write(2, "ncendef: ", 9ncendef: ) = 9 [pid 35994] write(2, "ncid 5", 6ncid 5) = 6 [pid 35994] write(2, ": No space left on device", 25: No space left on device) = 25
So there are two issues here I think:
1) nu_correct is saving things to /tmp when it could just use ./tmp 2) the singularity default for "sessiondir max size" is 16MB, which includes /tmp, is too small for the minc-tmp files
I changed the value from 16 to 1024 and it passed the nu_correct step.
cheers, Michael
On 10/5/18 5:57 PM, Greve, Douglas N.,Ph.D. wrote:
Not that I know of, but this is not a native FS script, it comes from the MNI, so it could be doing something we don't understand. Can you do an experiment for us? Can you run it outside of the container using reprozip (or something equivalent) to see all the files it touches?
On 10/5/18 5:31 AM, Jasper van den Bosch wrote:
External Email - Use Caution
We are trying to run recon_all inside an fmriprep singularity container, but are running into an issue with nu_correct. It says it cannot write to disk, however there is plenty of space on the tmpdir and output dir. Does this script try to write to any other locations? Because this would fail inside the container, where each writable location has to be 'mounted'.
Error message:
ncendef: ncid 5: No space left on device Error outputting volume: possibly disk full? nu_evaluate: crashed while running evaluate_field (termination status=139) nu_correct: crashed while running nu_evaluate (termination status=65280) ERROR: nu_correct
Command ran:
nu_correct -clobber ./tmp.mri_nu_correct.mni.1121/nu0.mnc ./tmp.mri_nu_correct.mni.1121/nu1.mnc -tmpdir ./tmp.mri_nu_correct.mni.1121/0/ -iterations 1000 -distance 50 [xxx@yyyy.cluster:/data/BIDS/derivatives/freesurfer/sub-1/mri/] [2018-10-04 14:59:29] running: /opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.1121/0/ ./tmp.mri_nu_correct.mni.1121/nu0.mnc ./tmp.mri_nu_correct.mni.1121/nu1.imp
this issue at the fmriprep repo: https://github.com/poldracklab/fmriprep/issues/1308 possibly related: https://github.com/freesurfer/freesurfer/issues/462
Many thanks for any advice, Jasper van den Bosch
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
External Email - Use Caution
External Email - Use Caution
Michael, I can't thank you enough.
Knowing that it writes to /tmp made all the difference. In my case I mounted an external directory on /tmp instead of the sessiondir approach, but it now runs fine. Jasper
On Fri, 5 Oct 2018 at 23:47, Michael Krause krause@mpib-berlin.mpg.de wrote:
Dear *,
we also run Freesurfer on a cluster and are starting to use singularity, so naturally I'm curious about this issue.
I was able to reproduce the error with a clean singularity run (-c -e). Then, I rebuilt a container to start nu_correct with "strace -f -e open,write" and I believe I found the problem:
[pid 35994] open("/subs/testsub/mri/tmp.mri_nu_correct.mni.35410/nu1.imp", O_RDONLY) = 3 [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_EXCL, 0600) = 5 [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_TRUNC, 0666) = 5 [pid 35994] write(5, "\0\0\0\0\0\0\0\0", 8) = 8 [pid 35994] write(5, "CDF\1\0\0\0\0\0\0\0\n\0\0\0\3\0\0\0\6xspace\0\0\0\0\1\0"..., 8192) = 8192 [pid 35994] write(5,
"|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"..., 8192) = 8192 [...] [pid 35994] write(5,
"|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"..., 8192) = -1 ENOSPC (No space left on device) [pid 35994] write(2, "ncendef: ", 9ncendef: ) = 9 [pid 35994] write(2, "ncid 5", 6ncid 5) = 6 [pid 35994] write(2, ": No space left on device", 25: No space left on device) = 25
So there are two issues here I think:
- nu_correct is saving things to /tmp when it could just use ./tmp
- the singularity default for "sessiondir max size" is 16MB, which
includes /tmp, is too small for the minc-tmp files
I changed the value from 16 to 1024 and it passed the nu_correct step.
cheers, Michael
On 10/5/18 5:57 PM, Greve, Douglas N.,Ph.D. wrote:
Not that I know of, but this is not a native FS script, it comes from the MNI, so it could be doing something we don't understand. Can you do an experiment for us? Can you run it outside of the container using reprozip (or something equivalent) to see all the files it touches?
On 10/5/18 5:31 AM, Jasper van den Bosch wrote:
External Email - Use CautionWe are trying to run recon_all inside an fmriprep singularity container, but are running into an issue with nu_correct. It says it cannot write
to
disk, however there is plenty of space on the tmpdir and output dir.
Does
this script try to write to any other locations? Because this would fail inside the container, where each writable location has to be 'mounted'.
Error message:
ncendef: ncid 5: No space left on device Error outputting volume: possibly disk full? nu_evaluate: crashed while running evaluate_field (termination
status=139)
nu_correct: crashed while running nu_evaluate (termination status=65280) ERROR: nu_correct
Command ran:
nu_correct -clobber ./tmp.mri_nu_correct.mni.1121/nu0.mnc ./tmp.mri_nu_correct.mni.1121/nu1.mnc -tmpdir ./tmp.mri_nu_correct.mni.1121/0/ -iterations 1000 -distance 50 [xxx@yyyy.cluster:/data/BIDS/derivatives/freesurfer/sub-1/mri/] [2018-10-04 14:59:29] running: /opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen
0.15
0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify
-b_spline
1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.1121/0/ ./tmp.mri_nu_correct.mni.1121/nu0.mnc ./tmp.mri_nu_correct.mni.1121/nu1.imp
this issue at the fmriprep repo: https://github.com/poldracklab/fmriprep/issues/1308 possibly related: https://github.com/freesurfer/freesurfer/issues/462
Many thanks for any advice, Jasper van den Bosch
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
External Email - Use Caution
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
BTW, this should not be a problem in the next version
On 10/07/2018 08:29 AM, Jasper van den Bosch wrote:
Michael, I can't thank you enough.
Knowing that it writes to /tmp made all the difference. In my case I mounted an external directory on /tmp instead of the sessiondir approach, but it now runs fine. Jasper
On Fri, 5 Oct 2018 at 23:47, Michael Krause <krause@mpib-berlin.mpg.de mailto:krause@mpib-berlin.mpg.de> wrote:
Dear *, we also run Freesurfer on a cluster and are starting to use singularity, so naturally I'm curious about this issue. I was able to reproduce the error with a clean singularity run (-c -e). Then, I rebuilt a container to start nu_correct with "strace -f -e open,write" and I believe I found the problem: [pid 35994] open("/subs/testsub/mri/tmp.mri_nu_correct.mni.35410/nu1.imp", O_RDONLY) = 3 [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_EXCL, 0600) = 5 [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_TRUNC, 0666) = 5 [pid 35994] write(5, "\0\0\0\0\0\0\0\0", 8) = 8 [pid 35994] write(5, "CDF\1\0\0\0\0\0\0\0\n\0\0\0\3\0\0\0\6xspace\0\0\0\0\1\0"..., 8192) = 8192 [pid 35994] write(5, "|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"..., 8192) = 8192 [...] [pid 35994] write(5, "|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"..., 8192) = -1 ENOSPC (No space left on device) [pid 35994] write(2, "ncendef: ", 9ncendef: ) = 9 [pid 35994] write(2, "ncid 5", 6ncid 5) = 6 [pid 35994] write(2, ": No space left on device", 25: No space left on device) = 25 So there are two issues here I think: 1) nu_correct is saving things to /tmp when it could just use ./tmp 2) the singularity default for "sessiondir max size" is 16MB, which includes /tmp, is too small for the minc-tmp files I changed the value from 16 to 1024 and it passed the nu_correct step. cheers, Michael On 10/5/18 5:57 PM, Greve, Douglas N.,Ph.D. wrote: > Not that I know of, but this is not a native FS script, it comes from the > MNI, so it could be doing something we don't understand. Can you do an > experiment for us? Can you run it outside of the container using reprozip > (or something equivalent) to see all the files it touches? > > On 10/5/18 5:31 AM, Jasper van den Bosch wrote: >> >> External Email - Use Caution >> >> We are trying to run recon_all inside an fmriprep singularity container, >> but are running into an issue with nu_correct. It says it cannot write to >> disk, however there is plenty of space on the tmpdir and output dir. Does >> this script try to write to any other locations? Because this would fail >> inside the container, where each writable location has to be 'mounted'. >> >> Error message: >> >> ncendef: ncid 5: No space left on device >> Error outputting volume: possibly disk full? >> nu_evaluate: crashed while running evaluate_field (termination status=139) >> nu_correct: crashed while running nu_evaluate (termination status=65280) >> ERROR: nu_correct >> >> Command ran: >> >> nu_correct -clobber ./tmp.mri_nu_correct.mni.1121/nu0.mnc >> ./tmp.mri_nu_correct.mni.1121/nu1.mnc -tmpdir >> ./tmp.mri_nu_correct.mni.1121/0/ -iterations 1000 -distance 50 >> [xxx@yyyy.cluster:/data/BIDS/derivatives/freesurfer/sub-1/mri/] >> [2018-10-04 14:59:29] running: >> /opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15 >> 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline >> 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir >> ./tmp.mri_nu_correct.mni.1121/0/ ./tmp.mri_nu_correct.mni.1121/nu0.mnc >> ./tmp.mri_nu_correct.mni.1121/nu1.imp >> >> >> this issue at the fmriprep >> repo: https://github.com/poldracklab/fmriprep/issues/1308 >> possibly related: https://github.com/freesurfer/freesurfer/issues/462 >> >> Many thanks for any advice, >> Jasper van den Bosch >> >> >> >> _______________________________________________ >> Freesurfer mailing list >> Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> >> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > External Email - Use Caution _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
freesurfer@nmr.mgh.harvard.edu
-
Greve, Douglas N.,Ph.D. -
Jasper van den Bosch -
Michael Krause