Hello,

In the past few months I have observed a persistent issue when running recon-all (FreeSurfer version 7.1.1) on subjects whose pial and white matter surfaces have been edited by students in our lab (they would edit brainmask.mgz or wm.mgz, or add control points). The issue started around May of this year, prior to which I had frequently ran recon-all on edited cases without any issues using the same version of FreeSurfer, on the same system (a linux server running on AWS), using various configurations (ranging from an 8 CPU, 32 GB build to a 64 CPU 256 GB build), both with and without the -threads option + different values depending on the build.

 

The problem is as follows: when I submit the cases in different terminals, one after another, using recon-all -s SubjectID/ -all -qcache,

(tested with/without every reasonable -threads or -openmp option), the jobs get stuck on the same step and stop processing, but if I stagger them and leave sufficient time between when each is submitted, they will work as usual and complete without error. Even if I run just two cases on an excessively large server build relative to the processing requirements, they will still get stuck if I start the jobs right after each other (~30 seconds – several minutes). I’ve tried installing the same version of FreeSurfer in a different location, but have observed no change in this pattern of behavior. I asked our IT support to check if anything else had changed, and they reported that there was nothing they could think of that has changed from an infrastructure perspective (same server configuration), and that the only thing that had changed was the size of the hard drive a couple times.

 

A very common step the jobs seem to get stuck is mri_em_reg+, and when I open the terminal, the last lines read  “Nine parameter search.  iteration # nscales = 1 . . .” at the bottom of the terminal followed by a line of asterisks (see screenshot below).

 

 

 

Here is an example of the output of top, which shows that all the recon jobs had a status of “sleeping” after 6+ hours of being submitted and were on the same step:

 

 

 

Any suggestions/insight into what is happening here would be greatly appreciated, many thanks!