External Email - Use Caution

Dear FreeSurfer Team,

I'm currently testing an LGI pipeline using FreeSurfer 8.0.0's -localGI recon-all command on our university's Linux-based computing cluster, aiming to get LGI statistics from 12,000ish subjects.

We've got it successfully working for the most part, but we've run into the following issue:

We currently run 50 subjects at a time in parallel (using TAC Launcher) across 4 nodes. In each batch of 50, 4 subjects (1 on each node) finish the left hemisphere of the -localGI command, and never start the right hemisphere process. They maintain the IsRunning.lh+rh file in their scripts directory, where it stays after the computation times out and is ended by the SLURM system. The other 46 complete both the lh and rh with no problems, apart from the occasional failure due to an MRI artifact.

We've tried shrinking the batch sizes, looking for missing files, and checking matlab dependencies, and haven't been able to figure out the problem, though we've ruled out it being a memory issue. We've also separated these one-hemisphere-run folks out and tried running just the right hemisphere -localGI command on them. Few if any of these go through.

I've checked the mailing list and haven't found anything relevant—any ideas?

I've attached recon-all logs and IsRunning files from one of these groups of 4, as well as an additional recon-all log from an attempted right hemisphere rerun.

Thank you for your time and attention, Robert Toms --------------------------------- Lab Manager, DAMMI Lab School of Brain & Behavioral Sciences University of Texas at Dallas ---------------------------------