consistency in recon-all parallel pipeline

List overview All Threads
Download

newer

older

recon-all-clinical synthseg error

Temporal lobe mask

Horn, Mitchell Jacob

2 Feb 2024 2 Feb '24

11:20 a.m.

Hi FS Devs,

I'm experiencing unreproducible thickness results when running any 7+ version with parallelization enabled. Running recon-all without parallelization produces consistent thickness results. I'm running this in AlmaLinux8 (a library-equivalent downstream OS to CentOS8).

I'm attaching a table (table1) of 12 recons with bert:

* 3 parallelized with CentOS8-compiled 7.4.1 * 3 non-parallelized with CentOS8-compiled 7.4.1 * 3 parallelized with CentOS7-compiled 7.4.1 * 3 non-parallelized with CentOS7-compiled 7.4.1

I suspected the downstream CentOS8 libm was the culprit (because of testing I did this last 2023 summer). I ran 3 more recons parallelized with the CentOS7-compiled 7.4.1, but before running the recon-all command, set LD_PRELOAD to a copy of the CentOS7 libm libraries. The thickness results were then consistent, see the second table below (table2). I could not run this experiment on the CentOS8-compiled version, as that one is obviously not backward compatible with CentOS7 libm.

As a quick test of the OS-dependency, I submitted 3 parallel recons on MLSC with 7.3.3. Each reported different thickness. See table 3 (table3).

I'm asking if you can please confirm whether running any 7.+ version with parallelization is generating reproducible results for you in CentOS8 (or equivalent)?

P.S. - I tested 6.0 (CentOS6-compilation) with parallelization, and the results were consistent.

Best, Mitch

Attachments:

attachment.html (text/html — 8.7 KB)
table1.png (image/png — 234.4 KB)
table2.png (image/png — 93.0 KB)
table3.png (image/png — 74.6 KB)

Show replies by date

Douglas N. Greve

23 Feb 23 Feb

11:02 a.m.

So, is the bottom line that when you run 7.4.1 on COS8 in parallel that you get (slightly) different results each time?

On 2/2/2024 11:20 AM, Horn, Mitchell Jacob wrote:

...

Hi FS Devs,

I’m experiencing unreproducible thickness results when running any 7+ version with parallelization enabled. Running recon-all without parallelization produces consistent thickness results. I’m running this in AlmaLinux8 (a library-equivalent downstream OS to CentOS8).

I’m attaching a table (table1) of 12 recons with bert:

3 parallelized with CentOS8-compiled 7.4.1

3 non-parallelized with CentOS8-compiled 7.4.1

3 parallelized with CentOS7-compiled 7.4.1

3 non-parallelized with CentOS7-compiled 7.4.1

I suspected the downstream CentOS8 libm was the culprit (because of testing I did this last 2023 summer). I ran 3 more recons parallelized with the CentOS7-compiled 7.4.1, but before running the recon-all command, set LD_PRELOAD to a copy of the CentOS7 libm libraries. The thickness results were then consistent, see the second table below (table2). I could not run this experiment on the CentOS8-compiled version, as that one is obviously not backward compatible with CentOS7 libm.

As a quick test of the OS-dependency, I submitted 3 parallel recons on MLSC with 7.3.3. Each reported different thickness. See table 3 (table3).

I’m asking if you can please confirm whether running any 7.+ version with parallelization is generating reproducible results for you in CentOS8 (or equivalent)?

P.S. - I tested 6.0 (CentOS6-compilation) with parallelization, and the results were consistent.

Best,

Mitch

Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

Horn, Mitchell Jacob

12:28 p.m.

Bottom line is that when I run any FreeSurfer version 7+ in parallel on COS8 I get different results each time.

Thanks, Mitch

From: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu On Behalf Of Douglas N. Greve Sent: Friday, February 23, 2024 11:03 AM To: freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] consistency in recon-all parallel pipeline

So, is the bottom line that when you run 7.4.1 on COS8 in parallel that you get (slightly) different results each time? On 2/2/2024 11:20 AM, Horn, Mitchell Jacob wrote:

Hi FS Devs,

I’m experiencing unreproducible thickness results when running any 7+ version with parallelization enabled. Running recon-all without parallelization produces consistent thickness results. I’m running this in AlmaLinux8 (a library-equivalent downstream OS to CentOS8).

I’m attaching a table (table1) of 12 recons with bert:

1. 3 parallelized with CentOS8-compiled 7.4.1 2. 3 non-parallelized with CentOS8-compiled 7.4.1 3. 3 parallelized with CentOS7-compiled 7.4.1 4. 3 non-parallelized with CentOS7-compiled 7.4.1

As a quick test of the OS-dependency, I submitted 3 parallel recons on MLSC with 7.3.3. Each reported different thickness. See table 3 (table3).

I’m asking if you can please confirm whether running any 7.+ version with parallelization is generating reproducible results for you in CentOS8 (or equivalent)?

P.S. - I tested 6.0 (CentOS6-compilation) with parallelization, and the results were consistent.

Best, Mitch

_______________________________________________

Freesurfer mailing list

Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu

https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

Huang, Yujing

2:30 p.m.

Do you run ‘recon-all -parallel’ or ‘recon-all –threads <nthreads>’?

From: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu On Behalf Of Horn, Mitchell Jacob Sent: Friday, February 23, 2024 12:28 PM To: Freesurfer support list freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] consistency in recon-all parallel pipeline

Bottom line is that when I run any FreeSurfer version 7+ in parallel on COS8 I get different results each time.

Thanks, Mitch

From: freesurfer-bounces@nmr.mgh.harvard.edumailto:freesurfer-bounces@nmr.mgh.harvard.edu <freesurfer-bounces@nmr.mgh.harvard.edumailto:freesurfer-bounces@nmr.mgh.harvard.edu> On Behalf Of Douglas N. Greve Sent: Friday, February 23, 2024 11:03 AM To: freesurfer@nmr.mgh.harvard.edumailto:freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] consistency in recon-all parallel pipeline

So, is the bottom line that when you run 7.4.1 on COS8 in parallel that you get (slightly) different results each time? On 2/2/2024 11:20 AM, Horn, Mitchell Jacob wrote:

Hi FS Devs,

I’m attaching a table (table1) of 12 recons with bert:

1. 3 parallelized with CentOS8-compiled 7.4.1 2. 3 non-parallelized with CentOS8-compiled 7.4.1 3. 3 parallelized with CentOS7-compiled 7.4.1 4. 3 non-parallelized with CentOS7-compiled 7.4.1

As a quick test of the OS-dependency, I submitted 3 parallel recons on MLSC with 7.3.3. Each reported different thickness. See table 3 (table3).

I’m asking if you can please confirm whether running any 7.+ version with parallelization is generating reproducible results for you in CentOS8 (or equivalent)?

P.S. - I tested 6.0 (CentOS6-compilation) with parallelization, and the results were consistent.

Best, Mitch

_______________________________________________

Freesurfer mailing list

Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu

https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

Horn, Mitchell Jacob

2:44 p.m.

This is all with ‘recon-all -parallel’

Thanks, Mitch

From: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu On Behalf Of Huang, Yujing Sent: Friday, February 23, 2024 2:30 PM To: Freesurfer support list freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] consistency in recon-all parallel pipeline

Do you run ‘recon-all -parallel’ or ‘recon-all –threads <nthreads>’?

From: freesurfer-bounces@nmr.mgh.harvard.edumailto:freesurfer-bounces@nmr.mgh.harvard.edu <freesurfer-bounces@nmr.mgh.harvard.edumailto:freesurfer-bounces@nmr.mgh.harvard.edu> On Behalf Of Horn, Mitchell Jacob Sent: Friday, February 23, 2024 12:28 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edumailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] consistency in recon-all parallel pipeline

Bottom line is that when I run any FreeSurfer version 7+ in parallel on COS8 I get different results each time.

Thanks, Mitch

So, is the bottom line that when you run 7.4.1 on COS8 in parallel that you get (slightly) different results each time? On 2/2/2024 11:20 AM, Horn, Mitchell Jacob wrote:

Hi FS Devs,

I’m attaching a table (table1) of 12 recons with bert:

1. 3 parallelized with CentOS8-compiled 7.4.1 2. 3 non-parallelized with CentOS8-compiled 7.4.1 3. 3 parallelized with CentOS7-compiled 7.4.1 4. 3 non-parallelized with CentOS7-compiled 7.4.1

As a quick test of the OS-dependency, I submitted 3 parallel recons on MLSC with 7.3.3. Each reported different thickness. See table 3 (table3).

I’m asking if you can please confirm whether running any 7.+ version with parallelization is generating reproducible results for you in CentOS8 (or equivalent)?

P.S. - I tested 6.0 (CentOS6-compilation) with parallelization, and the results were consistent.

Best, Mitch

_______________________________________________

Freesurfer mailing list

Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu

https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

Douglas N. Greve

24 Feb 24 Feb

1:48 p.m.

Can you try it with -threads instead? We don't typically use -parallel

On 2/23/2024 2:44 PM, Horn, Mitchell Jacob wrote:

...

This is all with ‘recon-all -parallel’

Thanks,

Mitch

*From:*freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu *On Behalf Of *Huang, Yujing *Sent:* Friday, February 23, 2024 2:30 PM *To:* Freesurfer support list freesurfer@nmr.mgh.harvard.edu *Subject:* Re: [Freesurfer] consistency in recon-all parallel pipeline

Do you run ‘recon-all -parallel’ or ‘recon-all –threads <nthreads>’?

*From:*freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu *On Behalf Of *Horn, Mitchell Jacob *Sent:* Friday, February 23, 2024 12:28 PM *To:* Freesurfer support list freesurfer@nmr.mgh.harvard.edu *Subject:* Re: [Freesurfer] consistency in recon-all parallel pipeline

Bottom line is that when I run any FreeSurfer version 7+ in parallel on COS8 I get different results each time.

Thanks,

Mitch

*From:*freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu *On Behalf Of *Douglas N. Greve *Sent:* Friday, February 23, 2024 11:03 AM *To:* freesurfer@nmr.mgh.harvard.edu *Subject:* Re: [Freesurfer] consistency in recon-all parallel pipeline

So, is the bottom line that when you run 7.4.1 on COS8 in parallel that you get (slightly) different results each time?

On 2/2/2024 11:20 AM, Horn, Mitchell Jacob wrote:
Hi FS Devs,

I’m experiencing unreproducible thickness results when running any
7+ version with parallelization enabled. Running recon-all without
parallelization produces consistent thickness results. I’m running
this in AlmaLinux8 (a library-equivalent downstream OS to CentOS8).

I’m attaching a table (table1) of 12 recons with bert:

 1. 3 parallelized with CentOS8-compiled 7.4.1
 2. 3 non-parallelized with CentOS8-compiled 7.4.1
 3. 3 parallelized with CentOS7-compiled 7.4.1
 4. 3 non-parallelized with CentOS7-compiled 7.4.1

I suspected the downstream CentOS8 libm was the culprit (because
of testing I did this last 2023 summer). I ran 3 more recons
parallelized with the CentOS7-compiled 7.4.1, but before running
the recon-all command, set LD_PRELOAD to a copy of the CentOS7
libm libraries. The thickness results were then consistent, see
the second table below (table2). I could not run this experiment
on the CentOS8-compiled version, as that one is obviously not
backward compatible with CentOS7 libm.

As a quick test of the OS-dependency, I submitted 3 parallel
recons on MLSC with 7.3.3. Each reported different thickness. See
table 3 (table3).

I’m asking if you can please confirm whether running any 7.+
version with parallelization is generating reproducible results
for you in CentOS8 (or equivalent)?

P.S. - I tested 6.0 (CentOS6-compilation) with parallelization,
and the results were consistent.

Best,

Mitch

_______________________________________________

Freesurfer mailing list

Freesurfer@nmr.mgh.harvard.edu

https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

Horn, Mitchell Jacob

7:49 p.m.

I tried with -threads instead, but it’s still inconsistent. See attached table with just ctx averages for 3 regions across 3 separate runs with bert.

Out of curiosity, what’s the difference between -threads and -parallel workflows?

Thanks, Mitch [A screenshot of a computer Description automatically generated]

From: freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu on behalf of Douglas N. Greve dgreve@mgh.harvard.edu Date: Saturday, February 24, 2024 at 1:48 PM To: freesurfer@nmr.mgh.harvard.edu freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] consistency in recon-all parallel pipeline Can you try it with -threads instead? We don't typically use -parallel On 2/23/2024 2:44 PM, Horn, Mitchell Jacob wrote: This is all with ‘recon-all -parallel’

Thanks, Mitch

From: freesurfer-bounces@nmr.mgh.harvard.edumailto:freesurfer-bounces@nmr.mgh.harvard.edu freesurfer-bounces@nmr.mgh.harvard.edu mailto:freesurfer-bounces@nmr.mgh.harvard.edu On Behalf Of Huang, Yujing Sent: Friday, February 23, 2024 2:30 PM To: Freesurfer support list freesurfer@nmr.mgh.harvard.edu mailto:freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] consistency in recon-all parallel pipeline

Do you run ‘recon-all -parallel’ or ‘recon-all –threads <nthreads>’?

From: freesurfer-bounces@nmr.mgh.harvard.edumailto:freesurfer-bounces@nmr.mgh.harvard.edu <freesurfer-bounces@nmr.mgh.harvard.edumailto:freesurfer-bounces@nmr.mgh.harvard.edu> On Behalf Of Horn, Mitchell Jacob Sent: Friday, February 23, 2024 12:28 PM To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edumailto:freesurfer@nmr.mgh.harvard.edu> Subject: Re: [Freesurfer] consistency in recon-all parallel pipeline

Bottom line is that when I run any FreeSurfer version 7+ in parallel on COS8 I get different results each time.

Thanks, Mitch

So, is the bottom line that when you run 7.4.1 on COS8 in parallel that you get (slightly) different results each time? On 2/2/2024 11:20 AM, Horn, Mitchell Jacob wrote:

Hi FS Devs,

I’m attaching a table (table1) of 12 recons with bert:

1. 3 parallelized with CentOS8-compiled 7.4.1 2. 3 non-parallelized with CentOS8-compiled 7.4.1 3. 3 parallelized with CentOS7-compiled 7.4.1 4. 3 non-parallelized with CentOS7-compiled 7.4.1

As a quick test of the OS-dependency, I submitted 3 parallel recons on MLSC with 7.3.3. Each reported different thickness. See table 3 (table3).

I’m asking if you can please confirm whether running any 7.+ version with parallelization is generating reproducible results for you in CentOS8 (or equivalent)?

P.S. - I tested 6.0 (CentOS6-compilation) with parallelization, and the results were consistent.

Best, Mitch

_______________________________________________

Freesurfer mailing list

Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu

https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer

_______________________________________________

Freesurfer mailing list

Freesurfer@nmr.mgh.harvard.edumailto:Freesurfer@nmr.mgh.harvard.edu

https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer