Hello,
When running more than 3 analyses in parallel in FreeSurfer 5.3, I am getting matlab crashes. Mathworks has politely declined to provide support unless I can replicate outside FsFAST.
The errors are variants of
"""
OMP: Error #34: System unable to allocate necessary resources for OMP thread: OMP: System error #11: Resource temporarily unavailable [...] MATLAB is exiting because of fatal error Killed ------------------------------------------ ERROR: fast_selxavg3() failed\n
"""
thread_monitor Resource temporarily unavailable in pthread_create *** glibc detected *** /usr/local/MATLAB/R2015b/bin/glnxa64/MATLAB: double free or corruption (!prev): 0x00007f585308c390 *** Segmentation fault (core dumped)
"""
I have copied Matlab version info below. Any tips on how to troubleshoot?
---------------------------------------------------------------------------------------------------- MATLAB Version: 8.6.0.267246 (R2015b) MATLAB License Number: 1098672 Operating System: Linux 2.6.32-573.18.1.el6.x86_64 #1 SMP Wed Jan 6 11:20:49 EST 2016 x86_64 Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode ---------------------------------------------------------------------------------------------------- MATLAB Version 8.6 (R2015b) Computer Vision System Toolbox Version 7.0 (R2015b) Image Processing Toolbox Version 9.3 (R2015b) Optimization Toolbox Version 7.3 (R2015b) Signal Processing Toolbox Version 7.1 (R2015b) Statistics and Machine Learning Toolbox Version 10.1 (R2015b)
*stephen v. shepherd phd* The Rockefeller University / 1230 York Avenue / New York NY 10065-6307 USA // 212.327.7699
sounds like you are running out of memory, no? On Wed, 18 May 2016, S.V.Shepherd [work] wrote:
Hello,
When running more than 3 analyses in parallel in FreeSurfer 5.3, I am getting matlab crashes. Mathworks has politely declined to provide support unless I can replicate outside FsFAST.
The errors are variants of
"""
OMP: Error #34: System unable to allocate necessary resources for OMP thread: OMP: System error #11: Resource temporarily unavailable [...] MATLAB is exiting because of fatal error Killed
ERROR: fast_selxavg3() failed\n
"""
thread_monitor Resource temporarily unavailable in pthread_create *** glibc detected *** /usr/local/MATLAB/R2015b/bin/glnxa64/MATLAB: double free or corruption (!prev): 0x00007f585308c390 *** Segmentation fault (core dumped)
"""
I have copied Matlab version info below. Any tips on how to troubleshoot?
MATLAB Version: 8.6.0.267246 (R2015b) MATLAB License Number: 1098672 Operating System: Linux 2.6.32-573.18.1.el6.x86_64 #1 SMP Wed Jan 6 11:20:49 EST 2016 x86_64 Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
MATLAB Version 8.6 (R2015b) Computer Vision System Toolbox Version 7.0 (R2015b) Image Processing Toolbox Version 9.3 (R2015b) Optimization Toolbox Version 7.3 (R2015b) Signal Processing Toolbox Version 7.1 (R2015b) Statistics and Machine Learning Toolbox Version 10.1 (R2015b)
stephen v. shepherd phd The Rockefeller University / 1230 York Avenue / New York NY 10065-6307 USA // 212.327.7699
Does it always die like that? It may be out of memory.
On 05/18/2016 01:50 PM, S.V.Shepherd [work] wrote:
Hello,
When running more than 3 analyses in parallel in FreeSurfer 5.3, I am getting matlab crashes. Mathworks has politely declined to provide support unless I can replicate outside FsFAST.
The errors are variants of
"""
OMP: Error #34: System unable to allocate necessary resources for OMP thread: OMP: System error #11: Resource temporarily unavailable [...] MATLAB is exiting because of fatal error Killed
ERROR: fast_selxavg3() failed\n
"""
thread_monitor Resource temporarily unavailable in pthread_create *** glibc detected *** /usr/local/MATLAB/R2015b/bin/glnxa64/MATLAB: double free or corruption (!prev): 0x00007f585308c390 *** Segmentation fault (core dumped)
"""
I have copied Matlab version info below. Any tips on how to troubleshoot?
MATLAB Version: 8.6.0.267246 (R2015b) MATLAB License Number: 1098672 Operating System: Linux 2.6.32-573.18.1.el6.x86_64 #1 SMP Wed Jan 6 11:20:49 EST 2016 x86_64 Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
MATLAB Version 8.6 (R2015b) Computer Vision System Toolbox Version 7.0 (R2015b) Image Processing Toolbox Version 9.3 (R2015b) Optimization Toolbox Version 7.3 (R2015b) Signal Processing Toolbox Version 7.1 (R2015b) Statistics and Machine Learning Toolbox Version 10.1 (R2015b)
_stephen v. shepherd phd_ The Rockefeller University / 1230 York Avenue / New York NY 10065-6307 USA // 212.327.7699
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
It happens when I run more than 3-4 analyses at once. It appears to be sensitive to other instances of Matlab being open. however, I don't see clear evidence I'm memory limited. for example, I'm currently running four selxavg in parallel, along with a separate melodic analysis. One more would cause a crash.
free -g returns, in the "free" column: mem: 4 buffers/cache 480 swap 127
i think this means I have 480gb free, no?
stephen
*stephen v. shepherd phd* The Rockefeller University / 1230 York Avenue / New York NY 10065-6307 USA // 212.327.7699
On Thu, May 19, 2016 at 11:03 AM, Douglas N Greve <greve@nmr.mgh.harvard.edu
wrote:
Does it always die like that? It may be out of memory.
On 05/18/2016 01:50 PM, S.V.Shepherd [work] wrote:
Hello,
When running more than 3 analyses in parallel in FreeSurfer 5.3, I am getting matlab crashes. Mathworks has politely declined to provide support unless I can replicate outside FsFAST.
The errors are variants of
"""
OMP: Error #34: System unable to allocate necessary resources for OMP thread: OMP: System error #11: Resource temporarily unavailable [...] MATLAB is exiting because of fatal error Killed
ERROR: fast_selxavg3() failed\n
"""
thread_monitor Resource temporarily unavailable in pthread_create *** glibc detected *** /usr/local/MATLAB/R2015b/bin/glnxa64/MATLAB: double free or corruption (!prev): 0x00007f585308c390 *** Segmentation fault (core dumped)
"""
I have copied Matlab version info below. Any tips on how to troubleshoot?
MATLAB Version: 8.6.0.267246 (R2015b) MATLAB License Number: 1098672 Operating System: Linux 2.6.32-573.18.1.el6.x86_64 #1 SMP Wed Jan 6 11:20:49 EST 2016 x86_64 Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
MATLAB Version 8.6 (R2015b) Computer Vision System Toolbox Version 7.0 (R2015b) Image Processing Toolbox Version 9.3 (R2015b) Optimization Toolbox Version 7.3 (R2015b) Signal Processing Toolbox Version 7.1 (R2015b) Statistics and Machine Learning Toolbox Version 10.1 (R2015b)
_stephen v. shepherd phd_ The Rockefeller University / 1230 York Avenue / New York NY 10065-6307 USA // 212.327.7699
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
sorry, I have no idea
On 05/19/2016 11:35 AM, S.V.Shepherd [work] wrote:
It happens when I run more than 3-4 analyses at once. It appears to be sensitive to other instances of Matlab being open. however, I don't see clear evidence I'm memory limited. for example, I'm currently running four selxavg in parallel, along with a separate melodic analysis. One more would cause a crash.
free -g returns, in the "free" column: mem: 4 buffers/cache 480 swap 127
i think this means I have 480gb free, no?
stephen
_stephen v. shepherd phd_ The Rockefeller University / 1230 York Avenue / New York NY 10065-6307 USA // 212.327.7699
On Thu, May 19, 2016 at 11:03 AM, Douglas N Greve <greve@nmr.mgh.harvard.edu mailto:greve@nmr.mgh.harvard.edu> wrote:
Does it always die like that? It may be out of memory. On 05/18/2016 01:50 PM, S.V.Shepherd [work] wrote: > Hello, > > When running more than 3 analyses in parallel in FreeSurfer 5.3, I am > getting matlab crashes. Mathworks has politely declined to provide > support unless I can replicate outside FsFAST. > > The errors are variants of > > """ > > OMP: Error #34: System unable to allocate necessary resources for OMP > thread: > OMP: System error #11: Resource temporarily unavailable > [...] > MATLAB is exiting because of fatal error > Killed > ------------------------------------------ > ERROR: fast_selxavg3() failed\n > > """ > > thread_monitor Resource temporarily unavailable in pthread_create > *** glibc detected *** /usr/local/MATLAB/R2015b/bin/glnxa64/MATLAB: > double free or corruption (!prev): 0x00007f585308c390 *** > Segmentation fault (core dumped) > > """ > > I have copied Matlab version info below. Any tips on how to troubleshoot? > > ---------------------------------------------------------------------------------------------------- > MATLAB Version: 8.6.0.267246 (R2015b) > MATLAB License Number: 1098672 > Operating System: Linux 2.6.32-573.18.1.el6.x86_64 #1 SMP Wed Jan 6 > 11:20:49 EST 2016 x86_64 > Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java > HotSpot(TM) 64-Bit Server VM mixed mode > ---------------------------------------------------------------------------------------------------- > MATLAB Version 8.6 (R2015b) > Computer Vision System Toolbox Version 7.0 (R2015b) > Image Processing Toolbox Version 9.3 (R2015b) > Optimization Toolbox Version 7.3 (R2015b) > Signal Processing Toolbox Version 7.1 (R2015b) > Statistics and Machine Learning Toolbox Version 10.1 (R2015b) > > > > > > _stephen v. shepherd phd_ > The Rockefeller University / 1230 York Avenue / New York NY 10065-6307 > USA // 212.327.7699 <tel:212.327.7699> > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer -- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu <mailto:greve@nmr.mgh.harvard.edu> Phone Number: 617-724-2358 <tel:617-724-2358> Fax: 617-726-7422 <tel:617-726-7422> Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting <http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting> FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html <http://www.nmr.mgh.harvard.edu/facility/filedrop/index.html> Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/ _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
A follow up, in case anyone else encounters thread-related bugs: It *appears* that this error was resolved by adding the line
maxNumCompThreads(4);
to *startup.m*
stephen
*stephen v. shepherd phd* The Rockefeller University / 1230 York Avenue / New York NY 10065-6307 USA // 212.327.7699
On Thu, May 19, 2016 at 11:41 AM, Douglas N Greve <greve@nmr.mgh.harvard.edu
wrote:
sorry, I have no idea
On 05/19/2016 11:35 AM, S.V.Shepherd [work] wrote:
It happens when I run more than 3-4 analyses at once. It appears to be sensitive to other instances of Matlab being open. however, I don't see clear evidence I'm memory limited. for example, I'm currently running four selxavg in parallel, along with a separate melodic analysis. One more would cause a crash.
free -g returns, in the "free" column: mem: 4 buffers/cache 480 swap 127
i think this means I have 480gb free, no?
stephen
_stephen v. shepherd phd_ The Rockefeller University / 1230 York Avenue / New York NY 10065-6307 USA // 212.327.7699
On Thu, May 19, 2016 at 11:03 AM, Douglas N Greve <greve@nmr.mgh.harvard.edu mailto:greve@nmr.mgh.harvard.edu> wrote:
Does it always die like that? It may be out of memory. On 05/18/2016 01:50 PM, S.V.Shepherd [work] wrote: > Hello, > > When running more than 3 analyses in parallel in FreeSurfer 5.3, I am > getting matlab crashes. Mathworks has politely declined to provide > support unless I can replicate outside FsFAST. > > The errors are variants of > > """ > > OMP: Error #34: System unable to allocate necessary resources for OMP > thread: > OMP: System error #11: Resource temporarily unavailable > [...] > MATLAB is exiting because of fatal error > Killed > ------------------------------------------ > ERROR: fast_selxavg3() failed\n > > """ > > thread_monitor Resource temporarily unavailable in pthread_create > *** glibc detected *** /usr/local/MATLAB/R2015b/bin/glnxa64/MATLAB: > double free or corruption (!prev): 0x00007f585308c390 *** > Segmentation fault (core dumped) > > """ > > I have copied Matlab version info below. Any tips on how to troubleshoot? > >
> MATLAB Version: 8.6.0.267246 (R2015b) > MATLAB License Number: 1098672 > Operating System: Linux 2.6.32-573.18.1.el6.x86_64 #1 SMP Wed Jan 6 > 11:20:49 EST 2016 x86_64 > Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java > HotSpot(TM) 64-Bit Server VM mixed mode >
> MATLAB Version 8.6 (R2015b) > Computer Vision System Toolbox Version 7.0 (R2015b) > Image Processing Toolbox Version 9.3 (R2015b) > Optimization Toolbox Version 7.3 (R2015b) > Signal Processing Toolbox Version 7.1 (R2015b) > Statistics and Machine Learning Toolbox Version 10.1(R2015b)
> > > > > > _stephen v. shepherd phd_ > The Rockefeller University / 1230 York Avenue / New York NY10065-6307
> USA // 212.327.7699 <tel:212.327.7699> > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer -- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu <mailto:greve@nmr.mgh.harvard.edu> Phone Number: 617-724-2358 <tel:617-724-2358> Fax: 617-726-7422 <tel:617-726-7422> Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting <http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting> FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html <http://www.nmr.mgh.harvard.edu/facility/filedrop/index.html> Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/ _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu <mailto:Freesurfer@nmr.mgh.harvard.edu>
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
-- Douglas N. Greve, Ph.D. MGH-NMR Center greve@nmr.mgh.harvard.edu Phone Number: 617-724-2358 Fax: 617-726-7422
Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 www.nmr.mgh.harvard.edu/facility/filedrop/index.html Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
freesurfer@nmr.mgh.harvard.edu