[Mne_analysis] Python - problem with PSD if n_jobs exceeds number of available CPUs?
Jon Houck
jhouck at unm.edu
Fri May 23 19:50:10 EDT 2014
Hi Alex,
Sure, it's at https://gist.github.com/jhouck/0a61a4f420c80356d54e
Thanks,
Jon
On Fri, May 23, 2014 at 3:35 PM, Alexandre Gramfort <
alexandre.gramfort at telecom-paristech.fr> wrote:
> Hi Jon,
>
> can you share a code snippet to reproduce
> the problem? Ideally using the MNE sample
> data.
>
> Please use: https://gist.github.com/
>
> thanks
> Alex
>
>
> On Fri, May 23, 2014 at 9:34 PM, Jon Houck <jhouck at unm.edu> wrote:
> > Hi all,
> >
> > I ran into what is probably a rare error using compute_source_psd_epochs
> > with adaptive=True. For certain labels, if n_jobs is greater than the
> number
> > of CPUs, the process will crash. In my case it was n_jobs=32, for a
> > condition with 20 epochs, in G_front_inf-Orbital-rh from aparc.a2009s.
> For
> > several other labels the process ran without error. Using n_jobs =19 or
> 20
> > was fine, 21 or higher failed reliably.
> >
> > This may not affect much else. My workaround was to set n_jobs and test
> the
> > value against multiprocessing.cpu_count() early in the script. Other
> > options would be for the user to pay attention to which machine they are
> > using, or to be less greedy with CPU usage...
> >
> > The traceback is below, for those who are into that kind of thing. This
> was
> > using the master downloaded on May 19 (0.8.git).
> >
> > Jon
> >
> >
> >
> >
> >
> >
> >
> >
> > Computing PSD for offset event 8192 label G_front_inf-Orbital-rh at
> index 10
> > Considering frequencies 4 ... 20 Hz
> > Preparing the inverse operator for use...
> > Scaled noise and source covariance from nave = 1 to nave = 1
> > Created the regularized inverter
> > Created an SSP operator (subspace dimension = 7)
> > Created the whitener using a full noise covariance matrix (239 small
> > eigenvalues omitted)
> > Picked 306 channels from the data
> > Computing inverse...
> > (eigenleads need to be weighted)...
> > Reducing data rank to 21
> > Using 5 tapers with bandwidth 4.0Hz
> > Processing epoch : 1
> > [Parallel(n_jobs=32)]: Done 1 out of 32 | elapsed: 0.0s remaining:
> > 0.1s
> > [Parallel(n_jobs=32)]: Done 6 out of 32 | elapsed: 0.0s remaining:
> > 0.1s
> > [Parallel(n_jobs=32)]: Done 13 out of 32 | elapsed: 0.0s remaining:
> > 0.1s
> > [Parallel(n_jobs=32)]: Done 20 out of 32 | elapsed: 0.0s remaining:
> > 0.0s
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> >
> >
> >
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> >
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> >
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> >
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> >
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> >
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> >
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> >
> > An unexpected error occurred while tokenizing input
> > The following traceback may be corrupted or invalid
> > The error message is: ('EOF in multi-line statement', (3, 0))
> >
> > Traceback (most recent call last):
> > File "<stdin>", line 15, in <module>
> > File
> >
> "/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/minimum_norm/time_frequency.py",
> > line 572, in _compute_source_psd_epochs
> > for x in np.array_split(x_mt_src, n_jobs))
> > File
> >
> "/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py",
> > line 651, in __call__
> > self.retrieve()
> > File
> >
> "/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py",
> > line 534, in retrieve
> > raise exception_type(report)
> >
> joblib.my_exceptions.JoblibValueError/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/my_exceptions.py:26:
> > DeprecationWarning: BaseException.message has been deprecated as of
> Python
> > 2.6
> > self.message,
> > : JoblibValueError
> >
> ___________________________________________________________________________
> > Multiprocessing exception:
> >
> >
> ...........................................................................
> > /na/homes/jhouck/docs/<stdin> in <module>()
> > 10
> > 11
> > 12
> > 13
> > 14
> > ---> 15
> > 16
> > 17
> > 18
> > 19
> >
> >
> ...........................................................................
> >
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/minimum_norm/time_frequency.py
> > in _compute_source_psd_epochs(epochs=<Epochs | n_events : 20 (all
> good),
> > tmin : -1.5 (s), tmax : 0.1 (s), baseline : (None, 0)>,
> > inverse_operator={'coord_frame': array([4], dtype=int32), 'depth_prior':
> > {'bads': [], 'data': array([ 0.03416839, 0.03416839, 0.03416839, ...,
> > 0.21104597,
> > 0.21104597, 0.21104597]), 'diag': True, 'dim': 24588, 'eig':
> None,
> > 'eigvec': None, 'kind': 5, 'names': [], 'nfree': 1, 'projs': []},
> > 'eigen_fields': {'col_names': [u'MEG0113', u'MEG0112', u'MEG0111',
> > u'MEG0122', u'MEG0123', u'MEG0121', u'MEG0132', u'MEG0133', u'MEG0131',
> > u'MEG0143', u'MEG0142', u'MEG0141', u'MEG0213', u'MEG0212', u'MEG0211',
> > u'MEG0222', u'MEG0223', u'MEG0221', u'MEG0232', u'MEG0233', ...], 'data':
> > array([[ 0.00000000e+00, 6.42533777e-08, 4.....00000000e+00,
> > 0.00000000e+00]], dtype=float32), 'ncol': 306, 'nrow': 306, 'row_names':
> > []}, 'eigen_leads': {'col_names': [], 'data': array([[ -2.40336126e-03,
> > 1.22079987e-03, 7.....37850686e-05, -7.04127597e-05]], dtype=float32),
> > 'ncol': 306, 'nrow': 24588, 'row_names': []}, 'eigen_leads_weighted':
> False,
> > 'fmri_prior': None, 'info': {'bads': [], 'ch_names': ['MEG0113',
> 'MEG0112',
> > 'MEG0111', 'MEG0122', 'MEG0123', 'MEG0121', 'MEG0132', 'MEG0133',
> 'MEG0131',
> > 'MEG0143', 'MEG0142', 'MEG0141', 'MEG0213', 'MEG0212', 'MEG0211',
> 'MEG0222',
> > 'MEG0223', 'MEG0221', 'MEG0232', 'MEG0233', ...], 'chs': [{'cal':
> > 3.250000046861601e-09, 'ch_name': 'MEG0113', 'coil_trans':
> > array([[-0.01938988, -0.18108977, -0.98325199, -... , 0. ,
> 0.
> > , 1. ]]), 'coil_type': 3012, 'coord_frame': 1, 'eeg_loc': None,
> > 'kind': 1, 'loc': array([-0.1066 , 0.0464 , -0.0604 , -0...
> > 0.18097401, 0.020333 ], dtype=float32), 'logno': 113, 'range':
> > 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0112',
> > 'coil_trans': array([[-0.18108977, 0.01938988, -0.98325199, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1066 , 0.0464 ,
> > -0.0604 , -0...
> > 0.18097401, 0.020333 ], dtype=float32), 'logno': 112, 'range':
> > 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0111',
> > 'coil_trans': array([[-0.01938988, -0.18108977, -0.98325199, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3024,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1066 , 0.0464 ,
> > -0.0604 , -0...
> > 0.18097401, 0.020333 ], dtype=float32), 'logno': 111, 'range':
> > 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0122',
> > 'coil_trans': array([[ 9.03011824e-04, -2.53265113e-01, -9....00,
> > 0.00000000e+00,
> > 1.00000000e+00]]), 'coil_type': 3012, 'coord_frame': 1,
> 'eeg_loc':
> > None, 'kind': 1, 'loc': array([ -1.01999998e-01, 6.31000027e-02,
> > -2.5...2.53259987e-01, 2.97999999e-04], dtype=float32), 'logno': 122,
> > 'range': 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0123',
> > 'coil_trans': array([[ -2.53265113e-01, -9.03011824e-04, -9....00,
> > 0.00000000e+00,
> > 1.00000000e+00]]), 'coil_type': 3012, 'coord_frame': 1,
> 'eeg_loc':
> > None, 'kind': 1, 'loc': array([ -1.01999998e-01, 6.31000027e-02,
> > -2.5...2.53259987e-01, 2.97999999e-04], dtype=float32), 'logno': 123,
> > 'range': 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0121',
> > 'coil_trans': array([[ 9.03011824e-04, -2.53265113e-01, -9....00,
> > 0.00000000e+00,
> > 1.00000000e+00]]), 'coil_type': 3024, 'coord_frame': 1,
> 'eeg_loc':
> > None, 'kind': 1, 'loc': array([ -1.01999998e-01, 6.31000027e-02,
> > -2.5...2.53259987e-01, 2.97999999e-04], dtype=float32), 'logno': 121,
> > 'range': 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0132',
> > 'coil_trans': array([[ 0.00592455, -0.07752677, -0.99696398, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085 , 0.0302 ,
> > -0.0266 , 0...
> > 0.077537 , -0.005751 ], dtype=float32), 'logno': 132, 'range':
> > 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0133',
> > 'coil_trans': array([[-0.07752677, -0.00592455, -0.99696398, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085 , 0.0302 ,
> > -0.0266 , -0...
> > 0.077537 , -0.005751 ], dtype=float32), 'logno': 133, 'range':
> > 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0131',
> > 'coil_trans': array([[ 0.00592455, -0.07752677, -0.99696398, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3024,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085 , 0.0302 ,
> > -0.0266 , 0...
> > 0.077537 , -0.005751 ], dtype=float32), 'logno': 131, 'range':
> > 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0143',
> > 'coil_trans': array([[-0.05253163, -0.03200371, -0.99809802, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099 , 0.0131 ,
> > -0.0627 , -0...
> > 0.031892 , 0.052595 ], dtype=float32), 'logno': 143, 'range':
> > 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0142',
> > 'coil_trans': array([[-0.03200371, 0.05253163, -0.99809802, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099 , 0.0131 ,
> > -0.0627 , -0...
> > 0.031892 , 0.052595 ], dtype=float32), 'logno': 142, 'range':
> > 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0141',
> > 'coil_trans': array([[-0.05253163, -0.03200371, -0.99809802, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3024,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099 , 0.0131 ,
> > -0.0627 , -0...
> > 0.031892 , 0.052595 ], dtype=float32), 'logno': 141, 'range':
> > 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0213',
> > 'coil_trans': array([[-0.05449655, -0.09574967, -0.99390799, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074 , 0.0329 ,
> > 0.008 , -0...
> > 0.095467 , 0.055009 ], dtype=float32), 'logno': 213, 'range':
> > 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0212',
> > 'coil_trans': array([[-0.09574967, 0.05449655, -0.99390799, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074 , 0.0329 ,
> > 0.008 , -0...
> > 0.095467 , 0.055009 ], dtype=float32), 'logno': 212, 'range':
> > 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0211',
> > 'coil_trans': array([[-0.05449655, -0.09574967, -0.99390799, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3024,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074 , 0.0329 ,
> > 0.008 , -0...
> > 0.095467 , 0.055009 ], dtype=float32), 'logno': 211, 'range':
> > 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0222',
> > 'coil_trans': array([[-0.34224641, -0.15804897, -0.92622799, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989 , 0.0403 ,
> > 0.0413 , -0...
> > 0.14645299, 0.34734601], dtype=float32), 'logno': 222, 'range':
> > 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0223',
> > 'coil_trans': array([[-0.15804897, 0.34224641, -0.92622799, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989 , 0.0403 ,
> > 0.0413 , -0...
> > 0.14645299, 0.34734601], dtype=float32), 'logno': 223, 'range':
> > 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0221',
> > 'coil_trans': array([[-0.34224641, -0.15804897, -0.92622799, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3024,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989 , 0.0403 ,
> > 0.0413 , -0...
> > 0.14645299, 0.34734601], dtype=float32), 'logno': 221, 'range':
> > 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0232',
> > 'coil_trans': array([[-0.34559116, 0.04283677, -0.93739903, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1011 , 0.0044 ,
> > 0.0408 , -0...
> > -0.048961 , 0.34478199], dtype=float32), 'logno': 232, 'range':
> > 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0233',
> > 'coil_trans': array([[ 0.04283677, 0.34559116, -0.93739903, -... ,
> > 0. , 0. , 1. ]]), 'coil_type': 3012,
> 'coord_frame':
> > 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1011 , 0.0044 ,
> > 0.0408 , 0...
> > -0.048961 , 0.34478199], dtype=float32), 'logno': 233, 'range':
> > 1.0, ...}, ...], 'dev_head_t': {'from': 1, 'to': 4, 'trans': array([[
> > 0.99667621, -0.07383786, 0.03442231, ... , 0. , 0.
> ,
> > 1. ]])}, 'meas_file':
> > u'M87110374_hallucinations_raw_all_tsss_mc_trans.fif', 'meas_id': None,
> > 'mri_file':
> > u'./M87110374_hallucinations_raw_all_tsss_mc_trans-ave-trans.fif',
> > 'mri_head_t': {'from': 5, 'to': 4, 'trans': array([[ 0.98450035,
> > -0.13287687, -0.11446673, ... , 0. , 0. , 1.
> > ]])}, 'mri_id': None, 'nchan': 306}, 'methods': 1, 'mri_head_t':
> {'from': 5,
> > 'to': 4, 'trans': array([[ 0.98450035, -0.13287687, -0.11446673, ...
> > , 0. , 0. , 1. ]])}, 'nave': 1, ...},
> lambda2=1.0,
> > method='MNE', fmin=4.0, fmax=20.0, pick_ori=None, label=<Label |
> > M87110374, u'G_front_inf-Orbital-rh', rh : 298 vertices>, nave=1,
> pca=True,
> > inv_split=None, bandwidth=4.0, adaptive=True, low_bias=True, n_jobs=32,
> > verbose=None)
> > 567 x_mt_src[:, i, :] = np.dot(K_part, x_mt[:, i, :])
> > 568
> > 569 # compute the psd
> > 570 if adaptive:
> > 571 out = parallel(my_psd_from_mt_adaptive(x,
> eigvals,
> > freq_mask)
> > --> 572 for x in np.array_split(x_mt_src, n_jobs))
> > x_mt_src = array([[[ -6.20846446e-11 +0.00000000e+00j,
> > ...j,
> > 1.28781173e-11 +1.14432688e-13j]]])
> > n_jobs = 32
> > 573 this_psd = np.concatenate(out)
> > 574 else:
> > 575 x_mt_src = x_mt_src[:, :, freq_mask]
> > 576 this_psd = _psd_from_mt(x_mt_src, weights)
> >
> >
> ...........................................................................
> >
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py
> > in __call__(self=Parallel(n_jobs=32), iterable=<generator object
> <genexpr>>)
> > 646 if pre_dispatch == "all" or n_jobs == 1:
> > 647 # The iterable was consumed all at once by the
> above
> > for loop.
> > 648 # No need to wait for async callbacks to trigger
> to
> > 649 # consumption.
> > 650 self._iterating = False
> > --> 651 self.retrieve()
> > self.retrieve = <bound method Parallel.retrieve of
> > Parallel(n_jobs=32)>
> > 652 # Make sure that we get a last message telling us we
> are
> > done
> > 653 elapsed_time = time.time() - self._start_time
> > 654 self._print('Done %3i out of %3i | elapsed: %s
> > finished',
> > 655 (len(self._output),
> >
> >
> >
> ---------------------------------------------------------------------------
> > Sub-process traceback:
> >
> >
> ---------------------------------------------------------------------------
> > ValueError Fri May 23
> 12:27:35
> > 2014
> > PID: 30246Python 2.7.6:
> >
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/bin/python
> >
> ...........................................................................
> >
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/time_frequency/multitaper.pyc
> > in _psd_from_mt_adaptive(x_mt=array([], dtype=float64), eigvals=array([
> > 0.99999996, 0.99999714, 0.99990416, 0.99810608, 0.97662124]),
> > freq_mask=array([False, False, False, False, False, False,... False,
> False,
> > False, False, False], dtype=bool), max_iter=150, return_weights=False)
> > 277 -----
> > 278
> > 279 The weights to use for making the multitaper estimate, such
> that
> > 280 :math:`S_{mt} = \sum_{k} |w_k|^2S_k^{mt} / \sum_{k} |w_k|^2`
> > 281 """
> > --> 282 n_signals, n_tapers, n_freqs = x_mt.shape
> > 283
> > 284 if len(eigvals) != n_tapers:
> > 285 raise ValueError('Need one eigenvalue for each taper')
> > 286
> >
> > ValueError: need more than 1 value to unpack
> >
> >
> >
> > _______________________________________________
> > Mne_analysis mailing list
> > Mne_analysis at nmr.mgh.harvard.edu
> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> >
> >
> > The information in this e-mail is intended only for the person to whom
> it is
> > addressed. If you believe this e-mail was sent to you in error and the
> > e-mail
> > contains patient information, please contact the Partners Compliance
> > HelpLine at
> > http://www.partners.org/complianceline . If the e-mail was sent to you
> in
> > error
> > but does not contain patient information, please contact the sender and
> > properly
> > dispose of the e-mail.
> >
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20140523/86230f65/attachment.html
More information about the Mne_analysis
mailing list