[Mne_analysis] Python - problem with PSD if n_jobs exceeds number of available CPUs?
Alexandre Gramfort
alexandre.gramfort at telecom-paristech.fr
Fri May 23 17:35:53 EDT 2014
Hi Jon,
can you share a code snippet to reproduce
the problem? Ideally using the MNE sample
data.
Please use: https://gist.github.com/
thanks
Alex
On Fri, May 23, 2014 at 9:34 PM, Jon Houck <jhouck at unm.edu> wrote:
> Hi all,
>
> I ran into what is probably a rare error using compute_source_psd_epochs
> with adaptive=True. For certain labels, if n_jobs is greater than the number
> of CPUs, the process will crash. In my case it was n_jobs=32, for a
> condition with 20 epochs, in G_front_inf-Orbital-rh from aparc.a2009s. For
> several other labels the process ran without error. Using n_jobs =19 or 20
> was fine, 21 or higher failed reliably.
>
> This may not affect much else. My workaround was to set n_jobs and test the
> value against multiprocessing.cpu_count() early in the script. Other
> options would be for the user to pay attention to which machine they are
> using, or to be less greedy with CPU usage...
>
> The traceback is below, for those who are into that kind of thing. This was
> using the master downloaded on May 19 (0.8.git).
>
> Jon
>
>
>
>
>
>
>
>
> Computing PSD for offset event 8192 label G_front_inf-Orbital-rh at index 10
> Considering frequencies 4 ... 20 Hz
> Preparing the inverse operator for use...
> Scaled noise and source covariance from nave = 1 to nave = 1
> Created the regularized inverter
> Created an SSP operator (subspace dimension = 7)
> Created the whitener using a full noise covariance matrix (239 small
> eigenvalues omitted)
> Picked 306 channels from the data
> Computing inverse...
> (eigenleads need to be weighted)...
> Reducing data rank to 21
> Using 5 tapers with bandwidth 4.0Hz
> Processing epoch : 1
> [Parallel(n_jobs=32)]: Done 1 out of 32 | elapsed: 0.0s remaining:
> 0.1s
> [Parallel(n_jobs=32)]: Done 6 out of 32 | elapsed: 0.0s remaining:
> 0.1s
> [Parallel(n_jobs=32)]: Done 13 out of 32 | elapsed: 0.0s remaining:
> 0.1s
> [Parallel(n_jobs=32)]: Done 20 out of 32 | elapsed: 0.0s remaining:
> 0.0s
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
>
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> Traceback (most recent call last):
> File "<stdin>", line 15, in <module>
> File
> "/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/minimum_norm/time_frequency.py",
> line 572, in _compute_source_psd_epochs
> for x in np.array_split(x_mt_src, n_jobs))
> File
> "/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py",
> line 651, in __call__
> self.retrieve()
> File
> "/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py",
> line 534, in retrieve
> raise exception_type(report)
> joblib.my_exceptions.JoblibValueError/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/my_exceptions.py:26:
> DeprecationWarning: BaseException.message has been deprecated as of Python
> 2.6
> self.message,
> : JoblibValueError
> ___________________________________________________________________________
> Multiprocessing exception:
>
> ...........................................................................
> /na/homes/jhouck/docs/<stdin> in <module>()
> 10
> 11
> 12
> 13
> 14
> ---> 15
> 16
> 17
> 18
> 19
>
> ...........................................................................
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/minimum_norm/time_frequency.py
> in _compute_source_psd_epochs(epochs=<Epochs | n_events : 20 (all good),
> tmin : -1.5 (s), tmax : 0.1 (s), baseline : (None, 0)>,
> inverse_operator={'coord_frame': array([4], dtype=int32), 'depth_prior':
> {'bads': [], 'data': array([ 0.03416839, 0.03416839, 0.03416839, ...,
> 0.21104597,
> 0.21104597, 0.21104597]), 'diag': True, 'dim': 24588, 'eig': None,
> 'eigvec': None, 'kind': 5, 'names': [], 'nfree': 1, 'projs': []},
> 'eigen_fields': {'col_names': [u'MEG0113', u'MEG0112', u'MEG0111',
> u'MEG0122', u'MEG0123', u'MEG0121', u'MEG0132', u'MEG0133', u'MEG0131',
> u'MEG0143', u'MEG0142', u'MEG0141', u'MEG0213', u'MEG0212', u'MEG0211',
> u'MEG0222', u'MEG0223', u'MEG0221', u'MEG0232', u'MEG0233', ...], 'data':
> array([[ 0.00000000e+00, 6.42533777e-08, 4.....00000000e+00,
> 0.00000000e+00]], dtype=float32), 'ncol': 306, 'nrow': 306, 'row_names':
> []}, 'eigen_leads': {'col_names': [], 'data': array([[ -2.40336126e-03,
> 1.22079987e-03, 7.....37850686e-05, -7.04127597e-05]], dtype=float32),
> 'ncol': 306, 'nrow': 24588, 'row_names': []}, 'eigen_leads_weighted': False,
> 'fmri_prior': None, 'info': {'bads': [], 'ch_names': ['MEG0113', 'MEG0112',
> 'MEG0111', 'MEG0122', 'MEG0123', 'MEG0121', 'MEG0132', 'MEG0133', 'MEG0131',
> 'MEG0143', 'MEG0142', 'MEG0141', 'MEG0213', 'MEG0212', 'MEG0211', 'MEG0222',
> 'MEG0223', 'MEG0221', 'MEG0232', 'MEG0233', ...], 'chs': [{'cal':
> 3.250000046861601e-09, 'ch_name': 'MEG0113', 'coil_trans':
> array([[-0.01938988, -0.18108977, -0.98325199, -... , 0. , 0.
> , 1. ]]), 'coil_type': 3012, 'coord_frame': 1, 'eeg_loc': None,
> 'kind': 1, 'loc': array([-0.1066 , 0.0464 , -0.0604 , -0...
> 0.18097401, 0.020333 ], dtype=float32), 'logno': 113, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0112',
> 'coil_trans': array([[-0.18108977, 0.01938988, -0.98325199, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1066 , 0.0464 ,
> -0.0604 , -0...
> 0.18097401, 0.020333 ], dtype=float32), 'logno': 112, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0111',
> 'coil_trans': array([[-0.01938988, -0.18108977, -0.98325199, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1066 , 0.0464 ,
> -0.0604 , -0...
> 0.18097401, 0.020333 ], dtype=float32), 'logno': 111, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0122',
> 'coil_trans': array([[ 9.03011824e-04, -2.53265113e-01, -9....00,
> 0.00000000e+00,
> 1.00000000e+00]]), 'coil_type': 3012, 'coord_frame': 1, 'eeg_loc':
> None, 'kind': 1, 'loc': array([ -1.01999998e-01, 6.31000027e-02,
> -2.5...2.53259987e-01, 2.97999999e-04], dtype=float32), 'logno': 122,
> 'range': 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0123',
> 'coil_trans': array([[ -2.53265113e-01, -9.03011824e-04, -9....00,
> 0.00000000e+00,
> 1.00000000e+00]]), 'coil_type': 3012, 'coord_frame': 1, 'eeg_loc':
> None, 'kind': 1, 'loc': array([ -1.01999998e-01, 6.31000027e-02,
> -2.5...2.53259987e-01, 2.97999999e-04], dtype=float32), 'logno': 123,
> 'range': 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0121',
> 'coil_trans': array([[ 9.03011824e-04, -2.53265113e-01, -9....00,
> 0.00000000e+00,
> 1.00000000e+00]]), 'coil_type': 3024, 'coord_frame': 1, 'eeg_loc':
> None, 'kind': 1, 'loc': array([ -1.01999998e-01, 6.31000027e-02,
> -2.5...2.53259987e-01, 2.97999999e-04], dtype=float32), 'logno': 121,
> 'range': 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0132',
> 'coil_trans': array([[ 0.00592455, -0.07752677, -0.99696398, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085 , 0.0302 ,
> -0.0266 , 0...
> 0.077537 , -0.005751 ], dtype=float32), 'logno': 132, 'range':
> 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0133',
> 'coil_trans': array([[-0.07752677, -0.00592455, -0.99696398, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085 , 0.0302 ,
> -0.0266 , -0...
> 0.077537 , -0.005751 ], dtype=float32), 'logno': 133, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0131',
> 'coil_trans': array([[ 0.00592455, -0.07752677, -0.99696398, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085 , 0.0302 ,
> -0.0266 , 0...
> 0.077537 , -0.005751 ], dtype=float32), 'logno': 131, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0143',
> 'coil_trans': array([[-0.05253163, -0.03200371, -0.99809802, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099 , 0.0131 ,
> -0.0627 , -0...
> 0.031892 , 0.052595 ], dtype=float32), 'logno': 143, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0142',
> 'coil_trans': array([[-0.03200371, 0.05253163, -0.99809802, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099 , 0.0131 ,
> -0.0627 , -0...
> 0.031892 , 0.052595 ], dtype=float32), 'logno': 142, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0141',
> 'coil_trans': array([[-0.05253163, -0.03200371, -0.99809802, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099 , 0.0131 ,
> -0.0627 , -0...
> 0.031892 , 0.052595 ], dtype=float32), 'logno': 141, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0213',
> 'coil_trans': array([[-0.05449655, -0.09574967, -0.99390799, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074 , 0.0329 ,
> 0.008 , -0...
> 0.095467 , 0.055009 ], dtype=float32), 'logno': 213, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0212',
> 'coil_trans': array([[-0.09574967, 0.05449655, -0.99390799, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074 , 0.0329 ,
> 0.008 , -0...
> 0.095467 , 0.055009 ], dtype=float32), 'logno': 212, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0211',
> 'coil_trans': array([[-0.05449655, -0.09574967, -0.99390799, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074 , 0.0329 ,
> 0.008 , -0...
> 0.095467 , 0.055009 ], dtype=float32), 'logno': 211, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0222',
> 'coil_trans': array([[-0.34224641, -0.15804897, -0.92622799, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989 , 0.0403 ,
> 0.0413 , -0...
> 0.14645299, 0.34734601], dtype=float32), 'logno': 222, 'range':
> 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0223',
> 'coil_trans': array([[-0.15804897, 0.34224641, -0.92622799, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989 , 0.0403 ,
> 0.0413 , -0...
> 0.14645299, 0.34734601], dtype=float32), 'logno': 223, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0221',
> 'coil_trans': array([[-0.34224641, -0.15804897, -0.92622799, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989 , 0.0403 ,
> 0.0413 , -0...
> 0.14645299, 0.34734601], dtype=float32), 'logno': 221, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0232',
> 'coil_trans': array([[-0.34559116, 0.04283677, -0.93739903, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1011 , 0.0044 ,
> 0.0408 , -0...
> -0.048961 , 0.34478199], dtype=float32), 'logno': 232, 'range':
> 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0233',
> 'coil_trans': array([[ 0.04283677, 0.34559116, -0.93739903, -... ,
> 0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1011 , 0.0044 ,
> 0.0408 , 0...
> -0.048961 , 0.34478199], dtype=float32), 'logno': 233, 'range':
> 1.0, ...}, ...], 'dev_head_t': {'from': 1, 'to': 4, 'trans': array([[
> 0.99667621, -0.07383786, 0.03442231, ... , 0. , 0. ,
> 1. ]])}, 'meas_file':
> u'M87110374_hallucinations_raw_all_tsss_mc_trans.fif', 'meas_id': None,
> 'mri_file':
> u'./M87110374_hallucinations_raw_all_tsss_mc_trans-ave-trans.fif',
> 'mri_head_t': {'from': 5, 'to': 4, 'trans': array([[ 0.98450035,
> -0.13287687, -0.11446673, ... , 0. , 0. , 1.
> ]])}, 'mri_id': None, 'nchan': 306}, 'methods': 1, 'mri_head_t': {'from': 5,
> 'to': 4, 'trans': array([[ 0.98450035, -0.13287687, -0.11446673, ...
> , 0. , 0. , 1. ]])}, 'nave': 1, ...}, lambda2=1.0,
> method='MNE', fmin=4.0, fmax=20.0, pick_ori=None, label=<Label |
> M87110374, u'G_front_inf-Orbital-rh', rh : 298 vertices>, nave=1, pca=True,
> inv_split=None, bandwidth=4.0, adaptive=True, low_bias=True, n_jobs=32,
> verbose=None)
> 567 x_mt_src[:, i, :] = np.dot(K_part, x_mt[:, i, :])
> 568
> 569 # compute the psd
> 570 if adaptive:
> 571 out = parallel(my_psd_from_mt_adaptive(x, eigvals,
> freq_mask)
> --> 572 for x in np.array_split(x_mt_src, n_jobs))
> x_mt_src = array([[[ -6.20846446e-11 +0.00000000e+00j,
> ...j,
> 1.28781173e-11 +1.14432688e-13j]]])
> n_jobs = 32
> 573 this_psd = np.concatenate(out)
> 574 else:
> 575 x_mt_src = x_mt_src[:, :, freq_mask]
> 576 this_psd = _psd_from_mt(x_mt_src, weights)
>
> ...........................................................................
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py
> in __call__(self=Parallel(n_jobs=32), iterable=<generator object <genexpr>>)
> 646 if pre_dispatch == "all" or n_jobs == 1:
> 647 # The iterable was consumed all at once by the above
> for loop.
> 648 # No need to wait for async callbacks to trigger to
> 649 # consumption.
> 650 self._iterating = False
> --> 651 self.retrieve()
> self.retrieve = <bound method Parallel.retrieve of
> Parallel(n_jobs=32)>
> 652 # Make sure that we get a last message telling us we are
> done
> 653 elapsed_time = time.time() - self._start_time
> 654 self._print('Done %3i out of %3i | elapsed: %s
> finished',
> 655 (len(self._output),
>
>
> ---------------------------------------------------------------------------
> Sub-process traceback:
>
> ---------------------------------------------------------------------------
> ValueError Fri May 23 12:27:35
> 2014
> PID: 30246Python 2.7.6:
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/bin/python
> ...........................................................................
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/time_frequency/multitaper.pyc
> in _psd_from_mt_adaptive(x_mt=array([], dtype=float64), eigvals=array([
> 0.99999996, 0.99999714, 0.99990416, 0.99810608, 0.97662124]),
> freq_mask=array([False, False, False, False, False, False,... False, False,
> False, False, False], dtype=bool), max_iter=150, return_weights=False)
> 277 -----
> 278
> 279 The weights to use for making the multitaper estimate, such that
> 280 :math:`S_{mt} = \sum_{k} |w_k|^2S_k^{mt} / \sum_{k} |w_k|^2`
> 281 """
> --> 282 n_signals, n_tapers, n_freqs = x_mt.shape
> 283
> 284 if len(eigvals) != n_tapers:
> 285 raise ValueError('Need one eigenvalue for each taper')
> 286
>
> ValueError: need more than 1 value to unpack
>
>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
>
More information about the Mne_analysis
mailing list