[Mne_analysis] Python - problem with PSD if n_jobs exceeds number of available CPUs?

Alexandre Gramfort alexandre.gramfort at telecom-paristech.fr
Fri May 23 17:35:53 EDT 2014
Search archives:

Hi Jon,

can you share a code snippet to reproduce
the problem? Ideally using the MNE sample
data.

Please use: https://gist.github.com/

thanks
Alex


On Fri, May 23, 2014 at 9:34 PM, Jon Houck <jhouck at unm.edu> wrote:
> Hi all,
>
> I ran into what is probably a rare error using compute_source_psd_epochs
> with adaptive=True. For certain labels, if n_jobs is greater than the number
> of CPUs, the process will crash.  In my case it was n_jobs=32, for a
> condition with 20 epochs, in G_front_inf-Orbital-rh from aparc.a2009s.  For
> several other labels the process ran without error.  Using n_jobs =19 or 20
> was fine, 21 or higher failed reliably.
>
> This may not affect much else.  My workaround was to set n_jobs and test the
> value against multiprocessing.cpu_count() early in the script.  Other
> options would be for the user to pay attention to which machine they are
> using, or to be less greedy with CPU usage...
>
> The traceback is below, for those who are into that kind of thing. This was
> using the master downloaded on May 19 (0.8.git).
>
> Jon
>
>
>
>
>
>
>
>
> Computing PSD for offset event 8192 label G_front_inf-Orbital-rh at index 10
> Considering frequencies 4 ... 20 Hz
> Preparing the inverse operator for use...
>     Scaled noise and source covariance from nave = 1 to nave = 1
>     Created the regularized inverter
>     Created an SSP operator (subspace dimension = 7)
>     Created the whitener using a full noise covariance matrix (239 small
> eigenvalues omitted)
> Picked 306 channels from the data
> Computing inverse...
> (eigenleads need to be weighted)...
> Reducing data rank to 21
> Using 5 tapers with bandwidth 4.0Hz
> Processing epoch : 1
> [Parallel(n_jobs=32)]: Done   1 out of  32 | elapsed:    0.0s remaining:
> 0.1s
> [Parallel(n_jobs=32)]: Done   6 out of  32 | elapsed:    0.0s remaining:
> 0.1s
> [Parallel(n_jobs=32)]: Done  13 out of  32 | elapsed:    0.0s remaining:
> 0.1s
> [Parallel(n_jobs=32)]: Done  20 out of  32 | elapsed:    0.0s remaining:
> 0.0s
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
>
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> An unexpected error occurred while tokenizing input
> The following traceback may be corrupted or invalid
> The error message is: ('EOF in multi-line statement', (3, 0))
>
> Traceback (most recent call last):
>   File "<stdin>", line 15, in <module>
>   File
> "/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/minimum_norm/time_frequency.py",
> line 572, in _compute_source_psd_epochs
>     for x in np.array_split(x_mt_src, n_jobs))
>   File
> "/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py",
> line 651, in __call__
>     self.retrieve()
>   File
> "/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py",
> line 534, in retrieve
>     raise exception_type(report)
> joblib.my_exceptions.JoblibValueError/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/my_exceptions.py:26:
> DeprecationWarning: BaseException.message has been deprecated as of Python
> 2.6
>   self.message,
> : JoblibValueError
> ___________________________________________________________________________
> Multiprocessing exception:
>
> ...........................................................................
> /na/homes/jhouck/docs/<stdin> in <module>()
>      10
>      11
>      12
>      13
>      14
> ---> 15
>      16
>      17
>      18
>      19
>
> ...........................................................................
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/minimum_norm/time_frequency.py
> in _compute_source_psd_epochs(epochs=<Epochs  |  n_events : 20 (all good),
> tmin : -1.5 (s), tmax : 0.1 (s), baseline : (None, 0)>,
> inverse_operator={'coord_frame': array([4], dtype=int32), 'depth_prior':
> {'bads': [], 'data': array([ 0.03416839,  0.03416839,  0.03416839, ...,
> 0.21104597,
>         0.21104597,  0.21104597]), 'diag': True, 'dim': 24588, 'eig': None,
> 'eigvec': None, 'kind': 5, 'names': [], 'nfree': 1, 'projs': []},
> 'eigen_fields': {'col_names': [u'MEG0113', u'MEG0112', u'MEG0111',
> u'MEG0122', u'MEG0123', u'MEG0121', u'MEG0132', u'MEG0133', u'MEG0131',
> u'MEG0143', u'MEG0142', u'MEG0141', u'MEG0213', u'MEG0212', u'MEG0211',
> u'MEG0222', u'MEG0223', u'MEG0221', u'MEG0232', u'MEG0233', ...], 'data':
> array([[  0.00000000e+00,   6.42533777e-08,   4.....00000000e+00,
> 0.00000000e+00]], dtype=float32), 'ncol': 306, 'nrow': 306, 'row_names':
> []}, 'eigen_leads': {'col_names': [], 'data': array([[ -2.40336126e-03,
> 1.22079987e-03,   7.....37850686e-05,  -7.04127597e-05]], dtype=float32),
> 'ncol': 306, 'nrow': 24588, 'row_names': []}, 'eigen_leads_weighted': False,
> 'fmri_prior': None, 'info': {'bads': [], 'ch_names': ['MEG0113', 'MEG0112',
> 'MEG0111', 'MEG0122', 'MEG0123', 'MEG0121', 'MEG0132', 'MEG0133', 'MEG0131',
> 'MEG0143', 'MEG0142', 'MEG0141', 'MEG0213', 'MEG0212', 'MEG0211', 'MEG0222',
> 'MEG0223', 'MEG0221', 'MEG0232', 'MEG0233', ...], 'chs': [{'cal':
> 3.250000046861601e-09, 'ch_name': 'MEG0113', 'coil_trans':
> array([[-0.01938988, -0.18108977, -0.98325199, -...       ,  0.        ,  0.
> ,  1.        ]]), 'coil_type': 3012, 'coord_frame': 1, 'eeg_loc': None,
> 'kind': 1, 'loc': array([-0.1066    ,  0.0464    , -0.0604    , -0...
>         0.18097401,  0.020333  ], dtype=float32), 'logno': 113, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0112',
> 'coil_trans': array([[-0.18108977,  0.01938988, -0.98325199, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1066    ,  0.0464    ,
> -0.0604    , -0...
>         0.18097401,  0.020333  ], dtype=float32), 'logno': 112, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0111',
> 'coil_trans': array([[-0.01938988, -0.18108977, -0.98325199, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1066    ,  0.0464    ,
> -0.0604    , -0...
>         0.18097401,  0.020333  ], dtype=float32), 'logno': 111, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0122',
> 'coil_trans': array([[  9.03011824e-04,  -2.53265113e-01,  -9....00,
> 0.00000000e+00,
>           1.00000000e+00]]), 'coil_type': 3012, 'coord_frame': 1, 'eeg_loc':
> None, 'kind': 1, 'loc': array([ -1.01999998e-01,   6.31000027e-02,
> -2.5...2.53259987e-01,   2.97999999e-04], dtype=float32), 'logno': 122,
> 'range': 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0123',
> 'coil_trans': array([[ -2.53265113e-01,  -9.03011824e-04,  -9....00,
> 0.00000000e+00,
>           1.00000000e+00]]), 'coil_type': 3012, 'coord_frame': 1, 'eeg_loc':
> None, 'kind': 1, 'loc': array([ -1.01999998e-01,   6.31000027e-02,
> -2.5...2.53259987e-01,   2.97999999e-04], dtype=float32), 'logno': 123,
> 'range': 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0121',
> 'coil_trans': array([[  9.03011824e-04,  -2.53265113e-01,  -9....00,
> 0.00000000e+00,
>           1.00000000e+00]]), 'coil_type': 3024, 'coord_frame': 1, 'eeg_loc':
> None, 'kind': 1, 'loc': array([ -1.01999998e-01,   6.31000027e-02,
> -2.5...2.53259987e-01,   2.97999999e-04], dtype=float32), 'logno': 121,
> 'range': 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0132',
> 'coil_trans': array([[ 0.00592455, -0.07752677, -0.99696398, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085    ,  0.0302    ,
> -0.0266    ,  0...
>         0.077537  , -0.005751  ], dtype=float32), 'logno': 132, 'range':
> 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0133',
> 'coil_trans': array([[-0.07752677, -0.00592455, -0.99696398, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085    ,  0.0302    ,
> -0.0266    , -0...
>         0.077537  , -0.005751  ], dtype=float32), 'logno': 133, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0131',
> 'coil_trans': array([[ 0.00592455, -0.07752677, -0.99696398, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085    ,  0.0302    ,
> -0.0266    ,  0...
>         0.077537  , -0.005751  ], dtype=float32), 'logno': 131, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0143',
> 'coil_trans': array([[-0.05253163, -0.03200371, -0.99809802, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099    ,  0.0131    ,
> -0.0627    , -0...
>         0.031892  ,  0.052595  ], dtype=float32), 'logno': 143, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0142',
> 'coil_trans': array([[-0.03200371,  0.05253163, -0.99809802, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099    ,  0.0131    ,
> -0.0627    , -0...
>         0.031892  ,  0.052595  ], dtype=float32), 'logno': 142, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0141',
> 'coil_trans': array([[-0.05253163, -0.03200371, -0.99809802, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099    ,  0.0131    ,
> -0.0627    , -0...
>         0.031892  ,  0.052595  ], dtype=float32), 'logno': 141, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0213',
> 'coil_trans': array([[-0.05449655, -0.09574967, -0.99390799, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074    ,  0.0329    ,
> 0.008     , -0...
>         0.095467  ,  0.055009  ], dtype=float32), 'logno': 213, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0212',
> 'coil_trans': array([[-0.09574967,  0.05449655, -0.99390799, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074    ,  0.0329    ,
> 0.008     , -0...
>         0.095467  ,  0.055009  ], dtype=float32), 'logno': 212, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0211',
> 'coil_trans': array([[-0.05449655, -0.09574967, -0.99390799, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074    ,  0.0329    ,
> 0.008     , -0...
>         0.095467  ,  0.055009  ], dtype=float32), 'logno': 211, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0222',
> 'coil_trans': array([[-0.34224641, -0.15804897, -0.92622799, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989    ,  0.0403    ,
> 0.0413    , -0...
>         0.14645299,  0.34734601], dtype=float32), 'logno': 222, 'range':
> 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0223',
> 'coil_trans': array([[-0.15804897,  0.34224641, -0.92622799, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989    ,  0.0403    ,
> 0.0413    , -0...
>         0.14645299,  0.34734601], dtype=float32), 'logno': 223, 'range':
> 1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0221',
> 'coil_trans': array([[-0.34224641, -0.15804897, -0.92622799, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3024, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989    ,  0.0403    ,
> 0.0413    , -0...
>         0.14645299,  0.34734601], dtype=float32), 'logno': 221, 'range':
> 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0232',
> 'coil_trans': array([[-0.34559116,  0.04283677, -0.93739903, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1011    ,  0.0044    ,
> 0.0408    , -0...
>        -0.048961  ,  0.34478199], dtype=float32), 'logno': 232, 'range':
> 1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0233',
> 'coil_trans': array([[ 0.04283677,  0.34559116, -0.93739903, -...       ,
> 0.        ,  0.        ,  1.        ]]), 'coil_type': 3012, 'coord_frame':
> 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1011    ,  0.0044    ,
> 0.0408    ,  0...
>        -0.048961  ,  0.34478199], dtype=float32), 'logno': 233, 'range':
> 1.0, ...}, ...], 'dev_head_t': {'from': 1, 'to': 4, 'trans': array([[
> 0.99667621, -0.07383786,  0.03442231,  ...       ,  0.        ,  0.        ,
> 1.        ]])}, 'meas_file':
> u'M87110374_hallucinations_raw_all_tsss_mc_trans.fif', 'meas_id': None,
> 'mri_file':
> u'./M87110374_hallucinations_raw_all_tsss_mc_trans-ave-trans.fif',
> 'mri_head_t': {'from': 5, 'to': 4, 'trans': array([[ 0.98450035,
> -0.13287687, -0.11446673,  ...       ,  0.        ,  0.        ,  1.
> ]])}, 'mri_id': None, 'nchan': 306}, 'methods': 1, 'mri_head_t': {'from': 5,
> 'to': 4, 'trans': array([[ 0.98450035, -0.13287687, -0.11446673,  ...
> ,  0.        ,  0.        ,  1.        ]])}, 'nave': 1, ...}, lambda2=1.0,
> method='MNE', fmin=4.0, fmax=20.0, pick_ori=None, label=<Label  |
> M87110374, u'G_front_inf-Orbital-rh', rh : 298 vertices>, nave=1, pca=True,
> inv_split=None, bandwidth=4.0, adaptive=True, low_bias=True, n_jobs=32,
> verbose=None)
>     567                 x_mt_src[:, i, :] = np.dot(K_part, x_mt[:, i, :])
>     568
>     569             # compute the psd
>     570             if adaptive:
>     571                 out = parallel(my_psd_from_mt_adaptive(x, eigvals,
> freq_mask)
> --> 572                        for x in np.array_split(x_mt_src, n_jobs))
>         x_mt_src = array([[[ -6.20846446e-11 +0.00000000e+00j,
>     ...j,
>            1.28781173e-11 +1.14432688e-13j]]])
>         n_jobs = 32
>     573                 this_psd = np.concatenate(out)
>     574             else:
>     575                 x_mt_src = x_mt_src[:, :, freq_mask]
>     576                 this_psd = _psd_from_mt(x_mt_src, weights)
>
> ...........................................................................
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py
> in __call__(self=Parallel(n_jobs=32), iterable=<generator object <genexpr>>)
>     646             if pre_dispatch == "all" or n_jobs == 1:
>     647                 # The iterable was consumed all at once by the above
> for loop.
>     648                 # No need to wait for async callbacks to trigger to
>     649                 # consumption.
>     650                 self._iterating = False
> --> 651             self.retrieve()
>         self.retrieve = <bound method Parallel.retrieve of
> Parallel(n_jobs=32)>
>     652             # Make sure that we get a last message telling us we are
> done
>     653             elapsed_time = time.time() - self._start_time
>     654             self._print('Done %3i out of %3i | elapsed: %s
> finished',
>     655                         (len(self._output),
>
>
> ---------------------------------------------------------------------------
>     Sub-process traceback:
>
> ---------------------------------------------------------------------------
>     ValueError                                         Fri May 23 12:27:35
> 2014
> PID: 30246Python 2.7.6:
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/bin/python
> ...........................................................................
> /export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/time_frequency/multitaper.pyc
> in _psd_from_mt_adaptive(x_mt=array([], dtype=float64), eigvals=array([
> 0.99999996,  0.99999714,  0.99990416,  0.99810608,  0.97662124]),
> freq_mask=array([False, False, False, False, False, False,...  False, False,
> False, False, False], dtype=bool), max_iter=150, return_weights=False)
>     277     -----
>     278
>     279     The weights to use for making the multitaper estimate, such that
>     280     :math:`S_{mt} = \sum_{k} |w_k|^2S_k^{mt} / \sum_{k} |w_k|^2`
>     281     """
> --> 282     n_signals, n_tapers, n_freqs = x_mt.shape
>     283
>     284     if len(eigvals) != n_tapers:
>     285         raise ValueError('Need one eigenvalue for each taper')
>     286
>
> ValueError: need more than 1 value to unpack
>
>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
>



More information about the Mne_analysis mailing list