[Mne_analysis] Python - problem with PSD if n_jobs exceeds number of available CPUs?
Jon Houck
jhouck at unm.edu
Fri May 23 15:34:14 EDT 2014
Hi all,
I ran into what is probably a rare error using compute_source_psd_epochs
with adaptive=True. For certain labels, if n_jobs is greater than the
number of CPUs, the process will crash. In my case it was n_jobs=32, for a
condition with 20 epochs, in G_front_inf-Orbital-rh from aparc.a2009s. For
several other labels the process ran without error. Using n_jobs =19 or 20
was fine, 21 or higher failed reliably.
This may not affect much else. My workaround was to set n_jobs and test
the value against multiprocessing.cpu_count() early in the script. Other
options would be for the user to pay attention to which machine they are
using, or to be less greedy with CPU usage...
The traceback is below, for those who are into that kind of thing. This was
using the master downloaded on May 19 (0.8.git).
Jon
Computing PSD for offset event 8192 label G_front_inf-Orbital-rh at index 10
Considering frequencies 4 ... 20 Hz
Preparing the inverse operator for use...
Scaled noise and source covariance from nave = 1 to nave = 1
Created the regularized inverter
Created an SSP operator (subspace dimension = 7)
Created the whitener using a full noise covariance matrix (239 small
eigenvalues omitted)
Picked 306 channels from the data
Computing inverse...
(eigenleads need to be weighted)...
Reducing data rank to 21
Using 5 tapers with bandwidth 4.0Hz
Processing epoch : 1
[Parallel(n_jobs=32)]: Done 1 out of 32 | elapsed: 0.0s remaining:
0.1s
[Parallel(n_jobs=32)]: Done 6 out of 32 | elapsed: 0.0s remaining:
0.1s
[Parallel(n_jobs=32)]: Done 13 out of 32 | elapsed: 0.0s remaining:
0.1s
[Parallel(n_jobs=32)]: Done 20 out of 32 | elapsed: 0.0s remaining:
0.0s
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (3, 0))
Traceback (most recent call last):
File "<stdin>", line 15, in <module>
File
"/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/minimum_norm/time_frequency.py",
line 572, in _compute_source_psd_epochs
for x in np.array_split(x_mt_src, n_jobs))
File
"/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py",
line 651, in __call__
self.retrieve()
File
"/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py",
line 534, in retrieve
raise exception_type(report)
joblib.my_exceptions.JoblibValueError/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/my_exceptions.py:26:
DeprecationWarning: BaseException.message has been deprecated as of Python
2.6
self.message,
: JoblibValueError
___________________________________________________________________________
Multiprocessing exception:
...........................................................................
/na/homes/jhouck/docs/<stdin> in <module>()
10
11
12
13
14
---> 15
16
17
18
19
...........................................................................
/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/minimum_norm/time_frequency.py
in _compute_source_psd_epochs(epochs=<Epochs | n_events : 20 (all good),
tmin : -1.5 (s), tmax : 0.1 (s), baseline : (None, 0)>,
inverse_operator={'coord_frame': array([4], dtype=int32), 'depth_prior':
{'bads': [], 'data': array([ 0.03416839, 0.03416839, 0.03416839, ...,
0.21104597,
0.21104597, 0.21104597]), 'diag': True, 'dim': 24588, 'eig': None,
'eigvec': None, 'kind': 5, 'names': [], 'nfree': 1, 'projs': []},
'eigen_fields': {'col_names': [u'MEG0113', u'MEG0112', u'MEG0111',
u'MEG0122', u'MEG0123', u'MEG0121', u'MEG0132', u'MEG0133', u'MEG0131',
u'MEG0143', u'MEG0142', u'MEG0141', u'MEG0213', u'MEG0212', u'MEG0211',
u'MEG0222', u'MEG0223', u'MEG0221', u'MEG0232', u'MEG0233', ...], 'data':
array([[ 0.00000000e+00, 6.42533777e-08, 4.....00000000e+00,
0.00000000e+00]], dtype=float32), 'ncol': 306, 'nrow': 306, 'row_names':
[]}, 'eigen_leads': {'col_names': [], 'data': array([[ -2.40336126e-03,
1.22079987e-03, 7.....37850686e-05, -7.04127597e-05]], dtype=float32),
'ncol': 306, 'nrow': 24588, 'row_names': []}, 'eigen_leads_weighted':
False, 'fmri_prior': None, 'info': {'bads': [], 'ch_names': ['MEG0113',
'MEG0112', 'MEG0111', 'MEG0122', 'MEG0123', 'MEG0121', 'MEG0132',
'MEG0133', 'MEG0131', 'MEG0143', 'MEG0142', 'MEG0141', 'MEG0213',
'MEG0212', 'MEG0211', 'MEG0222', 'MEG0223', 'MEG0221', 'MEG0232',
'MEG0233', ...], 'chs': [{'cal': 3.250000046861601e-09, 'ch_name':
'MEG0113', 'coil_trans': array([[-0.01938988, -0.18108977, -0.98325199,
-... , 0. , 0. , 1. ]]), 'coil_type': 3012,
'coord_frame': 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1066 ,
0.0464 , -0.0604 , -0...
0.18097401, 0.020333 ], dtype=float32), 'logno': 113, 'range':
1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0112',
'coil_trans': array([[-0.18108977, 0.01938988, -0.98325199, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1066 , 0.0464 ,
-0.0604 , -0...
0.18097401, 0.020333 ], dtype=float32), 'logno': 112, 'range':
1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0111',
'coil_trans': array([[-0.01938988, -0.18108977, -0.98325199, -... ,
0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1066 , 0.0464 ,
-0.0604 , -0...
0.18097401, 0.020333 ], dtype=float32), 'logno': 111, 'range':
1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0122',
'coil_trans': array([[ 9.03011824e-04, -2.53265113e-01, -9....00,
0.00000000e+00,
1.00000000e+00]]), 'coil_type': 3012, 'coord_frame': 1,
'eeg_loc': None, 'kind': 1, 'loc': array([ -1.01999998e-01,
6.31000027e-02, -2.5...2.53259987e-01, 2.97999999e-04], dtype=float32),
'logno': 122, 'range': 1.0, ...}, {'cal': -3.250000046861601e-09,
'ch_name': 'MEG0123', 'coil_trans': array([[ -2.53265113e-01,
-9.03011824e-04, -9....00, 0.00000000e+00,
1.00000000e+00]]), 'coil_type': 3012, 'coord_frame': 1,
'eeg_loc': None, 'kind': 1, 'loc': array([ -1.01999998e-01,
6.31000027e-02, -2.5...2.53259987e-01, 2.97999999e-04], dtype=float32),
'logno': 123, 'range': 1.0, ...}, {'cal': 4.1400001482561066e-11,
'ch_name': 'MEG0121', 'coil_trans': array([[ 9.03011824e-04,
-2.53265113e-01, -9....00, 0.00000000e+00,
1.00000000e+00]]), 'coil_type': 3024, 'coord_frame': 1,
'eeg_loc': None, 'kind': 1, 'loc': array([ -1.01999998e-01,
6.31000027e-02, -2.5...2.53259987e-01, 2.97999999e-04], dtype=float32),
'logno': 121, 'range': 1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name':
'MEG0132', 'coil_trans': array([[ 0.00592455, -0.07752677, -0.99696398,
-... , 0. , 0. , 1. ]]), 'coil_type': 3012,
'coord_frame': 1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085 ,
0.0302 , -0.0266 , 0...
0.077537 , -0.005751 ], dtype=float32), 'logno': 132, 'range':
1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0133',
'coil_trans': array([[-0.07752677, -0.00592455, -0.99696398, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085 , 0.0302 ,
-0.0266 , -0...
0.077537 , -0.005751 ], dtype=float32), 'logno': 133, 'range':
1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0131',
'coil_trans': array([[ 0.00592455, -0.07752677, -0.99696398, -... ,
0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1085 , 0.0302 ,
-0.0266 , 0...
0.077537 , -0.005751 ], dtype=float32), 'logno': 131, 'range':
1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0143',
'coil_trans': array([[-0.05253163, -0.03200371, -0.99809802, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099 , 0.0131 ,
-0.0627 , -0...
0.031892 , 0.052595 ], dtype=float32), 'logno': 143, 'range':
1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0142',
'coil_trans': array([[-0.03200371, 0.05253163, -0.99809802, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099 , 0.0131 ,
-0.0627 , -0...
0.031892 , 0.052595 ], dtype=float32), 'logno': 142, 'range':
1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0141',
'coil_trans': array([[-0.05253163, -0.03200371, -0.99809802, -... ,
0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1099 , 0.0131 ,
-0.0627 , -0...
0.031892 , 0.052595 ], dtype=float32), 'logno': 141, 'range':
1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0213',
'coil_trans': array([[-0.05449655, -0.09574967, -0.99390799, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074 , 0.0329 ,
0.008 , -0...
0.095467 , 0.055009 ], dtype=float32), 'logno': 213, 'range':
1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0212',
'coil_trans': array([[-0.09574967, 0.05449655, -0.99390799, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074 , 0.0329 ,
0.008 , -0...
0.095467 , 0.055009 ], dtype=float32), 'logno': 212, 'range':
1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0211',
'coil_trans': array([[-0.05449655, -0.09574967, -0.99390799, -... ,
0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1074 , 0.0329 ,
0.008 , -0...
0.095467 , 0.055009 ], dtype=float32), 'logno': 211, 'range':
1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0222',
'coil_trans': array([[-0.34224641, -0.15804897, -0.92622799, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989 , 0.0403 ,
0.0413 , -0...
0.14645299, 0.34734601], dtype=float32), 'logno': 222, 'range':
1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0223',
'coil_trans': array([[-0.15804897, 0.34224641, -0.92622799, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989 , 0.0403 ,
0.0413 , -0...
0.14645299, 0.34734601], dtype=float32), 'logno': 223, 'range':
1.0, ...}, {'cal': 4.1400001482561066e-11, 'ch_name': 'MEG0221',
'coil_trans': array([[-0.34224641, -0.15804897, -0.92622799, -... ,
0. , 0. , 1. ]]), 'coil_type': 3024, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.0989 , 0.0403 ,
0.0413 , -0...
0.14645299, 0.34734601], dtype=float32), 'logno': 221, 'range':
1.0, ...}, {'cal': 3.250000046861601e-09, 'ch_name': 'MEG0232',
'coil_trans': array([[-0.34559116, 0.04283677, -0.93739903, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1011 , 0.0044 ,
0.0408 , -0...
-0.048961 , 0.34478199], dtype=float32), 'logno': 232, 'range':
1.0, ...}, {'cal': -3.250000046861601e-09, 'ch_name': 'MEG0233',
'coil_trans': array([[ 0.04283677, 0.34559116, -0.93739903, -... ,
0. , 0. , 1. ]]), 'coil_type': 3012, 'coord_frame':
1, 'eeg_loc': None, 'kind': 1, 'loc': array([-0.1011 , 0.0044 ,
0.0408 , 0...
-0.048961 , 0.34478199], dtype=float32), 'logno': 233, 'range':
1.0, ...}, ...], 'dev_head_t': {'from': 1, 'to': 4, 'trans': array([[
0.99667621, -0.07383786, 0.03442231, ... , 0. , 0.
, 1. ]])}, 'meas_file':
u'M87110374_hallucinations_raw_all_tsss_mc_trans.fif', 'meas_id': None,
'mri_file':
u'./M87110374_hallucinations_raw_all_tsss_mc_trans-ave-trans.fif',
'mri_head_t': {'from': 5, 'to': 4, 'trans': array([[ 0.98450035,
-0.13287687, -0.11446673, ... , 0. , 0. , 1.
]])}, 'mri_id': None, 'nchan': 306}, 'methods': 1, 'mri_head_t': {'from':
5, 'to': 4, 'trans': array([[ 0.98450035, -0.13287687, -0.11446673,
... , 0. , 0. , 1. ]])}, 'nave': 1, ...},
lambda2=1.0, method='MNE', fmin=4.0, fmax=20.0, pick_ori=None,
label=<Label | M87110374, u'G_front_inf-Orbital-rh', rh : 298 vertices>,
nave=1, pca=True, inv_split=None, bandwidth=4.0, adaptive=True,
low_bias=True, n_jobs=32, verbose=None)
567 x_mt_src[:, i, :] = np.dot(K_part, x_mt[:, i, :])
568
569 # compute the psd
570 if adaptive:
571 out = parallel(my_psd_from_mt_adaptive(x, eigvals,
freq_mask)
--> 572 for x in np.array_split(x_mt_src, n_jobs))
x_mt_src = array([[[ -6.20846446e-11 +0.00000000e+00j,
...j,
1.28781173e-11 +1.14432688e-13j]]])
n_jobs = 32
573 this_psd = np.concatenate(out)
574 else:
575 x_mt_src = x_mt_src[:, :, freq_mask]
576 this_psd = _psd_from_mt(x_mt_src, weights)
...........................................................................
/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/joblib-0.8.0a3-py2.7.egg/joblib/parallel.py
in __call__(self=Parallel(n_jobs=32), iterable=<generator object <genexpr>>)
646 if pre_dispatch == "all" or n_jobs == 1:
647 # The iterable was consumed all at once by the
above for loop.
648 # No need to wait for async callbacks to trigger to
649 # consumption.
650 self._iterating = False
--> 651 self.retrieve()
self.retrieve = <bound method Parallel.retrieve of
Parallel(n_jobs=32)>
652 # Make sure that we get a last message telling us we
are done
653 elapsed_time = time.time() - self._start_time
654 self._print('Done %3i out of %3i | elapsed: %s
finished',
655 (len(self._output),
---------------------------------------------------------------------------
Sub-process traceback:
---------------------------------------------------------------------------
ValueError Fri May 23 12:27:35
2014
PID: 30246Python 2.7.6:
/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/bin/python
...........................................................................
/export/research/analysis/human/jhouck/namaste_20628/tools/anaconda/lib/python2.7/site-packages/mne-0.8.git-py2.7.egg/mne/time_frequency/multitaper.pyc
in _psd_from_mt_adaptive(x_mt=array([], dtype=float64), eigvals=array([
0.99999996, 0.99999714, 0.99990416, 0.99810608, 0.97662124]),
freq_mask=array([False, False, False, False, False, False,... False,
False, False, False, False], dtype=bool), max_iter=150,
return_weights=False)
277 -----
278
279 The weights to use for making the multitaper estimate, such that
280 :math:`S_{mt} = \sum_{k} |w_k|^2S_k^{mt} / \sum_{k} |w_k|^2`
281 """
--> 282 n_signals, n_tapers, n_freqs = x_mt.shape
283
284 if len(eigvals) != n_tapers:
285 raise ValueError('Need one eigenvalue for each taper')
286
ValueError: need more than 1 value to unpack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20140523/be14d1aa/attachment.html
More information about the Mne_analysis
mailing list