[Mne_analysis] PSDEstimator using multiple cores even when n_jobs=1
alexandre.gramfort at inria.fr
Tue Jul 3 16:59:41 EDT 2018
External Email - Use Caution
even with n_jobs=1 if you use a multithreaded linear algebra library like
MKL or openblas you will use more than 1 core.
to prevent this you should read about the environment variable :
On Mon, Jul 2, 2018 at 1:52 PM Lee, KyuHwa <lee.kyuh at gmail.com> wrote:
> External Email - Use Caution
> Hi all,
> I found a problem in mne.decoding.PSDEstimator.transform() method. In
> short, it does not respect the attribute "n_jobs". This problem does not
> exist in MNE 0.14 and only exists in MNE 0.15 and above. I tested only on
> stable versions.
> Normally, I restrict the PSD estimation to use a single core (n_jobs=1)
> because there are many other time-critical processes running together in
> the system. In MNE 0.15 and above, the transform() function uses up to 4
> cores equivalent (over 50% total usage on an 8-core system). This is in
> contrast with the same function in MNE 0.14, where only 1 core is used as
> The following minimal code replicates the problem.
> import numpy as npimport mneimport time
> psde = mne.decoding.PSDEstimator(sfreq=512, fmin=1, fmax=40, n_jobs=1)
> iterations = 1000
> w = np.random.rand(1, 64, 256)print('Start %d iteratinos' % iterations)
> t_start = time.time()for i in range(iterations):
> psd = psde.transform(w)print('Took %.3f seconds' % (time.time() - t_start))
> In Windows 10 Pro with i7-4790K CPU (8 logical cores), I ran the above
> code 5 times each with 1000 computations. In the idle period, the cpu usage
> was maximum 3%.
> - mne 0.14: CPU usage 17% (equivalent to approx. 1 core = 12.5%)
> 10.461, 10.755, 10.861, 10.843, 10.636 seconds.
> Average: 10.711s per 1000 computations.
> - mne 0.15, 0.16: CPU usage 56% (equivalent to approx. 4 cores = 50%)
> 11.608, 10.737, 10.734, 11.283, 10.909 seconds.
> Average: 11.054s per 1000 computations.
> So, I don't see much difference in computation time but the CPU usage was
> 4x higher in MNE 0.15 and above. It seems to me that there's a bug in
> PSDEstimator class that always prefers to use multiple cores even when
> n_jobs=1, causing a lot of overhead by forced multiprocessing.
> Please have a look and fix this issue. Thanks!
> Kyuhwa Lee
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> The information in this e-mail is intended only for the person to whom it
> addressed. If you believe this e-mail was sent to you in error and the
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> but does not contain patient information, please contact the sender and
> dispose of the e-mail.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Mne_analysis