[Mne_analysis] PSDEstimator using multiple cores even when n_jobs=1

[Alexandre Gramfort]

        External Email - Use Caution        

hi Kyuhwa,

even with n_jobs=1 if you use a multithreaded linear algebra library like
MKL or openblas you will use more than 1 core.

to prevent this you should read about the environment variable :
OMP_NUM_THREADS

HTH
Alex

On Mon, Jul 2, 2018 at 1:52 PM Lee, KyuHwa <lee.kyuh at gmail.com> wrote:

>         External Email - Use Caution
>
> Hi all,
>
> I found a problem in mne.decoding.PSDEstimator.transform() method. In
> short, it does not respect the attribute "n_jobs". This problem does not
> exist in MNE 0.14 and only exists in MNE 0.15 and above. I tested only on
> stable versions.
>
> Normally, I restrict the PSD estimation to use a single core (n_jobs=1)
> because there are many other time-critical processes running together in
> the system. In MNE 0.15 and above, the transform() function uses up to 4
> cores equivalent (over 50% total usage on an 8-core system). This is in
> contrast with the same function in MNE 0.14, where only 1 core is used as
> expected.
>
> The following minimal code replicates the problem.
>
> import numpy as npimport mneimport time
> mne.set_log_level('WARNING')
>
> psde = mne.decoding.PSDEstimator(sfreq=512, fmin=1, fmax=40, n_jobs=1)
> iterations = 1000
> w = np.random.rand(1, 64, 256)print('Start %d iteratinos' % iterations)
> t_start = time.time()for i in range(iterations):
>     psd = psde.transform(w)print('Took %.3f seconds' % (time.time() - t_start))
>
> In Windows 10 Pro with i7-4790K CPU (8 logical cores), I ran the above
> code 5 times each with 1000 computations. In the idle period, the cpu usage
> was maximum 3%.
>
> *Results*
> - mne 0.14: CPU usage 17% (equivalent to approx. 1 core = 12.5%)
> 10.461, 10.755, 10.861, 10.843, 10.636 seconds.
> Average: 10.711s per 1000 computations.
>
> - mne 0.15, 0.16: CPU usage 56% (equivalent to approx. 4 cores = 50%)
> 11.608, 10.737, 10.734, 11.283, 10.909  seconds.
> Average: 11.054s per 1000 computations.
>
> So, I don't see much difference in computation time but the CPU usage was
> 4x higher in MNE 0.15 and above. It seems to me that there's a bug in
> PSDEstimator class that always prefers to use multiple cores even when
> n_jobs=1, causing a lot of overhead by forced multiprocessing.
>
> Please have a look and fix this issue. Thanks!
>
> Best,
> Kyuhwa Lee
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to whom it
> is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20180703/83530733/attachment.html