[Mne_analysis] shared memory using joblib and clusterering-permutation tests

[Denis-Alexander Engemann]

It turns out the minimum size for memmaping has to be set to make it work.

mne.set_memmap_min_size('100K')

or mne.set_memmap_min_size('1M')

etc.

On success the process will create a memmap / caching in the cache
directory set.

I'm now running computations on a 10GB array using 4 jobs without hangs /
crashes.
Nice.




On Wed, Aug 6, 2014 at 12:38 PM, Denis-Alexander Engemann <
denis.engemann at gmail.com> wrote:

> Dear joblib experts on this list,
>
> I'm wondering which steps are required to use joblib's shared memory
> feature.
> I used `mne.set_cache_dir` and made sure my `stat_fun` treats variables
> independently.
> I started with a moderate buffer_size of 500.
> As a result my memory consumption increases radically with 4 jobs and my
> system starts swapping excessively.
> This is not expected, I think. Given the small buffer size I would not
> have expected any significant memory increase.
> After some time I witness a `SystemError: error return without exception
> set` which is not good.
>
> How can I be really sure it actually works?
> I feel we should add more verbosity and logging here.
>
> Best,
> Denis
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20140806/8129bb3c/attachment.html