<div dir="ltr">It turns out the minimum size for memmaping has to be set to make it work.<div><br></div><div>mne.set_memmap_min_size('100K') <div><br></div></div><div>or mne.set_memmap_min_size('1M')</div><div>
<br></div><div>etc.</div><div><br></div><div>On success the process will create a memmap / caching in the cache directory set.</div><div><br></div><div>I'm now running computations on a 10GB array using 4 jobs without hangs / crashes. </div>
<div>Nice.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Aug 6, 2014 at 12:38 PM, Denis-Alexander Engemann <span dir="ltr"><<a href="mailto:denis.engemann@gmail.com" target="_blank">denis.engemann@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Dear joblib experts on this list,<div><br></div><div>I'm wondering which steps are required to use joblib's shared memory feature.</div>
<div>I used `mne.set_cache_dir` and made sure my `stat_fun` treats variables independently.</div>
<div>I started with a moderate buffer_size of 500.</div><div>As a result my memory consumption increases radically with 4 jobs and my system starts swapping excessively. </div><div>This is not expected, I think. Given the small buffer size I would not have expected any significant memory increase.</div>
<div>After some time I witness a `SystemError: error return without exception set` which is not good.</div><div><br></div><div>How can I be really sure it actually works?</div><div>I feel we should add more verbosity and logging here.</div>
<div><br></div><div>Best,</div><div>Denis</div></div>
</blockquote></div><br></div>