[Mne_analysis] Filtering and ICA memory issues

Christopher Bailey cjb at cfin.au.dk
Fri Nov 27 03:52:00 EST 2015
Search archives:

Hi Alex, I’ll move this to github, with an example script using memory_profiler. My initial experiments some time ago fizzled when I got the feeling I was thinking about memmapping the wrong way (and that it would not in fact be beneficial in the filtering-scenario). /Chris

> On 26 Nov 2015, at 21:48, Alexandre Gramfort <alexandre.gramfort at telecom-paristech.fr> wrote:
> 
> hi Chris,
> 
> memory usage is certainly a topic where we can progress.
> 
> For memmap just pass a string to preload parameter. I wrote this
> years ago, but I admit I almost never used it. So not sure how much
> it can actually help.
> 
> if you can share a script that we can use for memory monitoring eg on
> sample data
> that would be great.
> 
> we'll see what we can do.
> 
> Best,
> Alex
> 
> 
> 
> 
> On Thu, Nov 26, 2015 at 2:31 PM, Christopher Bailey <cjb at cfin.au.dk> wrote:
>> Hi Denis,
>> 
>> Thanks for you advice (I hadn’t considered the number of ECG epochs!). I
>> would like to follow up on Mads’ original post regarding the large
>> memory-footprint.
>> 
>> An hour of MEG+EEG at 1kHz adds up to about 5GB of data saved as 32-bit
>> floats. However, preloading raw data (regardless of origin) forces a cast to
>> float-64 (10 GB). This makes sense: preloading raw data means you are going
>> to want to filter it, which requires the bit-depth to work.
>> 
>> However, reading all 400+ channels into memory does not make sense from an
>> implementation perspective, since (most of) the relevant operations are
>> channel-wise (certainly filtering). A much more efficient strategy would be
>> to support block-reads of channel subsets, probably combined with
>> np.memmap:ing. I can see some almost-implementations of memmap in the code,
>> but the standard API does not expose them (for simplicity, understandably).
>> Nor am I completely certain a memmap is helpful here...
>> 
>> There’s been some chatter on related topic on github (where this discussion
>> also belongs), but I wonder if there’s any developments in this direction?
>> 
>> https://github.com/mne-tools/mne-python/issues/1766
>> 
>> /Chris
>> 
>> On 25 Nov 2015, at 16:11, Denis-Alexander Engemann
>> <denis.engemann at gmail.com> wrote:
>> 
>> On top of that you can delete the epochs objects once they did their job.
>> Also take a look at my repo how I'm using the MNE-Python ICA.
>> 
>> https://github.com/dengemann/meeg-preprocessing/blob/master/examples/plot_preprocess_filter_ica.py
>> 
>> and
>> 
>> https://github.com/dengemann/meeg-preprocessing/blob/master/meeg_preprocessing/preprocessing.py#L109
>> 
>> It's basically the pimped example inside a function + reporting
>> functionality.
>> 
>> 
>> 
>> 
>> 
>> 
>> On Wed, Nov 25, 2015 at 4:06 PM, Mads Jensen <mje.mads at gmail.com> wrote:
>>> 
>>> Hi Dennis,
>>> 
>>> In this particular script you are right that I didn't use the decim
>>> parameter. I just tried it playing around a bit in a previous version.
>>> 
>>> I have combined MEG & EEG so even a decim = 10 could work then.
>>> 
>>> I haven't tried picking only the first 100-200 ecg events. I'll try that.
>>> 
>>> cheers,
>>> mads
>>> 
>>> 
>>> On 25/11/15 15:54, Denis-Alexander Engemann wrote:
>>>> Hi Mads,
>>>> 
>>>> it seems you don't use the decim parameter, do you?
>>>> Two things that I see immediately:
>>>> It should in fact save a lot. With 1000HZ you can decimate even more,
>>>> for ECG/EOG your sampling frequency should not be lower than 50 or so.
>>>> Second as you have 1 hour of data the ecg_epochs will be huge, assuming
>>>> you find many events.
>>>> Very often only a few are necessary to do the detection. Have you tried
>>>> picking the 100-200 first events?
>>>> 
>>>> We'll have a closer look soon.
>>>> Denis
>>>> 
>>>> On Wed, Nov 25, 2015 at 3:47 PM, Mads Jensen <mje.mads at gmail.com
>>>> <mailto:mje.mads at gmail.com>> wrote:
>>>> 
>>>>    Hi,
>>>> 
>>>>    I use sklearn 0.17 (from anaconda). I have tried to the
>>>>    "decim" param. I remember it as being "3" for data with 1000Hz
>>>>    sfreq. But it didn't help much.
>>>> 
>>>>    I have attach a script to show how I used it.
>>>> 
>>>>    cheers,
>>>>    mads
>>>> 
>>>> 
>>>> 
>>>>    On 25/11/15 15:29, Denis-Alexander Engemann wrote:
>>>> 
>>>>        Hi Mads,
>>>> 
>>>>        Which version of sklearn are you using?
>>>>        Do you use the decim parameter for ICA?
>>>>        How do axactly do you use ICA?
>>>>        50GB of memory is unexpected, it would mean that you make up to
>>>> 10
>>>>        copies of your data.
>>>> 
>>>> 
>>>>        On Wed, Nov 25, 2015 at 3:24 PM, Mads Jensen <mje.mads at gmail.com
>>>>        <mailto:mje.mads at gmail.com>
>>>>        <mailto:mje.mads at gmail.com <mailto:mje.mads at gmail.com>>> wrote:
>>>> 
>>>>             Hi all,
>>>> 
>>>>             I would like to hear what people do to filter and run ICA
>>>>        and if there
>>>>             is any advise.
>>>> 
>>>>             We usually have around an hour of recording which gives
>>>>        ~4.5 to 5GB of
>>>>             raw fiff files. First filtering and then running ICA in
>>>>        MNE-python
>>>>             requires a lot of memory, sometimes as much as 50GB. So, I
>>>>        fairly often
>>>>             get a memory error.
>>>> 
>>>>             I would prefer not to downsample at this stage in the
>>>>        process. So, I
>>>>             kindly ask if anybody has any thoughts and/or practises to
>>>>        avoid very
>>>>             heavy memory use.
>>>> 
>>>>             best wishes,
>>>>             mads
>>>>             _______________________________________________
>>>>             Mne_analysis mailing list
>>>>        Mne_analysis at nmr.mgh.harvard.edu
>>>>        <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>>>>             <mailto:Mne_analysis at nmr.mgh.harvard.edu
>>>>        <mailto:Mne_analysis at nmr.mgh.harvard.edu>>
>>>>        https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>> 
>>>> 
>>>>             The information in this e-mail is intended only for the
>>>>        person to
>>>>             whom it is
>>>>             addressed. If you believe this e-mail was sent to you in
>>>>        error and
>>>>             the e-mail
>>>>             contains patient information, please contact the Partners
>>>>        Compliance
>>>>             HelpLine at
>>>>        http://www.partners.org/complianceline . If the e-mail was sent
>>>> to
>>>>             you in error
>>>>             but does not contain patient information, please contact
>>>>        the sender
>>>>             and properly
>>>>             dispose of the e-mail.
>>>> 
>>>> 
>>>> 
>>>> 
>>>>        _______________________________________________
>>>>        Mne_analysis mailing list
>>>>        Mne_analysis at nmr.mgh.harvard.edu
>>>>        <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>>>>        https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>> 
>>>> 
>>>>        The information in this e-mail is intended only for the person
>>>>        to whom it is
>>>>        addressed. If you believe this e-mail was sent to you in error
>>>>        and the e-mail
>>>>        contains patient information, please contact the Partners
>>>>        Compliance HelpLine at
>>>>        http://www.partners.org/complianceline . If the e-mail was sent
>>>>        to you in error
>>>>        but does not contain patient information, please contact the
>>>>        sender and properly
>>>>        dispose of the e-mail.
>>>> 
>>>> 
>>>>    _______________________________________________
>>>>    Mne_analysis mailing list
>>>>    Mne_analysis at nmr.mgh.harvard.edu
>>>>    <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>>>>    https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>> 
>>>> 
>>>>    The information in this e-mail is intended only for the person to
>>>>    whom it is
>>>>    addressed. If you believe this e-mail was sent to you in error and
>>>>    the e-mail
>>>>    contains patient information, please contact the Partners Compliance
>>>>    HelpLine at
>>>>    http://www.partners.org/complianceline . If the e-mail was sent to
>>>>    you in error
>>>>    but does not contain patient information, please contact the sender
>>>>    and properly
>>>>    dispose of the e-mail.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Mne_analysis mailing list
>>>> Mne_analysis at nmr.mgh.harvard.edu
>>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>> 
>>>> 
>>>> The information in this e-mail is intended only for the person to whom
>>>> it is
>>>> addressed. If you believe this e-mail was sent to you in error and the
>>>> e-mail
>>>> contains patient information, please contact the Partners Compliance
>>>> HelpLine at
>>>> http://www.partners.org/complianceline . If the e-mail was sent to you
>>>> in error
>>>> but does not contain patient information, please contact the sender and
>>>> properly
>>>> dispose of the e-mail.
>>>> 
>>> _______________________________________________
>>> Mne_analysis mailing list
>>> Mne_analysis at nmr.mgh.harvard.edu
>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>> 
>> 
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>> 
>> 
>> The information in this e-mail is intended only for the person to whom it is
>> addressed. If you believe this e-mail was sent to you in error and the
>> e-mail
>> contains patient information, please contact the Partners Compliance
>> HelpLine at
>> http://www.partners.org/complianceline . If the e-mail was sent to you in
>> error
>> but does not contain patient information, please contact the sender and
>> properly
>> dispose of the e-mail.
>> 
>> 
>> 
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>> 
>> 
>> The information in this e-mail is intended only for the person to whom it is
>> addressed. If you believe this e-mail was sent to you in error and the
>> e-mail
>> contains patient information, please contact the Partners Compliance
>> HelpLine at
>> http://www.partners.org/complianceline . If the e-mail was sent to you in
>> error
>> but does not contain patient information, please contact the sender and
>> properly
>> dispose of the e-mail.
>> 
> 
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis




More information about the Mne_analysis mailing list