[Mne_analysis] Filtering and ICA memory issues
Christopher Bailey
cjb at cfin.au.dk
Thu Nov 26 08:31:12 EST 2015
Hi Denis,
Thanks for you advice (I hadn’t considered the number of ECG epochs!). I would like to follow up on Mads’ original post regarding the large memory-footprint.
An hour of MEG+EEG at 1kHz adds up to about 5GB of data saved as 32-bit floats. However, preloading raw data (regardless of origin) forces a cast to float-64 (10 GB). This makes sense: preloading raw data means you are going to want to filter it, which requires the bit-depth to work.
However, reading all 400+ channels into memory does not make sense from an implementation perspective, since (most of) the relevant operations are channel-wise (certainly filtering). A much more efficient strategy would be to support block-reads of channel subsets, probably combined with np.memmap:ing. I can see some almost-implementations of memmap in the code, but the standard API does not expose them (for simplicity, understandably). Nor am I completely certain a memmap is helpful here...
There’s been some chatter on related topic on github (where this discussion also belongs), but I wonder if there’s any developments in this direction?
https://github.com/mne-tools/mne-python/issues/1766
/Chris
On 25 Nov 2015, at 16:11, Denis-Alexander Engemann <denis.engemann at gmail.com<mailto:denis.engemann at gmail.com>> wrote:
On top of that you can delete the epochs objects once they did their job.
Also take a look at my repo how I'm using the MNE-Python ICA.
https://github.com/dengemann/meeg-preprocessing/blob/master/examples/plot_preprocess_filter_ica.py
and
https://github.com/dengemann/meeg-preprocessing/blob/master/meeg_preprocessing/preprocessing.py#L109
It's basically the pimped example inside a function + reporting functionality.
On Wed, Nov 25, 2015 at 4:06 PM, Mads Jensen <mje.mads at gmail.com<mailto:mje.mads at gmail.com>> wrote:
Hi Dennis,
In this particular script you are right that I didn't use the decim
parameter. I just tried it playing around a bit in a previous version.
I have combined MEG & EEG so even a decim = 10 could work then.
I haven't tried picking only the first 100-200 ecg events. I'll try that.
cheers,
mads
On 25/11/15 15:54, Denis-Alexander Engemann wrote:
> Hi Mads,
>
> it seems you don't use the decim parameter, do you?
> Two things that I see immediately:
> It should in fact save a lot. With 1000HZ you can decimate even more,
> for ECG/EOG your sampling frequency should not be lower than 50 or so.
> Second as you have 1 hour of data the ecg_epochs will be huge, assuming
> you find many events.
> Very often only a few are necessary to do the detection. Have you tried
> picking the 100-200 first events?
>
> We'll have a closer look soon.
> Denis
>
> On Wed, Nov 25, 2015 at 3:47 PM, Mads Jensen <mje.mads at gmail.com<mailto:mje.mads at gmail.com>
> <mailto:mje.mads at gmail.com<mailto:mje.mads at gmail.com>>> wrote:
>
> Hi,
>
> I use sklearn 0.17 (from anaconda). I have tried to the
> "decim" param. I remember it as being "3" for data with 1000Hz
> sfreq. But it didn't help much.
>
> I have attach a script to show how I used it.
>
> cheers,
> mads
>
>
>
> On 25/11/15 15:29, Denis-Alexander Engemann wrote:
>
> Hi Mads,
>
> Which version of sklearn are you using?
> Do you use the decim parameter for ICA?
> How do axactly do you use ICA?
> 50GB of memory is unexpected, it would mean that you make up to 10
> copies of your data.
>
>
> On Wed, Nov 25, 2015 at 3:24 PM, Mads Jensen <mje.mads at gmail.com<mailto:mje.mads at gmail.com>
> <mailto:mje.mads at gmail.com<mailto:mje.mads at gmail.com>>
> <mailto:mje.mads at gmail.com<mailto:mje.mads at gmail.com> <mailto:mje.mads at gmail.com<mailto:mje.mads at gmail.com>>>> wrote:
>
> Hi all,
>
> I would like to hear what people do to filter and run ICA
> and if there
> is any advise.
>
> We usually have around an hour of recording which gives
> ~4.5 to 5GB of
> raw fiff files. First filtering and then running ICA in
> MNE-python
> requires a lot of memory, sometimes as much as 50GB. So, I
> fairly often
> get a memory error.
>
> I would prefer not to downsample at this stage in the
> process. So, I
> kindly ask if anybody has any thoughts and/or practises to
> avoid very
> heavy memory use.
>
> best wishes,
> mads
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>
> <mailto:Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>>
> <mailto:Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>
> <mailto:Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>>>
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the
> person to
> whom it is
> addressed. If you believe this e-mail was sent to you in
> error and
> the e-mail
> contains patient information, please contact the Partners
> Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to
> you in error
> but does not contain patient information, please contact
> the sender
> and properly
> dispose of the e-mail.
>
>
>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>
> <mailto:Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>>
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person
> to whom it is
> addressed. If you believe this e-mail was sent to you in error
> and the e-mail
> contains patient information, please contact the Partners
> Compliance HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent
> to you in error
> but does not contain patient information, please contact the
> sender and properly
> dispose of the e-mail.
>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>
> <mailto:Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>>
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to
> whom it is
> addressed. If you believe this e-mail was sent to you in error and
> the e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to
> you in error
> but does not contain patient information, please contact the sender
> and properly
> dispose of the e-mail.
>
>
>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the e-mail
> contains patient information, please contact the Partners Compliance HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in error
> but does not contain patient information, please contact the sender and properly
> dispose of the e-mail.
>
_______________________________________________
Mne_analysis mailing list
Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
_______________________________________________
Mne_analysis mailing list
Mne_analysis at nmr.mgh.harvard.edu<mailto:Mne_analysis at nmr.mgh.harvard.edu>
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20151126/0215cb4a/attachment-0001.html
More information about the Mne_analysis
mailing list