[Mne_analysis] Filtering and ICA memory issues

Denis-Alexander Engemann denis.engemann at gmail.com
Wed Nov 25 10:11:54 EST 2015
Search archives:

On top of that you can delete the epochs objects once they did their job.
Also take a look at my repo how I'm using the MNE-Python ICA.

https://github.com/dengemann/meeg-preprocessing/blob/master/examples/plot_preprocess_filter_ica.py

and

https://github.com/dengemann/meeg-preprocessing/blob/master/meeg_preprocessing/preprocessing.py#L109

It's basically the pimped example inside a function + reporting
functionality.






On Wed, Nov 25, 2015 at 4:06 PM, Mads Jensen <mje.mads at gmail.com> wrote:

> Hi Dennis,
>
> In this particular script you are right that I didn't use the decim
> parameter. I just tried it playing around a bit in a previous version.
>
> I have combined MEG & EEG so even a decim = 10 could work then.
>
> I haven't tried picking only the first 100-200 ecg events. I'll try that.
>
> cheers,
> mads
>
>
> On 25/11/15 15:54, Denis-Alexander Engemann wrote:
> > Hi Mads,
> >
> > it seems you don't use the decim parameter, do you?
> > Two things that I see immediately:
> > It should in fact save a lot. With 1000HZ you can decimate even more,
> > for ECG/EOG your sampling frequency should not be lower than 50 or so.
> > Second as you have 1 hour of data the ecg_epochs will be huge, assuming
> > you find many events.
> > Very often only a few are necessary to do the detection. Have you tried
> > picking the 100-200 first events?
> >
> > We'll have a closer look soon.
> > Denis
> >
> > On Wed, Nov 25, 2015 at 3:47 PM, Mads Jensen <mje.mads at gmail.com
> > <mailto:mje.mads at gmail.com>> wrote:
> >
> >     Hi,
> >
> >     I use sklearn 0.17 (from anaconda). I have tried to the
> >     "decim" param. I remember it as being "3" for data with 1000Hz
> >     sfreq. But it didn't help much.
> >
> >     I have attach a script to show how I used it.
> >
> >     cheers,
> >     mads
> >
> >
> >
> >     On 25/11/15 15:29, Denis-Alexander Engemann wrote:
> >
> >         Hi Mads,
> >
> >         Which version of sklearn are you using?
> >         Do you use the decim parameter for ICA?
> >         How do axactly do you use ICA?
> >         50GB of memory is unexpected, it would mean that you make up to
> 10
> >         copies of your data.
> >
> >
> >         On Wed, Nov 25, 2015 at 3:24 PM, Mads Jensen <mje.mads at gmail.com
> >         <mailto:mje.mads at gmail.com>
> >         <mailto:mje.mads at gmail.com <mailto:mje.mads at gmail.com>>> wrote:
> >
> >              Hi all,
> >
> >              I would like to hear what people do to filter and run ICA
> >         and if there
> >              is any advise.
> >
> >              We usually have around an hour of recording which gives
> >         ~4.5 to 5GB of
> >              raw fiff files. First filtering and then running ICA in
> >         MNE-python
> >              requires a lot of memory, sometimes as much as 50GB. So, I
> >         fairly often
> >              get a memory error.
> >
> >              I would prefer not to downsample at this stage in the
> >         process. So, I
> >              kindly ask if anybody has any thoughts and/or practises to
> >         avoid very
> >              heavy memory use.
> >
> >              best wishes,
> >              mads
> >              _______________________________________________
> >              Mne_analysis mailing list
> >         Mne_analysis at nmr.mgh.harvard.edu
> >         <mailto:Mne_analysis at nmr.mgh.harvard.edu>
> >              <mailto:Mne_analysis at nmr.mgh.harvard.edu
> >         <mailto:Mne_analysis at nmr.mgh.harvard.edu>>
> >         https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> >
> >
> >              The information in this e-mail is intended only for the
> >         person to
> >              whom it is
> >              addressed. If you believe this e-mail was sent to you in
> >         error and
> >              the e-mail
> >              contains patient information, please contact the Partners
> >         Compliance
> >              HelpLine at
> >         http://www.partners.org/complianceline . If the e-mail was sent
> to
> >              you in error
> >              but does not contain patient information, please contact
> >         the sender
> >              and properly
> >              dispose of the e-mail.
> >
> >
> >
> >
> >         _______________________________________________
> >         Mne_analysis mailing list
> >         Mne_analysis at nmr.mgh.harvard.edu
> >         <mailto:Mne_analysis at nmr.mgh.harvard.edu>
> >         https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> >
> >
> >         The information in this e-mail is intended only for the person
> >         to whom it is
> >         addressed. If you believe this e-mail was sent to you in error
> >         and the e-mail
> >         contains patient information, please contact the Partners
> >         Compliance HelpLine at
> >         http://www.partners.org/complianceline . If the e-mail was sent
> >         to you in error
> >         but does not contain patient information, please contact the
> >         sender and properly
> >         dispose of the e-mail.
> >
> >
> >     _______________________________________________
> >     Mne_analysis mailing list
> >     Mne_analysis at nmr.mgh.harvard.edu
> >     <mailto:Mne_analysis at nmr.mgh.harvard.edu>
> >     https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> >
> >
> >     The information in this e-mail is intended only for the person to
> >     whom it is
> >     addressed. If you believe this e-mail was sent to you in error and
> >     the e-mail
> >     contains patient information, please contact the Partners Compliance
> >     HelpLine at
> >     http://www.partners.org/complianceline . If the e-mail was sent to
> >     you in error
> >     but does not contain patient information, please contact the sender
> >     and properly
> >     dispose of the e-mail.
> >
> >
> >
> >
> > _______________________________________________
> > Mne_analysis mailing list
> > Mne_analysis at nmr.mgh.harvard.edu
> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> >
> >
> > The information in this e-mail is intended only for the person to whom
> it is
> > addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> > contains patient information, please contact the Partners Compliance
> HelpLine at
> > http://www.partners.org/complianceline . If the e-mail was sent to you
> in error
> > but does not contain patient information, please contact the sender and
> properly
> > dispose of the e-mail.
> >
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20151125/aee6247d/attachment-0001.html 


More information about the Mne_analysis mailing list