[Mne_analysis] Filtering and ICA memory issues

[Mads Jensen]

Hi Dennis,

In this particular script you are right that I didn't use the decim 
parameter. I just tried it playing around a bit in a previous version.

I have combined MEG & EEG so even a decim = 10 could work then.

I haven't tried picking only the first 100-200 ecg events. I'll try that.

cheers,
mads


On 25/11/15 15:54, Denis-Alexander Engemann wrote:
> Hi Mads,
>
> it seems you don't use the decim parameter, do you?
> Two things that I see immediately:
> It should in fact save a lot. With 1000HZ you can decimate even more,
> for ECG/EOG your sampling frequency should not be lower than 50 or so.
> Second as you have 1 hour of data the ecg_epochs will be huge, assuming
> you find many events.
> Very often only a few are necessary to do the detection. Have you tried
> picking the 100-200 first events?
>
> We'll have a closer look soon.
> Denis
>
> On Wed, Nov 25, 2015 at 3:47 PM, Mads Jensen <mje.mads at gmail.com
> <mailto:mje.mads at gmail.com>> wrote:
>
>     Hi,
>
>     I use sklearn 0.17 (from anaconda). I have tried to the
>     "decim" param. I remember it as being "3" for data with 1000Hz
>     sfreq. But it didn't help much.
>
>     I have attach a script to show how I used it.
>
>     cheers,
>     mads
>
>
>
>     On 25/11/15 15:29, Denis-Alexander Engemann wrote:
>
>         Hi Mads,
>
>         Which version of sklearn are you using?
>         Do you use the decim parameter for ICA?
>         How do axactly do you use ICA?
>         50GB of memory is unexpected, it would mean that you make up to 10
>         copies of your data.
>
>
>         On Wed, Nov 25, 2015 at 3:24 PM, Mads Jensen <mje.mads at gmail.com
>         <mailto:mje.mads at gmail.com>
>         <mailto:mje.mads at gmail.com <mailto:mje.mads at gmail.com>>> wrote:
>
>              Hi all,
>
>              I would like to hear what people do to filter and run ICA
>         and if there
>              is any advise.
>
>              We usually have around an hour of recording which gives
>         ~4.5 to 5GB of
>              raw fiff files. First filtering and then running ICA in
>         MNE-python
>              requires a lot of memory, sometimes as much as 50GB. So, I
>         fairly often
>              get a memory error.
>
>              I would prefer not to downsample at this stage in the
>         process. So, I
>              kindly ask if anybody has any thoughts and/or practises to
>         avoid very
>              heavy memory use.
>
>              best wishes,
>              mads
>              _______________________________________________
>              Mne_analysis mailing list
>         Mne_analysis at nmr.mgh.harvard.edu
>         <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>              <mailto:Mne_analysis at nmr.mgh.harvard.edu
>         <mailto:Mne_analysis at nmr.mgh.harvard.edu>>
>         https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
>              The information in this e-mail is intended only for the
>         person to
>              whom it is
>              addressed. If you believe this e-mail was sent to you in
>         error and
>              the e-mail
>              contains patient information, please contact the Partners
>         Compliance
>              HelpLine at
>         http://www.partners.org/complianceline . If the e-mail was sent to
>              you in error
>              but does not contain patient information, please contact
>         the sender
>              and properly
>              dispose of the e-mail.
>
>
>
>
>         _______________________________________________
>         Mne_analysis mailing list
>         Mne_analysis at nmr.mgh.harvard.edu
>         <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>         https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
>         The information in this e-mail is intended only for the person
>         to whom it is
>         addressed. If you believe this e-mail was sent to you in error
>         and the e-mail
>         contains patient information, please contact the Partners
>         Compliance HelpLine at
>         http://www.partners.org/complianceline . If the e-mail was sent
>         to you in error
>         but does not contain patient information, please contact the
>         sender and properly
>         dispose of the e-mail.
>
>
>     _______________________________________________
>     Mne_analysis mailing list
>     Mne_analysis at nmr.mgh.harvard.edu
>     <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>     https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
>     The information in this e-mail is intended only for the person to
>     whom it is
>     addressed. If you believe this e-mail was sent to you in error and
>     the e-mail
>     contains patient information, please contact the Partners Compliance
>     HelpLine at
>     http://www.partners.org/complianceline . If the e-mail was sent to
>     you in error
>     but does not contain patient information, please contact the sender
>     and properly
>     dispose of the e-mail.
>
>
>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the e-mail
> contains patient information, please contact the Partners Compliance HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in error
> but does not contain patient information, please contact the sender and properly
> dispose of the e-mail.
>