<div dir="ltr">On top of that you can delete the epochs objects once they did their job.<div>Also take a look at my repo how I'm using the MNE-Python ICA.</div><div><br></div><div><a href="https://github.com/dengemann/meeg-preprocessing/blob/master/examples/plot_preprocess_filter_ica.py">https://github.com/dengemann/meeg-preprocessing/blob/master/examples/plot_preprocess_filter_ica.py</a><br></div><div><br></div><div>and</div><div><br></div><div><a href="https://github.com/dengemann/meeg-preprocessing/blob/master/meeg_preprocessing/preprocessing.py#L109">https://github.com/dengemann/meeg-preprocessing/blob/master/meeg_preprocessing/preprocessing.py#L109</a><br></div><div><br></div><div>It's basically the pimped example inside a function + reporting functionality.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 25, 2015 at 4:06 PM, Mads Jensen <span dir="ltr"><<a href="mailto:mje.mads@gmail.com" target="_blank">mje.mads@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Dennis,<br>
<br>
In this particular script you are right that I didn't use the decim<br>
parameter. I just tried it playing around a bit in a previous version.<br>
<br>
I have combined MEG & EEG so even a decim = 10 could work then.<br>
<br>
I haven't tried picking only the first 100-200 ecg events. I'll try that.<br>
<br>
cheers,<br>
mads<br>
<span class=""><br>
<br>
On 25/11/15 15:54, Denis-Alexander Engemann wrote:<br>
> Hi Mads,<br>
><br>
</span><span class="">> it seems you don't use the decim parameter, do you?<br>
> Two things that I see immediately:<br>
> It should in fact save a lot. With 1000HZ you can decimate even more,<br>
> for ECG/EOG your sampling frequency should not be lower than 50 or so.<br>
> Second as you have 1 hour of data the ecg_epochs will be huge, assuming<br>
> you find many events.<br>
> Very often only a few are necessary to do the detection. Have you tried<br>
> picking the 100-200 first events?<br>
><br>
> We'll have a closer look soon.<br>
> Denis<br>
><br>
> On Wed, Nov 25, 2015 at 3:47 PM, Mads Jensen <<a href="mailto:mje.mads@gmail.com">mje.mads@gmail.com</a><br>
</span><span class="">> <mailto:<a href="mailto:mje.mads@gmail.com">mje.mads@gmail.com</a>>> wrote:<br>
><br>
> Hi,<br>
><br>
> I use sklearn 0.17 (from anaconda). I have tried to the<br>
> "decim" param. I remember it as being "3" for data with 1000Hz<br>
> sfreq. But it didn't help much.<br>
><br>
> I have attach a script to show how I used it.<br>
><br>
> cheers,<br>
> mads<br>
><br>
><br>
><br>
> On 25/11/15 15:29, Denis-Alexander Engemann wrote:<br>
><br>
> Hi Mads,<br>
><br>
> Which version of sklearn are you using?<br>
> Do you use the decim parameter for ICA?<br>
> How do axactly do you use ICA?<br>
> 50GB of memory is unexpected, it would mean that you make up to 10<br>
> copies of your data.<br>
><br>
><br>
> On Wed, Nov 25, 2015 at 3:24 PM, Mads Jensen <<a href="mailto:mje.mads@gmail.com">mje.mads@gmail.com</a><br>
> <mailto:<a href="mailto:mje.mads@gmail.com">mje.mads@gmail.com</a>><br>
</span><span class="">> <mailto:<a href="mailto:mje.mads@gmail.com">mje.mads@gmail.com</a> <mailto:<a href="mailto:mje.mads@gmail.com">mje.mads@gmail.com</a>>>> wrote:<br>
><br>
> Hi all,<br>
><br>
> I would like to hear what people do to filter and run ICA<br>
> and if there<br>
> is any advise.<br>
><br>
> We usually have around an hour of recording which gives<br>
> ~4.5 to 5GB of<br>
> raw fiff files. First filtering and then running ICA in<br>
> MNE-python<br>
> requires a lot of memory, sometimes as much as 50GB. So, I<br>
> fairly often<br>
> get a memory error.<br>
><br>
> I would prefer not to downsample at this stage in the<br>
> process. So, I<br>
> kindly ask if anybody has any thoughts and/or practises to<br>
> avoid very<br>
> heavy memory use.<br>
><br>
> best wishes,<br>
> mads<br>
> _______________________________________________<br>
> Mne_analysis mailing list<br>
> <a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a><br>
> <mailto:<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a>><br>
</span>> <mailto:<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a><br>
<div class="HOEnZb"><div class="h5">> <mailto:<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a>>><br>
> <a href="https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis" rel="noreferrer" target="_blank">https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis</a><br>
><br>
><br>
> The information in this e-mail is intended only for the<br>
> person to<br>
> whom it is<br>
> addressed. If you believe this e-mail was sent to you in<br>
> error and<br>
> the e-mail<br>
> contains patient information, please contact the Partners<br>
> Compliance<br>
> HelpLine at<br>
> <a href="http://www.partners.org/complianceline" rel="noreferrer" target="_blank">http://www.partners.org/complianceline</a> . If the e-mail was sent to<br>
> you in error<br>
> but does not contain patient information, please contact<br>
> the sender<br>
> and properly<br>
> dispose of the e-mail.<br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Mne_analysis mailing list<br>
> <a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a><br>
> <mailto:<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a>><br>
> <a href="https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis" rel="noreferrer" target="_blank">https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis</a><br>
><br>
><br>
> The information in this e-mail is intended only for the person<br>
> to whom it is<br>
> addressed. If you believe this e-mail was sent to you in error<br>
> and the e-mail<br>
> contains patient information, please contact the Partners<br>
> Compliance HelpLine at<br>
> <a href="http://www.partners.org/complianceline" rel="noreferrer" target="_blank">http://www.partners.org/complianceline</a> . If the e-mail was sent<br>
> to you in error<br>
> but does not contain patient information, please contact the<br>
> sender and properly<br>
> dispose of the e-mail.<br>
><br>
><br>
> _______________________________________________<br>
> Mne_analysis mailing list<br>
> <a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a><br>
> <mailto:<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a>><br>
> <a href="https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis" rel="noreferrer" target="_blank">https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis</a><br>
><br>
><br>
> The information in this e-mail is intended only for the person to<br>
> whom it is<br>
> addressed. If you believe this e-mail was sent to you in error and<br>
> the e-mail<br>
> contains patient information, please contact the Partners Compliance<br>
> HelpLine at<br>
> <a href="http://www.partners.org/complianceline" rel="noreferrer" target="_blank">http://www.partners.org/complianceline</a> . If the e-mail was sent to<br>
> you in error<br>
> but does not contain patient information, please contact the sender<br>
> and properly<br>
> dispose of the e-mail.<br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Mne_analysis mailing list<br>
> <a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a><br>
> <a href="https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis" rel="noreferrer" target="_blank">https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis</a><br>
><br>
><br>
> The information in this e-mail is intended only for the person to whom it is<br>
> addressed. If you believe this e-mail was sent to you in error and the e-mail<br>
> contains patient information, please contact the Partners Compliance HelpLine at<br>
> <a href="http://www.partners.org/complianceline" rel="noreferrer" target="_blank">http://www.partners.org/complianceline</a> . If the e-mail was sent to you in error<br>
> but does not contain patient information, please contact the sender and properly<br>
> dispose of the e-mail.<br>
><br>
_______________________________________________<br>
Mne_analysis mailing list<br>
<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a><br>
<a href="https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis" rel="noreferrer" target="_blank">https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis</a><br>
</div></div></blockquote></div><br></div>