<div dir="ltr">Hi Mads,<div><br></div><div>it seems you don't use the decim parameter, do you?</div><div>Two things that I see immediately:</div><div>It should in fact save a lot. With 1000HZ you can decimate even more, for ECG/EOG your sampling frequency should not be lower than 50 or so.</div><div>Second as you have 1 hour of data the ecg_epochs will be huge, assuming you find many events.</div><div>Very often only a few are necessary to do the detection. Have you tried picking the 100-200 first events?</div><div><br></div><div>We'll have a closer look soon.</div><div>Denis</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 25, 2015 at 3:47 PM, Mads Jensen <span dir="ltr"><<a href="mailto:mje.mads@gmail.com" target="_blank">mje.mads@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I use sklearn 0.17 (from anaconda). I have tried to the<br>
"decim" param. I remember it as being "3" for data with 1000Hz sfreq. But it didn't help much.<br>
<br>
I have attach a script to show how I used it.<br>
<br>
cheers,<br>
mads<span class=""><br>
<br>
<br>
<br>
On 25/11/15 15:29, Denis-Alexander Engemann wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Hi Mads,<br>
<br>
Which version of sklearn are you using?<br>
Do you use the decim parameter for ICA?<br>
How do axactly do you use ICA?<br>
50GB of memory is unexpected, it would mean that you make up to 10<br>
copies of your data.<br>
<br>
<br>
On Wed, Nov 25, 2015 at 3:24 PM, Mads Jensen <<a href="mailto:mje.mads@gmail.com" target="_blank">mje.mads@gmail.com</a><br></span><span class="">
<mailto:<a href="mailto:mje.mads@gmail.com" target="_blank">mje.mads@gmail.com</a>>> wrote:<br>
<br>
Hi all,<br>
<br>
I would like to hear what people do to filter and run ICA and if there<br>
is any advise.<br>
<br>
We usually have around an hour of recording which gives ~4.5 to 5GB of<br>
raw fiff files. First filtering and then running ICA in MNE-python<br>
requires a lot of memory, sometimes as much as 50GB. So, I fairly often<br>
get a memory error.<br>
<br>
I would prefer not to downsample at this stage in the process. So, I<br>
kindly ask if anybody has any thoughts and/or practises to avoid very<br>
heavy memory use.<br>
<br>
best wishes,<br>
mads<br>
_______________________________________________<br>
Mne_analysis mailing list<br>
<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu" target="_blank">Mne_analysis@nmr.mgh.harvard.edu</a><br></span>
<mailto:<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu" target="_blank">Mne_analysis@nmr.mgh.harvard.edu</a>><span class=""><br>
<a href="https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis" rel="noreferrer" target="_blank">https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis</a><br>
<br>
<br>
The information in this e-mail is intended only for the person to<br>
whom it is<br>
addressed. If you believe this e-mail was sent to you in error and<br>
the e-mail<br>
contains patient information, please contact the Partners Compliance<br>
HelpLine at<br>
<a href="http://www.partners.org/complianceline" rel="noreferrer" target="_blank">http://www.partners.org/complianceline</a> . If the e-mail was sent to<br>
you in error<br>
but does not contain patient information, please contact the sender<br>
and properly<br>
dispose of the e-mail.<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Mne_analysis mailing list<br>
<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu" target="_blank">Mne_analysis@nmr.mgh.harvard.edu</a><br>
<a href="https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis" rel="noreferrer" target="_blank">https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis</a><br>
<br>
<br>
The information in this e-mail is intended only for the person to whom it is<br>
addressed. If you believe this e-mail was sent to you in error and the e-mail<br>
contains patient information, please contact the Partners Compliance HelpLine at<br>
<a href="http://www.partners.org/complianceline" rel="noreferrer" target="_blank">http://www.partners.org/complianceline</a> . If the e-mail was sent to you in error<br>
but does not contain patient information, please contact the sender and properly<br>
dispose of the e-mail.<br>
<br>
</span></blockquote>
<br>_______________________________________________<br>
Mne_analysis mailing list<br>
<a href="mailto:Mne_analysis@nmr.mgh.harvard.edu">Mne_analysis@nmr.mgh.harvard.edu</a><br>
<a href="https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis" rel="noreferrer" target="_blank">https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis</a><br>
<br>
<br>
The information in this e-mail is intended only for the person to whom it is<br>
addressed. If you believe this e-mail was sent to you in error and the e-mail<br>
contains patient information, please contact the Partners Compliance HelpLine at<br>
<a href="http://www.partners.org/complianceline" rel="noreferrer" target="_blank">http://www.partners.org/complianceline</a> . If the e-mail was sent to you in error<br>
but does not contain patient information, please contact the sender and properly<br>
dispose of the e-mail.<br>
<br></blockquote></div><br></div>