[Mne_analysis] Filtering and ICA memory issues
Denis A. Engemann
denis.engemann at gmail.com
Sat Nov 28 10:03:22 EST 2015
I think this is what I implemented in the repo shared above, it draws uniformly k=200 samples from all the
Epoche if I remember correctly.
> On 28 Nov 2015, at 14:21, dgw <dgwakeman at gmail.com> wrote:
>
> One thing to add to Denis's suggestion about ECG events. I encourage you to use a sample of ~300 plus throughout the recording (i.e. Find the events then take every 10th or something). As the ECG components/SSPs can shift over time depending on participant movement.
> Hth
> D
>
>
> Sent from my Phone
>
>> On Nov 25, 2015, at 11:54, Denis-Alexander Engemann <denis.engemann at gmail.com> wrote:
>>
>> Hi Mads,
>>
>> it seems you don't use the decim parameter, do you?
>> Two things that I see immediately:
>> It should in fact save a lot. With 1000HZ you can decimate even more, for ECG/EOG your sampling frequency should not be lower than 50 or so.
>> Second as you have 1 hour of data the ecg_epochs will be huge, assuming you find many events.
>> Very often only a few are necessary to do the detection. Have you tried picking the 100-200 first events?
>>
>> We'll have a closer look soon.
>> Denis
>>
>>> On Wed, Nov 25, 2015 at 3:47 PM, Mads Jensen <mje.mads at gmail.com> wrote:
>>> Hi,
>>>
>>> I use sklearn 0.17 (from anaconda). I have tried to the
>>> "decim" param. I remember it as being "3" for data with 1000Hz sfreq. But it didn't help much.
>>>
>>> I have attach a script to show how I used it.
>>>
>>> cheers,
>>> mads
>>>
>>>
>>>
>>>> On 25/11/15 15:29, Denis-Alexander Engemann wrote:
>>>> Hi Mads,
>>>>
>>>> Which version of sklearn are you using?
>>>> Do you use the decim parameter for ICA?
>>>> How do axactly do you use ICA?
>>>> 50GB of memory is unexpected, it would mean that you make up to 10
>>>> copies of your data.
>>>>
>>>>
>>>> On Wed, Nov 25, 2015 at 3:24 PM, Mads Jensen <mje.mads at gmail.com
>>>> <mailto:mje.mads at gmail.com>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I would like to hear what people do to filter and run ICA and if there
>>>> is any advise.
>>>>
>>>> We usually have around an hour of recording which gives ~4.5 to 5GB of
>>>> raw fiff files. First filtering and then running ICA in MNE-python
>>>> requires a lot of memory, sometimes as much as 50GB. So, I fairly often
>>>> get a memory error.
>>>>
>>>> I would prefer not to downsample at this stage in the process. So, I
>>>> kindly ask if anybody has any thoughts and/or practises to avoid very
>>>> heavy memory use.
>>>>
>>>> best wishes,
>>>> mads
>>>> _______________________________________________
>>>> Mne_analysis mailing list
>>>> Mne_analysis at nmr.mgh.harvard.edu
>>>> <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>>
>>>>
>>>> The information in this e-mail is intended only for the person to
>>>> whom it is
>>>> addressed. If you believe this e-mail was sent to you in error and
>>>> the e-mail
>>>> contains patient information, please contact the Partners Compliance
>>>> HelpLine at
>>>> http://www.partners.org/complianceline . If the e-mail was sent to
>>>> you in error
>>>> but does not contain patient information, please contact the sender
>>>> and properly
>>>> dispose of the e-mail.
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mne_analysis mailing list
>>>> Mne_analysis at nmr.mgh.harvard.edu
>>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>>
>>>>
>>>> The information in this e-mail is intended only for the person to whom it is
>>>> addressed. If you believe this e-mail was sent to you in error and the e-mail
>>>> contains patient information, please contact the Partners Compliance HelpLine at
>>>> http://www.partners.org/complianceline . If the e-mail was sent to you in error
>>>> but does not contain patient information, please contact the sender and properly
>>>> dispose of the e-mail.
>>>
>>> _______________________________________________
>>> Mne_analysis mailing list
>>> Mne_analysis at nmr.mgh.harvard.edu
>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>
>>>
>>> The information in this e-mail is intended only for the person to whom it is
>>> addressed. If you believe this e-mail was sent to you in error and the e-mail
>>> contains patient information, please contact the Partners Compliance HelpLine at
>>> http://www.partners.org/complianceline . If the e-mail was sent to you in error
>>> but does not contain patient information, please contact the sender and properly
>>> dispose of the e-mail.
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>>
>> The information in this e-mail is intended only for the person to whom it is
>> addressed. If you believe this e-mail was sent to you in error and the e-mail
>> contains patient information, please contact the Partners Compliance HelpLine at
>> http://www.partners.org/complianceline . If the e-mail was sent to you in error
>> but does not contain patient information, please contact the sender and properly
>> dispose of the e-mail.
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the e-mail
> contains patient information, please contact the Partners Compliance HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in error
> but does not contain patient information, please contact the sender and properly
> dispose of the e-mail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20151128/ad5ef3c2/attachment-0001.html
More information about the Mne_analysis
mailing list