[Mne_analysis] Resampling with TMS artifacts

Alexandre Gramfort alexandre.gramfort at inria.fr
Wed Feb 14 15:22:42 EST 2018
Search archives: 
we load the data as float64 for stable numerics so you have data of 30GB in
float32
then yes it will blow up...

I will crop the raw, extract epochs in each chunk and then eventually
concatenate
the epochs.

HTH
Alex

On Wed, Feb 14, 2018 at 9:11 PM, Rockhill, Alexander P. <
AROCKHILL at mgh.harvard.edu> wrote:

> Hi Alex,
>
>     The raw .fif file is 30 GB and I have 60 GB of RAM. When I load the
> epochs, it takes up more than the 60 GB I have of RAM. I have tried
> iterating over the epochs, loading each section and then putting them back
> into an EpochsArray but then when I go to read them in I get the following
> error:
>
>    Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/autofs/space/karima_001/users/alex/software/anaconda2.
> 7/lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/epochs.py",
> line 448, in load_data
>     self._data = self._get_data()
>   File "<string>", line 2, in _get_data
>   File "/autofs/space/karima_001/users/alex/software/anaconda2.
> 7/lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/utils.py", line
> 728, in verbose
>     return function(*args, **kwargs)
>   File "/autofs/space/karima_001/users/alex/software/anaconda2.
> 7/lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/epochs.py",
> line 1170, in _get_data
>     epoch_noproj = self._get_epoch_from_raw(idx)
>   File "<string>", line 2, in _get_epoch_from_raw
>   File "/autofs/space/karima_001/users/alex/software/anaconda2.
> 7/lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/utils.py", line
> 728, in verbose
>     return function(*args, **kwargs)
>   File "/autofs/space/karima_001/users/alex/software/anaconda2.
> 7/lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/epochs.py",
> line 2606, in _get_epoch_from_raw
>     raise RuntimeError('Correct epoch could not be found, please '
> RuntimeError: Correct epoch could not be found, please contact mne-python
> developers
>
> ------------------------------
> *From:* mne_analysis-bounces at nmr.mgh.harvard.edu [
> mne_analysis-bounces at nmr.mgh.harvard.edu] on behalf of Alexandre Gramfort
> [alexandre.gramfort at inria.fr]
> *Sent:* Wednesday, February 14, 2018 3:04 PM
>
> *To:* Discussion and support forum for the users of MNE Software
> *Subject:* Re: [Mne_analysis] Resampling with TMS artifacts
>
> to interpolate the artifact you need to load the data from disk.
>
> it seems that your data are big and you cannot load them in memory.
>
> how big are your data? How much RAM do you have on your machine?
>
> Alex
>
> On Wed, Feb 14, 2018 at 8:11 PM, Rockhill, Alexander P. <
> AROCKHILL at mgh.harvard.edu> wrote:
>
>> Hi Mainak,
>>
>>     Thank you for the suggestion. When I load in the raw data using the
>> string preload option, there isn't a problem, but when try giving a string
>> for preload to the epochs, I get:
>>
>> Traceback (most recent call last):
>>   File "<stdin>", line 4, in <module>
>>   File "<string>", line 2, in __init__
>>   File "/autofs/space/karima_001/users/alex/software/anaconda2.7/
>> lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/utils.py", line
>> 728, in verbose
>>     return function(*args, **kwargs)
>>   File "/autofs/space/karima_001/users/alex/software/anaconda2.7/
>> lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/epochs.py", line
>> 1960, in __init__
>>     verbose=verbose)
>>   File "/autofs/space/karima_001/users/alex/software/anaconda2.7/
>> lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/epochs.py", line
>> 386, in __init__
>>     self.load_data()  # this will do the projection
>>   File "/autofs/space/karima_001/users/alex/software/anaconda2.7/
>> lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/epochs.py", line
>> 448, in load_data
>>     self._data = self._get_data()
>>   File "<string>", line 2, in _get_data
>>   File "/autofs/space/karima_001/users/alex/software/anaconda2.7/
>> lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/utils.py", line
>> 728, in verbose
>>     return function(*args, **kwargs)
>>   File "/autofs/space/karima_001/users/alex/software/anaconda2.7/
>> lib/python2.7/site-packages/mne-0.16.dev0-py2.7.egg/mne/epochs.py", line
>> 1212, in _get_data
>>     dtype=epoch_out.dtype, order='C')
>> MemoryError
>>
>>
>>
>> In order to interpolate the artifact and downsample the data, I am
>> following recommendations that the data be in epochs. Is there any
>> suggestions for how to avoid this?
>>
>> Thanks,
>>
>> Alex
>> ------------------------------
>> *From:* mne_analysis-bounces at nmr.mgh.harvard.edu [
>> mne_analysis-bounces at nmr.mgh.harvard.edu] on behalf of Mainak Jas [
>> mainakjas at gmail.com]
>> *Sent:* Wednesday, February 14, 2018 11:29 AM
>> *To:* Discussion and support forum for the users of MNE Software
>> *Subject:* Re: [Mne_analysis] Resampling with TMS artifacts
>>
>> Hi,
>>
>> How long is the artifact? Probably you could use fix_stim_artifact for
>> this: https://mne-tools.github.io/dev/generated/mne.preprocessing.
>> fix_stim_artifact.html
>>
>> or use decim parameter when epoching
>>
>> Mainak
>>
>> On Wed, Feb 14, 2018 at 3:49 PM, Rockhill, Alexander P. <
>> AROCKHILL at mgh.harvard.edu> wrote:
>>
>>> Hi all,
>>>
>>>     I was wondering about the best way to resample data with a
>>> high-frequency TMS artifact. I first epoched the raw data so that the TMS
>>> events would be kept but when I resample the epochs, a low-pass filter is
>>> applied which makes sense but causes a large filter artifact from the TMS
>>> pulse. Is there any way to not apply this filter?
>>>
>>> Thanks,
>>>
>>> Alex
>>>
>>> _______________________________________________
>>> Mne_analysis mailing list
>>> Mne_analysis at nmr.mgh.harvard.edu
>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>
>>>
>>> The information in this e-mail is intended only for the person to whom
>>> it is
>>> addressed. If you believe this e-mail was sent to you in error and the
>>> e-mail
>>> contains patient information, please contact the Partners Compliance
>>> HelpLine at
>>> http://www.partners.org/complianceline . If the e-mail was sent to you
>>> in error
>>> but does not contain patient information, please contact the sender and
>>> properly
>>> dispose of the e-mail.
>>>
>>>
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>>
>> The information in this e-mail is intended only for the person to whom it
>> is
>> addressed. If you believe this e-mail was sent to you in error and the
>> e-mail
>> contains patient information, please contact the Partners Compliance
>> HelpLine at
>> http://www.partners.org/complianceline . If the e-mail was sent to you
>> in error
>> but does not contain patient information, please contact the sender and
>> properly
>> dispose of the e-mail.
>>
>>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to whom it
> is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20180214/7e75d37e/attachment.html

More information about the Mne_analysis mailing list