[Mne_analysis] Mne_analysis Digest, Vol 150, Issue 44

Fri Jul 24 13:52:25 EDT 2020

        External Email - Use Caution        

It's hard to know for sure why Epochs are getting dropped because your example code doesn't include the definition of `dataconv1` (nor `event_id`). However, with your given code you are guaranteed to have the first epoch get dropped, because your call to `make_fixed_length_events` starts at zero, but you're trying to make epochs with `tmin=-0.2`. If your first event is at time zero, there isn't any data before that, so the first epoch will always get dropped with error "NO_DATA". If you change the `start` parameter of `make_fixed_length_events` to be >0.2, that won't happen.

Here is a (simplified) version of your code that uses fake data (random numbers); when I run this I see the first epoch dropped because "NO_DATA" and the last epoch dropped because "TOO_SHORT", but the other 8 epochs in the middle are retained. I skipped defining `picks` (your example was picking all channels, which is the default anyway) and I skipped filtering because I'm using fake data so there's no way to know from my code whether filtering is influencing the epoch dropping of your real data.

```
import numpy as np
import mne
rng = np.random.default_rng()
channames = ['Cz','Fz','C3','C4','F3','F4','P7','P8']
ch_types = ['eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg']
sfreq = 250 # Hz for eeg
# fake data
dataconv1 = rng.normal(size=(len(channames), 40 * sfreq)) # 40 seconds of data
# Create the info structure needed by MNE
info = mne.create_info(ch_names=channames, sfreq=sfreq, ch_types=ch_types)
raw = mne.io.RawArray(dataconv1, info)
evenfile = mne.make_fixed_length_events(raw, id=1, start=0, stop=None,
duration=4.0, first_samp=True,
overlap=0.0)
event_id = dict(foo=1)
epochs = mne.Epochs(raw, evenfile, event_id=event_id, tmin=-0.2, tmax=4,
baseline=(-0.2, 0), event_repeated='drop',
proj=False, preload=True, reject_by_annotation=None)
print(epochs.drop_log)
```

-- dan
Daniel McCloy
https://dan.mccloy.info
Research Scientist
Institute for Learning and Brain Sciences
University of Washington

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, July 24, 2020 5:44 AM, Ben Ighoyota Ajenaghughrure <ighoyota at tlu.ee> wrote:

> External Email - Use Caution
>
> Dear All,
>
> The response from epochs.drop_log is [['NO_DATA']], yet i get the message "1 bad epochs dropped"
>
> Is there anyway to figure out why all my epochs are being dropped?
>
> source code below
> channames = ['Cz','Fz','C3','C4','F3','F4','P7','P8']
> ch_types = ['eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg']
> # Sampling rate of the Nautilus machine
> sfreq = 250 # Hz for eeg
> # Create the info structure needed by MNE
> info = mne.create_info(ch_names=channames, sfreq=sfreq, ch_types=ch_types)
> raw = mne.io.RawArray(dataconv1, info)
> raw.set_annotations(None)
> raw.del_proj()
> picks =mne.pick_types(raw.info, meg=False, eeg=True, stim=True, eog=False,exclude='bads')
> raw=raw.filter(0.1, 120, fir_design='firwin')
> raw=raw.notch_filter(np.arange(50, 120, 50), picks=picks, filter_length='auto', phase='zero')
> evenfile=mne.make_fixed_length_events(raw, id=1, start=0, stop=None, duration=4.0, first_samp=True, overlap=0.0)
> epochs = mne.Epochs(raw, evenfile, event_id=event_id, tmin=-0.2, tmax=4, picks=picks, baseline=(-0.2, 0), event_repeated='drop',proj=False, preload=True, reject_by_annotation=None)
> print(epochs.drop_log)
> data = epochs.get_data()
>
> A. Ighoyota ben
> Junior Researcher HCI (PhD in-view)
> Tallinn University, Estonia
> School of digital Technologies.
> mobile:[+372582](tel:+372%205832%206393)78794
> skype: ighoyota-ben
>
> On Fri, 24 Jul 2020 at 04:58, <mne_analysis-request at nmr.mgh.harvard.edu> wrote:
>
>> Send Mne_analysis mailing list submissions to
>> mne_analysis at nmr.mgh.harvard.edu
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>> or, via email, send a message with subject or body 'help' to
>> mne_analysis-request at nmr.mgh.harvard.edu
>>
>> You can reach the person managing the list at
>> mne_analysis-owner at nmr.mgh.harvard.edu
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Mne_analysis digest..."
>>
>> Today's Topics:
>>
>> 1. Re: epoch dropped problem (Alexandre Gramfort)
>> 2. Re: loading the subject from BRAINSTORM (Alexandre Gramfort)
>> 3. Re: Mne_analysis Digest, Vol 150, Issue 40 (balandongiv at gmail.com)
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Thu, 23 Jul 2020 22:57:00 +0200
>> From: Alexandre Gramfort <alexandre.gramfort at inria.fr>
>> Subject: Re: [Mne_analysis] epoch dropped problem
>> To: Discussion and support forum for the users of MNE Software
>> <mne_analysis at nmr.mgh.harvard.edu>
>> Message-ID:
>> <CADeotZoS51gPzBn+oLTxCnLTgtpGy=1DyWrFEnOhfM8Ouuevjg at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> External Email - Use Caution
>>
>> What do epochs.drop_log and epochs.plot_drop_log() give you?
>>
>> Alex
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Thu, 23 Jul 2020 22:58:15 +0200
>> From: Alexandre Gramfort <alexandre.gramfort at inria.fr>
>> Subject: Re: [Mne_analysis] loading the subject from BRAINSTORM
>> To: Discussion and support forum for the users of MNE Software
>> <mne_analysis at nmr.mgh.harvard.edu>
>> Message-ID:
>> <CADeotZrjoB0oSOmbXpzESX-_39Dwmqr6VNiSobMqg_043nOD9w at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> External Email - Use Caution
>>
>> hi,
>>
>> no it's not that simple as Brainstorm decimates the cortex while MNE
>> subsamples the high resolution mesh.
>>
>> Alex
>>
>> On Thu, Jul 23, 2020 at 4:06 PM Abdallah Qusaibe
>> <abdallah.qusaibe at gmail.com> wrote:
>>>
>>> External Email - Use Caution
>>>
>>> Hi All,
>>>
>>> Can we load in mne the subject (inorder to use the cortex mesh) used in Brainstorm,
>>>
>>> Cheers
>>> Abdallah
>>> _______________________________________________
>>> Mne_analysis mailing list
>>> Mne_analysis at nmr.mgh.harvard.edu
>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Fri, 24 Jul 2020 09:55:33 +0800
>> From: <balandongiv at gmail.com>
>> Subject: Re: [Mne_analysis] Mne_analysis Digest, Vol 150, Issue 40
>> To: <mne_analysis at nmr.mgh.harvard.edu>
>> Message-ID: <004e01d6615d$85632b50$902981f0$@gmail.com>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> External Email - Use Caution
>>
>> Dear Clemens, Larson, Phillip,
>>
>> Thanks for the detail explanation, really appreciate it.
>>
>> Just a suggestion, maybe part of the discussion can be incorporated
>> somewhere along with mne FAQ or equivalent. This might be helpful,
>> especially to those new in the field.
>>
>> Rodney
>>
>> -----Original Message-----
>> From: mne_analysis-bounces at nmr.mgh.harvard.edu
>> <mne_analysis-bounces at nmr.mgh.harvard.edu> On Behalf Of
>> mne_analysis-request at nmr.mgh.harvard.edu
>> Sent: Thursday, 23 July, 2020 9:10 PM
>> To: mne_analysis at nmr.mgh.harvard.edu
>> Subject: Mne_analysis Digest, Vol 150, Issue 40
>>
>> Send Mne_analysis mailing list submissions to
>> mne_analysis at nmr.mgh.harvard.edu
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>> or, via email, send a message with subject or body 'help' to
>> mne_analysis-request at nmr.mgh.harvard.edu
>>
>> You can reach the person managing the list at
>> mne_analysis-owner at nmr.mgh.harvard.edu
>>
>> When replying, please edit your Subject line so it is more specific than
>> "Re: Contents of Mne_analysis digest..."
>>
>> Today's Topics:
>>
>> 1. Re: Why does MNE resample method does not sample the data
>> point to point? (Eric Larson)
>> 2. Re: Why does MNE resample method does not sample the data
>> point to point? (Brunner, Clemens (clemens.brunner at uni-graz.at))
>> 3. Re: Why does MNE resample method does not sample the data
>> point to point? (Phillip Alday)
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Thu, 23 Jul 2020 08:40:56 -0400
>> From: Eric Larson <larson.eric.d at gmail.com>
>> Subject: Re: [Mne_analysis] Why does MNE resample method does not
>> sample the data point to point?
>> To: Discussion and support forum for the users of MNE Software
>> <mne_analysis at nmr.mgh.harvard.edu>
>> Message-ID:
>> <[CAGu2niVV+79nq5Yu17B4VzB3PS71x7sFTy8HKNukDxnaDdi6hQ at mail.gmail.com](mailto:CAGu2niVV%2B79nq5Yu17B4VzB3PS71x7sFTy8HKNukDxnaDdi6hQ at mail.gmail.com)>
>> Content-Type: text/plain; charset="utf-8"
>>
>> External Email - Use Caution
>>
>>>
>>> My understanding of downsampling is that it is an operation to
>>> decrease the sample rate of x by keeping the first sample and then
>>> every nth sample after the first.
>>>
>>
>> Resampling typically consists of two steps: low-pass filtering to avoid
>> aliasing, then sample rate reduction (subselecting samples from the
>> resulting signal). The low-passing actually changes the values, so the
>> subselection-of-filtered-data step will not necessarily yield points that
>> were "on" the original signal.
>>
>>> May I know whether this issue is due to the ringing artifacts or due
>>> to other problems?
>>>
>>
>> In this case it's likely due to the (implicit) low-pass filtering in the
>> frequency-domain resampling of the signal. It looks pretty reasonable to me.
>> If you want to play around with it a bit, you can
>>
>> 1. Call scipy.signal.resample directly on your data and see how closely it
>> matches.
>> 2. Pad your signal, call scipy.signal.resample, and remove the (now
>> reduced-length) padding -- this is what MNE does internally.
>> 3. Use scipy.signal.resample_poly directly on your data.
>> 4. Manually low-pass filter and then directly subselect samples from the
>> low-passed signal, which is what resample_poly does internally.
>>
>> Hopefully these all give similar results for your signal(s).
>>
>> Eric
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> [http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200723/
>> a8812d4e/attachment-0001.html](http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200723/a8812d4e/attachment-0001.html)
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Thu, 23 Jul 2020 12:57:29 +0000
>> From: "Brunner, Clemens (clemens.brunner at uni-graz.at)"
>> <clemens.brunner at uni-graz.at>
>> Subject: Re: [Mne_analysis] Why does MNE resample method does not
>> sample the data point to point?
>> To: Discussion and support forum for the users of MNE Software
>> <mne_analysis at nmr.mgh.harvard.edu>
>> Message-ID: <21199B34-7BD6-4C1B-81CC-749DFB86E3FC at uni-graz.at>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> External Email - Use Caution
>>
>> Also note that the resample example
>> ([https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample.
>> html](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample.html)) shows upsampling, i.e. the data has a lower sampling rate than the
>> resampled result. However, in the case of downsampling it is usually
>> necessary to avoid aliasing of frequencies above the resampled Nyquist
>> frequency. Therefore, the signal is typically low-pass filtered before the
>> resampling step. As Eric mentioned, this anti-aliasing filter is what
>> actually changes the signal values, but it is necessary to avoid aliasing
>> artifacts.
>>
>> AFAIK, scipy.signal.resample doesn't include an anti-aliasing filter, but
>> both scipy.signal.resample_poly as well as scipy.signal.decimate apply such
>> a low-pass filter before resampling. That's also what MNE does.
>>
>> Clemens
>>
>>> On 23.07.2020, at 14:40, Eric Larson <larson.eric.d at gmail.com> wrote:
>>>
>>> External Email - Use Caution
>>>
>>>
>>> My understanding of downsampling is that it is an operation to decrease
>> the sample rate of x by keeping the first sample and then every nth sample
>> after the first.
>>>
>>> Resampling typically consists of two steps: low-pass filtering to avoid
>> aliasing, then sample rate reduction (subselecting samples from the
>> resulting signal). The low-passing actually changes the values, so the
>> subselection-of-filtered-data step will not necessarily yield points that
>> were "on" the original signal.
>>>
>>> May I know whether this issue is due to the ringing artifacts or due to
>> other problems?
>>>
>>> In this case it's likely due to the (implicit) low-pass filtering in
>>> the frequency-domain resampling of the signal. It looks pretty
>>> reasonable to me. If you want to play around with it a bit, you can
>>>
>>> 1. Call scipy.signal.resample directly on your data and see how closely it
>> matches.
>>> 2. Pad your signal, call scipy.signal.resample, and remove the (now
>> reduced-length) padding -- this is what MNE does internally.
>>> 3. Use scipy.signal.resample_poly directly on your data.
>>> 4. Manually low-pass filter and then directly subselect samples from the
>> low-passed signal, which is what resample_poly does internally.
>>>
>>> Hopefully these all give similar results for your signal(s).
>>>
>>> Eric
>>>
>>> _______________________________________________
>>> Mne_analysis mailing list
>>> Mne_analysis at nmr.mgh.harvard.edu
>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Thu, 23 Jul 2020 15:09:44 +0200
>> From: Phillip Alday <phillip.alday at mpi.nl>
>> Subject: Re: [Mne_analysis] Why does MNE resample method does not
>> sample the data point to point?
>> To: Discussion and support forum for the users of MNE Software
>> <mne_analysis at nmr.mgh.harvard.edu>, "Brunner, Clemens
>> (clemens.brunner at uni-graz.at)" <clemens.brunner at uni-graz.at>
>> Message-ID: <16e39842-34bd-465d-9491-b5651302add4 at mpi.nl>
>> Content-Type: text/plain; charset="utf-8"
>>
>> External Email - Use Caution
>>
>> I think the up- vs. downsampling distinction is also really important for
>> expectations here, as is the distinction between decimating and resampling
>> (I recall there was a thread about that a few years back with similar
>> confusion, if somebody wants to do the effort of searching for it)
>>
>> Phillip
>>
>> On 23/7/20 2:57 pm, Brunner, Clemens (clemens.brunner at uni-graz.at) wrote:
>>> External Email - Use Caution
>>>
>>> Also note that the resample example
>> ([https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample.
>> html](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample.html)) shows upsampling, i.e. the data has a lower sampling rate than the
>> resampled result. However, in the case of downsampling it is usually
>> necessary to avoid aliasing of frequencies above the resampled Nyquist
>> frequency. Therefore, the signal is typically low-pass filtered before the
>> resampling step. As Eric mentioned, this anti-aliasing filter is what
>> actually changes the signal values, but it is necessary to avoid aliasing
>> artifacts.
>>>
>>> AFAIK, scipy.signal.resample doesn't include an anti-aliasing filter, but
>> both scipy.signal.resample_poly as well as scipy.signal.decimate apply such
>> a low-pass filter before resampling. That's also what MNE does.
>>>
>>> Clemens
>>>
>>>
>>>> On 23.07.2020, at 14:40, Eric Larson <larson.eric.d at gmail.com> wrote:
>>>>
>>>> External Email - Use Caution
>>>>
>>>>
>>>> My understanding of downsampling is that it is an operation to decrease
>> the sample rate of x by keeping the first sample and then every nth sample
>> after the first.
>>>>
>>>> Resampling typically consists of two steps: low-pass filtering to avoid
>> aliasing, then sample rate reduction (subselecting samples from the
>> resulting signal). The low-passing actually changes the values, so the
>> subselection-of-filtered-data step will not necessarily yield points that
>> were "on" the original signal.
>>>>
>>>> May I know whether this issue is due to the ringing artifacts or due to
>> other problems?
>>>>
>>>> In this case it's likely due to the (implicit) low-pass filtering in
>>>> the frequency-domain resampling of the signal. It looks pretty
>>>> reasonable to me. If you want to play around with it a bit, you can
>>>>
>>>> 1. Call scipy.signal.resample directly on your data and see how closely
>> it matches.
>>>> 2. Pad your signal, call scipy.signal.resample, and remove the (now
>> reduced-length) padding -- this is what MNE does internally.
>>>> 3. Use scipy.signal.resample_poly directly on your data.
>>>> 4. Manually low-pass filter and then directly subselect samples from the
>> low-passed signal, which is what resample_poly does internally.
>>>>
>>>> Hopefully these all give similar results for your signal(s).
>>>>
>>>> Eric
>>>>
>>>> _______________________________________________
>>>> Mne_analysis mailing list
>>>> Mne_analysis at nmr.mgh.harvard.edu
>>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>
>>>
>>> _______________________________________________
>>> Mne_analysis mailing list
>>> Mne_analysis at nmr.mgh.harvard.edu
>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> ------------------------------
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> End of Mne_analysis Digest, Vol 150, Issue 40
>> *********************************************
>>
>> ------------------------------
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> End of Mne_analysis Digest, Vol 150, Issue 44
>> *********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20200724/b0bc14eb/attachment-0001.html 