[Mne_analysis] x----SPAM----xRe: epochs with no data?

Tue Jan 15 12:59:34 EST 2019

        External Email - Use Caution        

Hi Lizette,

I actually ended up solving my issue in another way; the problem was due to
the mismatch between the _first_samps of my custom-constructed raw eyes
open and eyes closed datasets and the time indices I was feeding in to
create my 2 second epochs. I didn't notice a discrepancy between where MNE
was reading the beginning of my files and the event time indices in
epoch_array because when I would .get_data() and manually extract data
using those indices, it was not a problem. Only in conversion to an epoch
object did it begin skipping to the first in _first_samps.

Thank you for this follow-up, though!

Best wishes,
Dillan

On Tue, Jan 15, 2019 at 4:45 AM HEINE LIZETTE <lizette.heine at univ-lyon1.fr>
wrote:

>         External Email - Use Caution
>
> Hi Dillan and the rest,
>
>
> I myself experienced an egi event exclusion problem and believe to have
> found a fix. I think (but am not completely sure) this might be the same
> issue as you are experiencing.
>
>
> My data is recorded with 89 different events, and MNE will consistently
> exclude one or multiple events from the data, which of course affects
> further processing. This is happening since version 15. I noted this once
> in an issue on GitHub but this got burried under other issues.
>
> If this fix works for you, I will try to open an issue again and maybe it
> can get fixed in the next version of MNE.
>
>
>
> What worked is to change line 238 of egimff.py from
>
> if event.sum() <=* 1* and event_codes[ii]:   to
>
> if event.sum() <=* 0* and event_codes[ii]:
>
>
> if MNE is installed via anaconda you can probably find this file here:
>
> C:\Users\ "your account" \Anaconda3\envs\mne\Lib\site-packages\mne\io\egi
>
>
> Greetings,
>
> Lizette Heine
>
>
> ps. First time posting on the mailinglist, sorry if this is not the usual
> way of doing things.
>
>
>
> ------------------------------
> *De :* mne_analysis-bounces at nmr.mgh.harvard.edu <
> mne_analysis-bounces at nmr.mgh.harvard.edu> de la part de Dillan Cellier <
> cellierdillan at gmail.com>
> *Envoyé :* jeudi 10 janvier 2019 22:33
> *À :* Discussion and support forum for the users of MNE Software
> *Objet :* x----SPAM----xRe: [Mne_analysis] epochs with no data?
>
>
>         External Email - Use Caution
>
> Hi all,
>
>
>
> I've written a previous email explaining in more detail the process
> leading up to this epoching issue, but it seems that the email has been
> hung up waiting for moderator approval. I'll start from the beginning:
>
>
>
> Here is where I load up the raw data, which was imported from the CMI
> database in a .raw format:
>
>
>
> raw=mne.io.read_raw_egi (raw_file_path, montage=
> mne.channels.read_montage(kind= 'GSM-HydrocCel-129'), preload=True,
> verbose= True)
>
>
> Output:
>
> Reading EGI header from [raw_file_path] ....
>
> Reading events..
>
> Assembling measurement info...
>
> Synthesizing trigger channel 'STI 014' ...
>
> Excluding events {101,102,103.....[etc] } (the events listed as being
> excluded here have the same labels as the nonsense channels talked about
> below-- not sure what to make of that)
>
> Reading 0 .... 168248 = 0.000 ... 336.496 secs ....
>
>
>
> Upon importing the raw file there are quite a few channels that seem to be
> not 'real' scalp or external electrodes included in the raw object. One of
> them, 'STI 014,' is useful for reading the eyes closed/eyes open events but
> there are about 50 other channels that contain no data and are randomly
> labeled.
>
>
>
> raw.info['ch_names'][129:] yields a list of these nonsensical channels: [
> '101' , '102' , '3' , '11' ,........]
>
>
>
> I split up and re-concatenate the raw dataset into its respective
> eyes-open and eyes-closed sections, and proceed from there with 2 sets of
> raw data.
>
>
>
> After going through several standard preprocessing steps (filtering,
> re-referencing, ICA) without a hitch, I run into my issue with epoching the
> data into 2 seconds.
>
>
>
> The sample frequency is 500 Hz, so I construct an array of trigger events
> by sampling every 1000th index and appending it to the array with a made-up
> trigger code of 7.
>
>
>
> print(epoch_array) yields [ [ 0, 0, 7], [1000, 0, 7], [2000, 0, 7], [
> 3000, 0, 7].....]
>
>
>
> I use this array to construct the epoch object:
>
>
>
> twoSec_epoch = mne.Epochs(eyes_closed_raw, events=epoch_array, tmin=0,
> tmax=0, event_id={'twoSec':7}, reject=None, flat=None,
> reject_by_annotation=False)
>
>
> Output:
>
> 93 matching events found
>
> Applying baseline correction (mode: mean)
>
> Not setting metadata
>
> 0 projection items activated
>
>
>
> And of the 93 epochs in twoSec_epoch, the first 28 of them are being
> dropped seemingly without reason.
>
>
> twoSec_epoch.get_data()
>
>
> Output:
>
> Loading data for 93 events and 1001 original time points ...
>
> 28 bad epochs dropped
>
>
> This happens whether or not the empty channels are dropped from the raw
> object before creating the epoch object.
>
>
>
> twoSec_epoch.drop_log[:28] yields [ ['NO_DATA'], ['NO_DATA'],
> ['NO_DATA'].....]
>
>
>
> But I know that the 'real' electrodes (with the random fake channels
> dropped) do contain data for these epochs:
>
>
>
> raw_data_mat= eyes_closed_raw.get_data()
>
> first_28_epochs= raw_data_mat[:128, 0:28*1000]
>
> print(sum(sum(np.isnan(first_28_epochs))))
>
> print(sum(sum(first_28_epochs==0)))
>
> yields 0 and 0, ie there is data for each of the 128 channels.
>
>
>
> I am unsure whether the issue is with the epoch_array I'm making or with
> something else about this particular dataset. Any help/guesses would be
> greatly appreciated!
>
>
>
> Thank you and best wishes,
>
> Dillan Cellier
>
> On Mon, Jan 7, 2019 at 10:56 AM Molfese, Peter (NIH/NIMH) [C] <
> peter.molfese at nih.gov> wrote:
>
>>         External Email - Use Caution
>>
>> Hi Dillan-
>>
>>
>>
>> How are you reading in the data?  From the EGI formatted file?  Can you
>> provide more of the code?
>>
>>
>>
>> --
>>
>>
>>
>> Peter J. Molfese, Ph.D. [C] | Scientist, NIMH
>> Section on Functional Imaging Methods
>> Building 10, 1D80
>> 10 Center Dr
>> Bethesda, MD 20892-1148
>> Email: peter.molfese at mail.nih.gov
>> Phone: (301) 402-1350
>>
>>
>>
>>
>>
>> *From: *Dillan Cellier <cellierdillan at gmail.com>
>> *Reply-To: *Discussion and support forum for the users of MNE Software <
>> mne_analysis at nmr.mgh.harvard.edu>
>> *Date: *Monday, January 7, 2019 at 11:53 AM
>> *To: *Discussion and support forum for the users of MNE Software <
>> mne_analysis at nmr.mgh.harvard.edu>
>> *Subject: *Re: [Mne_analysis] epochs with no data?
>>
>>
>>
>> *        External Email - Use Caution        *
>>
>> Hi Denis,
>>
>>
>>
>> Apologies for the delay in getting back to you. I've tried reproducing
>> this with a MNE sample dataset, and did not get the same effect. It would
>> appear then, that there might be something special about the data.
>> Unfortunately this dataset is not one that I collected. It was acquired
>> from the Child Mind Institute and as such, the extent of my knowledge about
>> it goes only so far as the public information provided on it!
>>
>>
>>
>> I tried this on a different subject's data from the same CMI dataset. I
>> got the same results where the epoch.plot() is dropping over half of the
>> epochs, however this time this the drop_log is empty (!?!?!) and at this
>> point i am thoroughly confused. Here is the code where I am constructing
>> this Epoch object, so that you may see that there shouldn't be any
>> automatic rejection taking place:
>>
>>
>>
>> mne.Epochs(eyes_closed, events=epoch_array, tmin=0, tmax=2,
>> event_id={'twoSec':7}, picks=scalpData, reject_by_annotation=False)
>>
>>
>>
>> Any other guesses? Thank you for your help!
>>
>>
>>
>> Best wishes,
>>
>> Dillan
>>
>>
>>
>> On Fri, Jan 4, 2019 at 3:30 PM Denis A. Engemann <
>> denis-alexander.engemann at inria.fr> wrote:
>>
>> *        External Email - Use Caution        *
>>
>> Hum. That sounds puzzling. Can you reproduce this somehow with the MNE
>> sample data? That could help to see if there is something wrong in the code
>> or if something is special about your data.
>>
>>
>>
>> Best,
>>
>> Denis
>>
>>
>> On 4 Jan 2019, at 22:24, Dillan Cellier <cellierdillan at gmail.com> wrote:
>>
>> *        External Email - Use Caution        *
>>
>> Hi Denis,
>>
>>
>>
>> The events are 2 seconds apart from each other.
>>
>>
>>
>> Thanks!
>>
>>
>>
>> Dillan
>>
>>
>>
>> On Fri, Jan 4, 2019 at 3:18 PM Denis A. Engemann <
>> denis-alexander.engemann at inria.fr> wrote:
>>
>> *        External Email - Use Caution        *
>>
>> Hi Dillan,
>>
>>
>>
>> What is then the distance in seconds between any two events that you
>> passed to the epochs constructor?
>>
>>
>>
>> Denis
>>
>>
>> On 4 Jan 2019, at 22:12, Dillan Cellier <cellierdillan at gmail.com> wrote:
>>
>> *        External Email - Use Caution        *
>>
>> Hi Denis,
>>
>>
>>
>> I limited the tmin and tmax of the epoch to 0 and 2, respectively, since
>> I don't any kind of stimulus onset in these epochs and don't want them to
>> overlap. This is partly what puzzles me, as I would think this would
>> prevent any inclusion of not-real data. My resting state data can be cut
>> into 93 two-second time windows, and this the number of events I am feeding
>> into the epoch object. It is labeling 41 of these 93 as 'NO_DATA'.
>>
>>
>>
>> Thank you very much!
>>
>>
>>
>> Dillan
>>
>>
>>
>> On Fri, Jan 4, 2019 at 3:05 PM Denis A. Engemann <
>> denis-alexander.engemann at inria.fr> wrote:
>>
>> *        External Email - Use Caution        *
>>
>> Hi Dillan,
>>
>>
>>
>> This can happen when the epochs selected include (theoretical) samples
>> beyond the (actual) data range. You should not have many of those epochs.
>> Can you roughly tell how many no_data labels you found? I‘d need to refresh
>> my knowledge of the epochs reading code a bit to see if there may be other
>> reasons for this. Let‘s see what the others say in the meantime.
>>
>>
>>
>> Best,
>>
>> Denis
>>
>>
>> On 4 Jan 2019, at 21:56, Dillan Cellier <cellierdillan at gmail.com> wrote:
>>
>> *        External Email - Use Caution        *
>>
>> Hello again!
>>
>>
>>
>> Thank you for your responses on my last question, they were very helpful.
>> I am running into another problem now, however. I am epoching resting state
>> data into arbitrary 2 second windows. I am not automatically rejecting
>> epochs by annotation nor by a rejection parameter. I see that the epoch
>> object's drop_log is recording nearly the whole first half of the epochs as
>> "NO_DATA" epochs, however the raw file contains data for those time
>> indices. I am unsure what could be going on to result in this loss of data,
>> however it seems especially problematic due to the amount of epochs
>> dropped. Thank you in advance, again!
>>
>>
>>
>> Best wishes,
>>
>> Dillan
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20190115/c51d3c5b/attachment-0001.html 