[Mne_analysis] x----SPAM----xRe: epochs with no data?

Alexandre Gramfort alexandre.gramfort at inria.fr
Tue Jan 15 08:26:19 EST 2019
Search archives: 
        External Email - Use Caution        

hi,

can you share a dataset to replicate this and we'll see how to
make this fix available for everyone.


A

On Tue, Jan 15, 2019 at 11:45 AM HEINE LIZETTE
<lizette.heine at univ-lyon1.fr> wrote:
>
>         External Email - Use Caution
>
> Hi Dillan and the rest,
>
>
> I myself experienced an egi event exclusion problem and believe to have found a fix. I think (but am not completely sure) this might be the same issue as you are experiencing.
>
>
> My data is recorded with 89 different events, and MNE will consistently exclude one or multiple events from the data, which of course affects further processing. This is happening since version 15. I noted this once in an issue on GitHub but this got burried under other issues.
>
> If this fix works for you, I will try to open an issue again and maybe it can get fixed in the next version of MNE.
>
>
>
> What worked is to change line 238 of egimff.py from
>
> if event.sum() <= 1 and event_codes[ii]:   to
>
> if event.sum() <= 0 and event_codes[ii]:
>
>
> if MNE is installed via anaconda you can probably find this file here:
>
> C:\Users\ "your account" \Anaconda3\envs\mne\Lib\site-packages\mne\io\egi
>
>
> Greetings,
>
> Lizette Heine
>
>
> ps. First time posting on the mailinglist, sorry if this is not the usual way of doing things.
>
>
>
> ________________________________
> De : mne_analysis-bounces at nmr.mgh.harvard.edu <mne_analysis-bounces at nmr.mgh.harvard.edu> de la part de Dillan Cellier <cellierdillan at gmail.com>
> Envoyé : jeudi 10 janvier 2019 22:33
> À : Discussion and support forum for the users of MNE Software
> Objet : x----SPAM----xRe: [Mne_analysis] epochs with no data?
>
>
>         External Email - Use Caution
>
> Hi all,
>
>
>
> I've written a previous email explaining in more detail the process leading up to this epoching issue, but it seems that the email has been hung up waiting for moderator approval. I'll start from the beginning:
>
>
>
> Here is where I load up the raw data, which was imported from the CMI database in a .raw format:
>
>
>
> raw=mne.io.read_raw_egi (raw_file_path, montage= mne.channels.read_montage(kind= 'GSM-HydrocCel-129'), preload=True, verbose= True)
>
>
> Output:
>
> Reading EGI header from [raw_file_path] ....
>
> Reading events..
>
> Assembling measurement info...
>
> Synthesizing trigger channel 'STI 014' ...
>
> Excluding events {101,102,103.....[etc] } (the events listed as being excluded here have the same labels as the nonsense channels talked about below-- not sure what to make of that)
>
> Reading 0 .... 168248 = 0.000 ... 336.496 secs ....
>
>
>
> Upon importing the raw file there are quite a few channels that seem to be not 'real' scalp or external electrodes included in the raw object. One of them, 'STI 014,' is useful for reading the eyes closed/eyes open events but there are about 50 other channels that contain no data and are randomly labeled.
>
>
>
> raw.info['ch_names'][129:] yields a list of these nonsensical channels: [ '101' , '102' , '3' , '11' ,........]
>
>
>
> I split up and re-concatenate the raw dataset into its respective eyes-open and eyes-closed sections, and proceed from there with 2 sets of raw data.
>
>
>
> After going through several standard preprocessing steps (filtering, re-referencing, ICA) without a hitch, I run into my issue with epoching the data into 2 seconds.
>
>
>
> The sample frequency is 500 Hz, so I construct an array of trigger events by sampling every 1000th index and appending it to the array with a made-up trigger code of 7.
>
>
>
> print(epoch_array) yields [ [ 0, 0, 7], [1000, 0, 7], [2000, 0, 7], [ 3000, 0, 7].....]
>
>
>
> I use this array to construct the epoch object:
>
>
>
> twoSec_epoch = mne.Epochs(eyes_closed_raw, events=epoch_array, tmin=0, tmax=0, event_id={'twoSec':7}, reject=None, flat=None, reject_by_annotation=False)
>
>
> Output:
>
> 93 matching events found
>
> Applying baseline correction (mode: mean)
>
> Not setting metadata
>
> 0 projection items activated
>
>
>
> And of the 93 epochs in twoSec_epoch, the first 28 of them are being dropped seemingly without reason.
>
>
> twoSec_epoch.get_data()
>
>
> Output:
>
> Loading data for 93 events and 1001 original time points ...
>
> 28 bad epochs dropped
>
>
> This happens whether or not the empty channels are dropped from the raw object before creating the epoch object.
>
>
>
> twoSec_epoch.drop_log[:28] yields [ ['NO_DATA'], ['NO_DATA'], ['NO_DATA'].....]
>
>
>
> But I know that the 'real' electrodes (with the random fake channels dropped) do contain data for these epochs:
>
>
>
> raw_data_mat= eyes_closed_raw.get_data()
>
> first_28_epochs= raw_data_mat[:128, 0:28*1000]
>
> print(sum(sum(np.isnan(first_28_epochs))))
>
> print(sum(sum(first_28_epochs==0)))
>
> yields 0 and 0, ie there is data for each of the 128 channels.
>
>
>
> I am unsure whether the issue is with the epoch_array I'm making or with something else about this particular dataset. Any help/guesses would be greatly appreciated!
>
>
>
> Thank you and best wishes,
>
> Dillan Cellier
>
>
> On Mon, Jan 7, 2019 at 10:56 AM Molfese, Peter (NIH/NIMH) [C] <peter.molfese at nih.gov> wrote:
>>
>>         External Email - Use Caution
>>
>> Hi Dillan-
>>
>>
>>
>> How are you reading in the data?  From the EGI formatted file?  Can you provide more of the code?
>>
>>
>>
>> --
>>
>>
>>
>> Peter J. Molfese, Ph.D. [C] | Scientist, NIMH
>> Section on Functional Imaging Methods
>> Building 10, 1D80
>> 10 Center Dr
>> Bethesda, MD 20892-1148
>> Email: peter.molfese at mail.nih.gov
>> Phone: (301) 402-1350
>>
>>
>>
>>
>>
>> From: Dillan Cellier <cellierdillan at gmail.com>
>> Reply-To: Discussion and support forum for the users of MNE Software <mne_analysis at nmr.mgh.harvard.edu>
>> Date: Monday, January 7, 2019 at 11:53 AM
>> To: Discussion and support forum for the users of MNE Software <mne_analysis at nmr.mgh.harvard.edu>
>> Subject: Re: [Mne_analysis] epochs with no data?
>>
>>
>>
>>         External Email - Use Caution
>>
>> Hi Denis,
>>
>>
>>
>> Apologies for the delay in getting back to you. I've tried reproducing this with a MNE sample dataset, and did not get the same effect. It would appear then, that there might be something special about the data. Unfortunately this dataset is not one that I collected. It was acquired from the Child Mind Institute and as such, the extent of my knowledge about it goes only so far as the public information provided on it!
>>
>>
>>
>> I tried this on a different subject's data from the same CMI dataset. I got the same results where the epoch.plot() is dropping over half of the epochs, however this time this the drop_log is empty (!?!?!) and at this point i am thoroughly confused. Here is the code where I am constructing this Epoch object, so that you may see that there shouldn't be any automatic rejection taking place:
>>
>>
>>
>> mne.Epochs(eyes_closed, events=epoch_array, tmin=0, tmax=2, event_id={'twoSec':7}, picks=scalpData, reject_by_annotation=False)
>>
>>
>>
>> Any other guesses? Thank you for your help!
>>
>>
>>
>> Best wishes,
>>
>> Dillan
>>
>>
>>
>> On Fri, Jan 4, 2019 at 3:30 PM Denis A. Engemann <denis-alexander.engemann at inria.fr> wrote:
>>
>>         External Email - Use Caution
>>
>> Hum. That sounds puzzling. Can you reproduce this somehow with the MNE sample data? That could help to see if there is something wrong in the code or if something is special about your data.
>>
>>
>>
>> Best,
>>
>> Denis
>>
>>
>> On 4 Jan 2019, at 22:24, Dillan Cellier <cellierdillan at gmail.com> wrote:
>>
>>         External Email - Use Caution
>>
>> Hi Denis,
>>
>>
>>
>> The events are 2 seconds apart from each other.
>>
>>
>>
>> Thanks!
>>
>>
>>
>> Dillan
>>
>>
>>
>> On Fri, Jan 4, 2019 at 3:18 PM Denis A. Engemann <denis-alexander.engemann at inria.fr> wrote:
>>
>>         External Email - Use Caution
>>
>> Hi Dillan,
>>
>>
>>
>> What is then the distance in seconds between any two events that you passed to the epochs constructor?
>>
>>
>>
>> Denis
>>
>>
>> On 4 Jan 2019, at 22:12, Dillan Cellier <cellierdillan at gmail.com> wrote:
>>
>>         External Email - Use Caution
>>
>> Hi Denis,
>>
>>
>>
>> I limited the tmin and tmax of the epoch to 0 and 2, respectively, since I don't any kind of stimulus onset in these epochs and don't want them to overlap. This is partly what puzzles me, as I would think this would prevent any inclusion of not-real data. My resting state data can be cut into 93 two-second time windows, and this the number of events I am feeding into the epoch object. It is labeling 41 of these 93 as 'NO_DATA'.
>>
>>
>>
>> Thank you very much!
>>
>>
>>
>> Dillan
>>
>>
>>
>> On Fri, Jan 4, 2019 at 3:05 PM Denis A. Engemann <denis-alexander.engemann at inria.fr> wrote:
>>
>>         External Email - Use Caution
>>
>> Hi Dillan,
>>
>>
>>
>> This can happen when the epochs selected include (theoretical) samples beyond the (actual) data range. You should not have many of those epochs. Can you roughly tell how many no_data labels you found? I‘d need to refresh my knowledge of the epochs reading code a bit to see if there may be other reasons for this. Let‘s see what the others say in the meantime.
>>
>>
>>
>> Best,
>>
>> Denis
>>
>>
>> On 4 Jan 2019, at 21:56, Dillan Cellier <cellierdillan at gmail.com> wrote:
>>
>>         External Email - Use Caution
>>
>> Hello again!
>>
>>
>>
>> Thank you for your responses on my last question, they were very helpful. I am running into another problem now, however. I am epoching resting state data into arbitrary 2 second windows. I am not automatically rejecting epochs by annotation nor by a rejection parameter. I see that the epoch object's drop_log is recording nearly the whole first half of the epochs as "NO_DATA" epochs, however the raw file contains data for those time indices. I am unsure what could be going on to result in this loss of data, however it seems especially problematic due to the amount of epochs dropped. Thank you in advance, again!
>>
>>
>>
>> Best wishes,
>>
>> Dillan
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis

More information about the Mne_analysis mailing list