[Mne_analysis] "Value too large to be stored in data type" when running mne_mark_bad_channels in tcsh

[Stephen Politzer-Ahles]

Hi Matti,

Thanks for the heads-up. In this case, how can I analyse a large file?
Should I split the raw file into multiple raw files, and then somehow feed
all these raw files into mne_process_raw when I want to create -ave.fif and
-cov.fif files?

Thank you,
Steve



Stephen Politzer-Ahles
New York University, Abu Dhabi
Neuroscience of Language Lab
http://www.nyu.edu/projects/politzer-ahles/


On Mon, Nov 25, 2013 at 1:33 PM, Matti Hamalainen
<msh at nmr.mgh.harvard.edu>wrote:

>
> Hi Steve,
>
> Just FYI, MNE-C does not do the splitting in mne_kit2fiff either.
>
> - Matti
>
>
> On Nov 25, 2013, at 3:39 AM, Alexandre Gramfort <
> alexandre.gramfort at telecom-paristech.fr> wrote:
>
> > hi Steve,
> >
> > looks like you've hit a problem in our python fiff writing code that
> does not
> > splits .fif files to avoid large files.
> >
> > We'll look into it asap
> >
> > Thanks for the report.
> >
> > Best,
> > Alex
> >
> >
> > On Mon, Nov 25, 2013 at 9:34 AM, Stephen Politzer-Ahles <spa268 at nyu.edu>
> wrote:
> >> Hi Alexandre,
> >>
> >> It started as Yokogawa .con data, I wrote it to fif using
> mne.gui.kit2fiff
> >> in the mne-python tools.
> >>
> >> Best,
> >> Steve
> >>
> >>
> >>
> >> Stephen Politzer-Ahles
> >> New York University, Abu Dhabi
> >> Neuroscience of Language Lab
> >> http://www.nyu.edu/projects/politzer-ahles/
> >>
> >>
> >> On Mon, Nov 25, 2013 at 12:32 PM, Alexandre Gramfort
> >> <alexandre.gramfort at telecom-paristech.fr> wrote:
> >>>
> >>> Steve,
> >>>
> >>> how did you write this file?
> >>>
> >>> standard .fif file should not exceed 2GB.
> >>>
> >>> Alex
> >>>
> >>>
> >>> On Mon, Nov 25, 2013 at 7:34 AM, Stephen Politzer-Ahles <
> spa268 at nyu.edu>
> >>> wrote:
> >>>> Hi Eric,
> >>>>
> >>>> Thanks, I will keep trying various things out in the meantime. I was
> >>>> also
> >>>> concerned about file size limits, but it seems weird that other
> >>>> functions
> >>>> like mne_process_raw still work on my big files.
> >>>>
> >>>> If I don't find a solution, is it possible instead to just run
> >>>> mne_mark_bad_channels on the averaged data (and make sure to use the
> >>>> averaged, rather than the raw, as the --meas file in later steps of
> the
> >>>> pipeline)?
> >>>>
> >>>> Best,
> >>>> Steve
> >>>>
> >>>>
> >>>>
> >>>> Stephen Politzer-Ahles
> >>>> New York University, Abu Dhabi
> >>>> Neuroscience of Language Lab
> >>>> http://www.nyu.edu/projects/politzer-ahles/
> >>>>
> >>>>
> >>>> On Mon, Nov 25, 2013 at 5:20 AM, Eric Larson <larson.eric.d at gmail.com
> >
> >>>> wrote:
> >>>>>
> >>>>> Hey Steve,
> >>>>>
> >>>>> It is possible that this has to do with file locations being stored
> >>>>> using
> >>>>> 32-bit integers. I thought FIFF had an effective file-size limit of
> 2GB
> >>>>> for
> >>>>> that reason (using 32-bit signed integers would make this the limit),
> >>>>> but
> >>>>> I'm not 100% sure. If nobody beats me to a more definitive answer and
> >>>>> you
> >>>>> can't find anything about this issue yourself by tomorrow or
> Tuesday, I
> >>>>> can
> >>>>> look into it more then.
> >>>>>
> >>>>> Cheers,
> >>>>> Eric
> >>>>>
> >>>>>
> >>>>> On Sun, Nov 24, 2013 at 12:33 PM, Stephen Politzer-Ahles
> >>>>> <spa268 at nyu.edu>
> >>>>> wrote:
> >>>>>>
> >>>>>> To whom it may concern:
> >>>>>>
> >>>>>> When I try to run mne_mark_bad_channels in tcsh on a raw .fif file
> >>>>>> (about
> >>>>>> 3.25 GB), I get an error like the following:
> >>>>>>
> >>>>>> "/Volumes/data/....../A0007.fif ... [failed] fseek : Value too large
> >>>>>> to
> >>>>>> be stored in data type"
> >>>>>>
> >>>>>> As far as I can tell this seems to be some kind of Unix error, not
> >>>>>> something specific to MNE...but this happens regardless of which
> drive
> >>>>>> I
> >>>>>> happen to be trying to modify data on, and also other functions that
> >>>>>> also
> >>>>>> need to open the data (such as mne_process_raw) work fine, so I
> don't
> >>>>>> think
> >>>>>> it's necessarily a permission or filesystem issue. Does anyone have
> >>>>>> any idea
> >>>>>> what might be causing this?
> >>>>>>
> >>>>>> Thanks for your input,
> >>>>>> Steve
> >>>>>>
> >>>>>>
> >>>>>> Stephen Politzer-Ahles
> >>>>>> New York University, Abu Dhabi
> >>>>>> Neuroscience of Language Lab
> >>>>>> http://www.nyu.edu/projects/politzer-ahles/
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Mne_analysis mailing list
> >>>>>> Mne_analysis at nmr.mgh.harvard.edu
> >>>>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> >>>>>>
> >>>>>>
> >>>>>> The information in this e-mail is intended only for the person to
> whom
> >>>>>> it
> >>>>>> is
> >>>>>> addressed. If you believe this e-mail was sent to you in error and
> the
> >>>>>> e-mail
> >>>>>> contains patient information, please contact the Partners Compliance
> >>>>>> HelpLine at
> >>>>>> http://www.partners.org/complianceline . If the e-mail was sent to
> you
> >>>>>> in
> >>>>>> error
> >>>>>> but does not contain patient information, please contact the sender
> >>>>>> and
> >>>>>> properly
> >>>>>> dispose of the e-mail.
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Mne_analysis mailing list
> >>>> Mne_analysis at nmr.mgh.harvard.edu
> >>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> >>>>
> >>>>
> >>>> The information in this e-mail is intended only for the person to whom
> >>>> it is
> >>>> addressed. If you believe this e-mail was sent to you in error and the
> >>>> e-mail
> >>>> contains patient information, please contact the Partners Compliance
> >>>> HelpLine at
> >>>> http://www.partners.org/complianceline . If the e-mail was sent to
> you
> >>>> in
> >>>> error
> >>>> but does not contain patient information, please contact the sender
> and
> >>>> properly
> >>>> dispose of the e-mail.
> >>>>
> >>
> >>
> > _______________________________________________
> > Mne_analysis mailing list
> > Mne_analysis at nmr.mgh.harvard.edu
> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20131125/40921d24/attachment.html