[Mne_analysis] "Value too large to be stored in data type" when running mne_mark_bad_channels in tcsh

[Alexandre Gramfort]

hi Steve,

looks like you've hit a problem in our python fiff writing code that does not
splits .fif files to avoid large files.

We'll look into it asap

Thanks for the report.

Best,
Alex


On Mon, Nov 25, 2013 at 9:34 AM, Stephen Politzer-Ahles <spa268 at nyu.edu> wrote:
> Hi Alexandre,
>
> It started as Yokogawa .con data, I wrote it to fif using mne.gui.kit2fiff
> in the mne-python tools.
>
> Best,
> Steve
>
>
>
> Stephen Politzer-Ahles
> New York University, Abu Dhabi
> Neuroscience of Language Lab
> http://www.nyu.edu/projects/politzer-ahles/
>
>
> On Mon, Nov 25, 2013 at 12:32 PM, Alexandre Gramfort
> <alexandre.gramfort at telecom-paristech.fr> wrote:
>>
>> Steve,
>>
>> how did you write this file?
>>
>> standard .fif file should not exceed 2GB.
>>
>> Alex
>>
>>
>> On Mon, Nov 25, 2013 at 7:34 AM, Stephen Politzer-Ahles <spa268 at nyu.edu>
>> wrote:
>> > Hi Eric,
>> >
>> > Thanks, I will keep trying various things out in the meantime. I was
>> > also
>> > concerned about file size limits, but it seems weird that other
>> > functions
>> > like mne_process_raw still work on my big files.
>> >
>> > If I don't find a solution, is it possible instead to just run
>> > mne_mark_bad_channels on the averaged data (and make sure to use the
>> > averaged, rather than the raw, as the --meas file in later steps of the
>> > pipeline)?
>> >
>> > Best,
>> > Steve
>> >
>> >
>> >
>> > Stephen Politzer-Ahles
>> > New York University, Abu Dhabi
>> > Neuroscience of Language Lab
>> > http://www.nyu.edu/projects/politzer-ahles/
>> >
>> >
>> > On Mon, Nov 25, 2013 at 5:20 AM, Eric Larson <larson.eric.d at gmail.com>
>> > wrote:
>> >>
>> >> Hey Steve,
>> >>
>> >> It is possible that this has to do with file locations being stored
>> >> using
>> >> 32-bit integers. I thought FIFF had an effective file-size limit of 2GB
>> >> for
>> >> that reason (using 32-bit signed integers would make this the limit),
>> >> but
>> >> I'm not 100% sure. If nobody beats me to a more definitive answer and
>> >> you
>> >> can't find anything about this issue yourself by tomorrow or Tuesday, I
>> >> can
>> >> look into it more then.
>> >>
>> >> Cheers,
>> >> Eric
>> >>
>> >>
>> >> On Sun, Nov 24, 2013 at 12:33 PM, Stephen Politzer-Ahles
>> >> <spa268 at nyu.edu>
>> >> wrote:
>> >>>
>> >>> To whom it may concern:
>> >>>
>> >>> When I try to run mne_mark_bad_channels in tcsh on a raw .fif file
>> >>> (about
>> >>> 3.25 GB), I get an error like the following:
>> >>>
>> >>> "/Volumes/data/....../A0007.fif ... [failed] fseek : Value too large
>> >>> to
>> >>> be stored in data type"
>> >>>
>> >>> As far as I can tell this seems to be some kind of Unix error, not
>> >>> something specific to MNE...but this happens regardless of which drive
>> >>> I
>> >>> happen to be trying to modify data on, and also other functions that
>> >>> also
>> >>> need to open the data (such as mne_process_raw) work fine, so I don't
>> >>> think
>> >>> it's necessarily a permission or filesystem issue. Does anyone have
>> >>> any idea
>> >>> what might be causing this?
>> >>>
>> >>> Thanks for your input,
>> >>> Steve
>> >>>
>> >>>
>> >>> Stephen Politzer-Ahles
>> >>> New York University, Abu Dhabi
>> >>> Neuroscience of Language Lab
>> >>> http://www.nyu.edu/projects/politzer-ahles/
>> >>>
>> >>> _______________________________________________
>> >>> Mne_analysis mailing list
>> >>> Mne_analysis at nmr.mgh.harvard.edu
>> >>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>> >>>
>> >>>
>> >>> The information in this e-mail is intended only for the person to whom
>> >>> it
>> >>> is
>> >>> addressed. If you believe this e-mail was sent to you in error and the
>> >>> e-mail
>> >>> contains patient information, please contact the Partners Compliance
>> >>> HelpLine at
>> >>> http://www.partners.org/complianceline . If the e-mail was sent to you
>> >>> in
>> >>> error
>> >>> but does not contain patient information, please contact the sender
>> >>> and
>> >>> properly
>> >>> dispose of the e-mail.
>> >>>
>> >>
>> >
>> >
>> > _______________________________________________
>> > Mne_analysis mailing list
>> > Mne_analysis at nmr.mgh.harvard.edu
>> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>> >
>> >
>> > The information in this e-mail is intended only for the person to whom
>> > it is
>> > addressed. If you believe this e-mail was sent to you in error and the
>> > e-mail
>> > contains patient information, please contact the Partners Compliance
>> > HelpLine at
>> > http://www.partners.org/complianceline . If the e-mail was sent to you
>> > in
>> > error
>> > but does not contain patient information, please contact the sender and
>> > properly
>> > dispose of the e-mail.
>> >
>
>