[Mne_analysis] IOError: 2GB file size limit reached. Support for larger raw files will be added in the future.

Parkkonen Lauri lauri.parkkonen at aalto.fi
Wed Mar 19 16:24:22 EDT 2014
Search archives:

Hello Denis et al.,

I just confirm what Eric already said: it's not easy to get rid of the 
2-GB file size limit as it is due to the fact the FIFF format internally 
uses 32-bit file pointers. Switching to 64-bit pointers would not be 
backwards compatible. The limit is an issue for Elekta, too, and they 
are looking for a good solution but I don't know about the time frame.

In the meantime we could perhaps make use of the "file chaining" FIFF 
tags that allow logical linking of consecutive files. If the MNE FIFF 
i/o functions could use these tags and split/assemble files accordingly, 
reading and writing data larger than 2 GB could be made transparent to 
the user. Comments?

Cheers,
Lauri

On 19.03.2014 21:42, Denis-Alexander Engemann wrote:
> Hi folks,
>
> it just occurs to me that we might want to think about adding a 
> Raw.split method.
>
> As a first quick and dirty solution, Lau, you might want think about a 
> function that splits the raw object and saves each split to fif files.
>
> Something like this:
>
> def split_save_raw(raw, splits , fname_pattern):
>     for i_split, (tmin, tmax) in enumerate(splits):
>         raw.copy().crop(tmin, tmax).save(fname_pattern % i_split)
>
>
> split_save_raw(raw, [(0, 1), (1, 3), (3, 4)], 'my-split-%i-raw.fif')
>
> Re-construction should then be trivial.
>
> Cheers,
> Denis
>
>
>
> On Wed, Mar 19, 2014 at 7:56 PM, Lau Møller Andersen 
> <lau.andersen at cnru.dk <mailto:lau.andersen at cnru.dk>> wrote:
>
>     Dear Eric and Alex
>
>     Thanks for your answers. I now understand that it is not a simple
>     issue.
>     The files themselves are not greater than 2.0 GB, but because I
>     have multiple recordings per subject, I read them in as a list.
>     The only reason that I ask is that I would like to save the
>     filtered file (i.e. the one that is greater than 2.0 GB, because
>     it is a list of raw files).
>
>     The way I work around it now, is simply to save the epoched files
>     and working from there. The only annoying thing is, if/when I
>     realize that I need to do something different for creating the
>     epochs, then I need to do the time-consuming process of filtering
>     again. But I can live with that.
>
>     Thanks again for a wonderful python toolbox.
>
>     Best
>
>     Lau
>
>
>     Den 19/03/2014 kl. 16.33 skrev Eric Larson
>     <larson.eric.d at gmail.com <mailto:larson.eric.d at gmail.com>>:
>
>>     It is challenging because the FIFF file format is not designed to
>>     handle files larger than that. Adding support for it would
>>     require redesigning and rewriting the I/O code fundamental to
>>     data analysis, and would likely break backward compatibility with
>>     many tools. AFAIK nobody is actively working on the issue.
>>
>>     There are different ways of working around the issue. If you have
>>     a specific use case, you could open an issue on the `mne-python`
>>     Github site, and perhaps we can provide some ideas. Ideally we
>>     could document the workarounds somewhere in case people hit this
>>     issue in the future, too.
>>
>>     Eric
>>
>>
>>
>>     On Wed, Mar 19, 2014 at 1:52 AM, Lau Møller Andersen
>>     <lau.andersen at cnru.dk <mailto:lau.andersen at cnru.dk>> wrote:
>>
>>         Dear MNE developers
>>
>>         Just out of curiosity, is this going to happen in the near
>>         future, or is
>>         it a greater challenge than I imagine?
>>
>>         Best
>>
>>         Lau
>>         _______________________________________________
>>         Mne_analysis mailing list
>>         Mne_analysis at nmr.mgh.harvard.edu
>>         <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>>         https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>>
>>         The information in this e-mail is intended only for the
>>         person to whom it is
>>         addressed. If you believe this e-mail was sent to you in
>>         error and the e-mail
>>         contains patient information, please contact the Partners
>>         Compliance HelpLine at
>>         http://www.partners.org/complianceline . If the e-mail was
>>         sent to you in error
>>         but does not contain patient information, please contact the
>>         sender and properly
>>         dispose of the e-mail.
>>
>>
>>     _______________________________________________
>>     Mne_analysis mailing list
>>     Mne_analysis at nmr.mgh.harvard.edu
>>     <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>>     https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>>
>>     The information in this e-mail is intended only for the person to
>>     whom it is
>>     addressed. If you believe this e-mail was sent to you in error
>>     and the e-mail
>>     contains patient information, please contact the Partners
>>     Compliance HelpLine at
>>     http://www.partners.org/complianceline . If the e-mail was sent
>>     to you in error
>>     but does not contain patient information, please contact the
>>     sender and properly
>>     dispose of the e-mail.
>
>
>     _______________________________________________
>     Mne_analysis mailing list
>     Mne_analysis at nmr.mgh.harvard.edu
>     <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>     https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
>     The information in this e-mail is intended only for the person to
>     whom it is
>     addressed. If you believe this e-mail was sent to you in error and
>     the e-mail
>     contains patient information, please contact the Partners
>     Compliance HelpLine at
>     http://www.partners.org/complianceline . If the e-mail was sent to
>     you in error
>     but does not contain patient information, please contact the
>     sender and properly
>     dispose of the e-mail.
>
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20140319/552d539b/attachment.html 


More information about the Mne_analysis mailing list