[Mne_analysis] IOError: 2GB file size limit reached. Support for larger raw files will be added in the future.
Parkkonen Lauri
lauri.parkkonen at aalto.fi
Wed Mar 19 16:24:22 EDT 2014
Hello Denis et al.,
I just confirm what Eric already said: it's not easy to get rid of the
2-GB file size limit as it is due to the fact the FIFF format internally
uses 32-bit file pointers. Switching to 64-bit pointers would not be
backwards compatible. The limit is an issue for Elekta, too, and they
are looking for a good solution but I don't know about the time frame.
In the meantime we could perhaps make use of the "file chaining" FIFF
tags that allow logical linking of consecutive files. If the MNE FIFF
i/o functions could use these tags and split/assemble files accordingly,
reading and writing data larger than 2 GB could be made transparent to
the user. Comments?
Cheers,
Lauri
On 19.03.2014 21:42, Denis-Alexander Engemann wrote:
> Hi folks,
>
> it just occurs to me that we might want to think about adding a
> Raw.split method.
>
> As a first quick and dirty solution, Lau, you might want think about a
> function that splits the raw object and saves each split to fif files.
>
> Something like this:
>
> def split_save_raw(raw, splits , fname_pattern):
> for i_split, (tmin, tmax) in enumerate(splits):
> raw.copy().crop(tmin, tmax).save(fname_pattern % i_split)
>
>
> split_save_raw(raw, [(0, 1), (1, 3), (3, 4)], 'my-split-%i-raw.fif')
>
> Re-construction should then be trivial.
>
> Cheers,
> Denis
>
>
>
> On Wed, Mar 19, 2014 at 7:56 PM, Lau Møller Andersen
> <lau.andersen at cnru.dk <mailto:lau.andersen at cnru.dk>> wrote:
>
> Dear Eric and Alex
>
> Thanks for your answers. I now understand that it is not a simple
> issue.
> The files themselves are not greater than 2.0 GB, but because I
> have multiple recordings per subject, I read them in as a list.
> The only reason that I ask is that I would like to save the
> filtered file (i.e. the one that is greater than 2.0 GB, because
> it is a list of raw files).
>
> The way I work around it now, is simply to save the epoched files
> and working from there. The only annoying thing is, if/when I
> realize that I need to do something different for creating the
> epochs, then I need to do the time-consuming process of filtering
> again. But I can live with that.
>
> Thanks again for a wonderful python toolbox.
>
> Best
>
> Lau
>
>
> Den 19/03/2014 kl. 16.33 skrev Eric Larson
> <larson.eric.d at gmail.com <mailto:larson.eric.d at gmail.com>>:
>
>> It is challenging because the FIFF file format is not designed to
>> handle files larger than that. Adding support for it would
>> require redesigning and rewriting the I/O code fundamental to
>> data analysis, and would likely break backward compatibility with
>> many tools. AFAIK nobody is actively working on the issue.
>>
>> There are different ways of working around the issue. If you have
>> a specific use case, you could open an issue on the `mne-python`
>> Github site, and perhaps we can provide some ideas. Ideally we
>> could document the workarounds somewhere in case people hit this
>> issue in the future, too.
>>
>> Eric
>>
>>
>>
>> On Wed, Mar 19, 2014 at 1:52 AM, Lau Møller Andersen
>> <lau.andersen at cnru.dk <mailto:lau.andersen at cnru.dk>> wrote:
>>
>> Dear MNE developers
>>
>> Just out of curiosity, is this going to happen in the near
>> future, or is
>> it a greater challenge than I imagine?
>>
>> Best
>>
>> Lau
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>>
>> The information in this e-mail is intended only for the
>> person to whom it is
>> addressed. If you believe this e-mail was sent to you in
>> error and the e-mail
>> contains patient information, please contact the Partners
>> Compliance HelpLine at
>> http://www.partners.org/complianceline . If the e-mail was
>> sent to you in error
>> but does not contain patient information, please contact the
>> sender and properly
>> dispose of the e-mail.
>>
>>
>> _______________________________________________
>> Mne_analysis mailing list
>> Mne_analysis at nmr.mgh.harvard.edu
>> <mailto:Mne_analysis at nmr.mgh.harvard.edu>
>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>
>>
>> The information in this e-mail is intended only for the person to
>> whom it is
>> addressed. If you believe this e-mail was sent to you in error
>> and the e-mail
>> contains patient information, please contact the Partners
>> Compliance HelpLine at
>> http://www.partners.org/complianceline . If the e-mail was sent
>> to you in error
>> but does not contain patient information, please contact the
>> sender and properly
>> dispose of the e-mail.
>
>
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> <mailto:Mne_analysis at nmr.mgh.harvard.edu>
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>
>
> The information in this e-mail is intended only for the person to
> whom it is
> addressed. If you believe this e-mail was sent to you in error and
> the e-mail
> contains patient information, please contact the Partners
> Compliance HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to
> you in error
> but does not contain patient information, please contact the
> sender and properly
> dispose of the e-mail.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20140319/552d539b/attachment.html
More information about the Mne_analysis
mailing list