[Mne_analysis] IOError: 2GB file size limit reached. Support for larger raw files will be added in the future.

Wed Mar 19 18:25:54 EDT 2014

Thanks,

Lauri --- that would mean, using such link tags, we could load the entire chain using a single filename? That sound like the best we can do given difficulties mentioned.

Sounds clean to me.
Denis

> On Mar 19, 2014, at 9:24 PM, Parkkonen Lauri <lauri.parkkonen at aalto.fi> wrote:
> 
> Hello Denis et al.,
> 
> I just confirm what Eric already said: it's not easy to get rid of the 2-GB file size limit as it is due to the fact the FIFF format internally uses 32-bit file pointers. Switching to 64-bit pointers would not be backwards compatible. The limit is an issue for Elekta, too, and they are looking for a good solution but I don't know about the time frame.
> 
> In the meantime we could perhaps make use of the "file chaining" FIFF tags that allow logical linking of consecutive files. If the MNE FIFF i/o functions could use these tags and split/assemble files accordingly, reading and writing data larger than 2 GB could be made transparent to the user. Comments?
> 
> Cheers,
> Lauri
> 
>> On 19.03.2014 21:42, Denis-Alexander Engemann wrote:
>> Hi folks,
>> 
>> it just occurs to me that we might want to think about adding a Raw.split method.
>> 
>> As a first quick and dirty solution, Lau, you might want think about a function that splits the raw object and saves each split to fif files.
>> 
>> Something like this:
>> 
>> def split_save_raw(raw, splits , fname_pattern):
>>     for i_split, (tmin, tmax) in enumerate(splits):
>>         raw.copy().crop(tmin, tmax).save(fname_pattern % i_split)
>> 
>> 
>> split_save_raw(raw, [(0, 1), (1, 3), (3, 4)], 'my-split-%i-raw.fif')
>> 
>> Re-construction should then be trivial.
>> 
>> Cheers,
>> Denis
>> 
>> 
>> 
>>> On Wed, Mar 19, 2014 at 7:56 PM, Lau Møller Andersen <lau.andersen at cnru.dk> wrote:
>>> Dear Eric and Alex
>>> 
>>> Thanks for your answers. I now understand that it is not a simple issue.
>>> The files themselves are not greater than 2.0 GB, but because I have multiple recordings per subject, I read them in as a list. The only reason that I ask is that I would like to save the filtered file (i.e. the one that is greater than 2.0 GB, because it is a list of raw files).
>>> 
>>> The way I work around it now, is simply to save the epoched files and working from there. The only annoying thing is, if/when I realize that I need to do something different for creating the epochs, then I need to do the time-consuming process of filtering again. But I can live with that.
>>> 
>>> Thanks again for a wonderful python toolbox.
>>> 
>>> Best
>>> 
>>> Lau
>>> 
>>> 
>>> Den 19/03/2014 kl. 16.33 skrev Eric Larson <larson.eric.d at gmail.com>:
>>> 
>>>> It is challenging because the FIFF file format is not designed to handle files larger than that. Adding support for it would require redesigning and rewriting the I/O code fundamental to data analysis, and would likely break backward compatibility with many tools. AFAIK nobody is actively working on the issue.
>>>> 
>>>> There are different ways of working around the issue. If you have a specific use case, you could open an issue on the                             `mne-python` Github site, and perhaps we can provide some ideas. Ideally we could document the workarounds somewhere in case people hit this issue in the future, too.
>>>> 
>>>> Eric
>>>> 
>>>> 
>>>> 
>>>> On Wed, Mar 19, 2014 at 1:52 AM, Lau Møller Andersen <lau.andersen at cnru.dk> wrote:
>>>>> Dear MNE developers
>>>>> 
>>>>> Just out of curiosity, is this going to happen in the near future, or is
>>>>> it a greater challenge than I imagine?
>>>>> 
>>>>> Best
>>>>> 
>>>>> Lau
>>>>> _______________________________________________
>>>>> Mne_analysis mailing list
>>>>> Mne_analysis at nmr.mgh.harvard.edu
>>>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>>> 
>>>>> 
>>>>> The information in this e-mail is intended only for the person to whom it is
>>>>> addressed. If you believe this e-mail was sent to you in error and the e-mail
>>>>> contains patient information, please contact the Partners Compliance HelpLine at
>>>>> http://www.partners.org/complianceline . If the e-mail was sent to you in error
>>>>> but does not contain patient information, please contact the sender and properly
>>>>> dispose of the e-mail.
>>>> 
>>>> _______________________________________________
>>>> Mne_analysis mailing list
>>>> Mne_analysis at nmr.mgh.harvard.edu
>>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>>> 
>>>> 
>>>> The information in this e-mail is intended only for the person to whom it is
>>>> addressed. If you believe this e-mail was sent to you in error and the e-mail
>>>> contains patient information, please contact the Partners Compliance HelpLine at
>>>> http://www.partners.org/complianceline . If the e-mail was sent to you in error
>>>> but does not contain patient information, please contact the sender and properly
>>>> dispose of the e-mail.
>>> 
>>> 
>>> _______________________________________________
>>> Mne_analysis mailing list
>>> Mne_analysis at nmr.mgh.harvard.edu
>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
>>> 
>>> 
>>> The information in this e-mail is intended only for the person to whom it is
>>> addressed. If you believe this e-mail was sent to you in error and the e-mail
>>> contains patient information, please contact the Partners Compliance HelpLine at
>>> http://www.partners.org/complianceline . If the e-mail was sent to you in error
>>> but does not contain patient information, please contact the sender and properly
>>> dispose of the e-mail.
> 
> 
> _______________________________________________
> Mne_analysis mailing list
> Mne_analysis at nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/mne_analysis
> 
> 
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the e-mail
> contains patient information, please contact the Partners Compliance HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in error
> but does not contain patient information, please contact the sender and properly
> dispose of the e-mail.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.nmr.mgh.harvard.edu/pipermail/mne_analysis/attachments/20140319/e1df46a2/attachment.html 