[issue39714] ElementTree limitation

[Stefan Behnel]

Stefan Behnel <stefan_ml at behnel.de> added the comment:

I'd suggest feeding the data into the parser in chunks, or letting it read from a file-like object, or something like that.

Also, you probably want to do incremental processing on the data (see the XMLPullParser and iterparse), because reading 3.5GB of XML data into an in-memory tree can easily result in 10x the memory usage. You may have 40GB of RAM on your machine, but even then, I'd still recommend processing the data in incrementally.

----------
nosy: +scoder
versions: +Python 3.10, Python 3.9 -Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39714>
_______________________________________