parsing multiple root element XML into text

[Stefan Behnel]

Marko Rauhamaa, 09.05.2014 14:38:
> Marko Rauhamaa:
>> Alain Ketterlin:
>>> Marko Rauhamaa writes:
>>>> Sometimes the XML elements come through a pipe as an endless
>>>> sequence. You can still use the wrapping technique and a SAX parser.
>>>> However, the other option is to write a tiny XML scanner that
>>>> identifies the end of each element. Then, you can cut out the
>>>> complete XML element and hand it over to a DOM parser.
>>>
>>> Well maybe, even though I see no point in doing so. If the whole
>>> transaction is a single document and you need to get sub-elements on
>>> the fly, just use the SAX parser: there is no need to use a "tiny XML
>>> scanner" (whatever that is), and building a DOM for a part of the
>>> document in your SAX handler is easy if needed (for the OP's case a
>>> simple state machine would be enough, probably).
>>
>> An example is <URL:
>> http://en.wikipedia.org/wiki/XMPP#XMPP_via_HTTP_and_WebSocket_transports>.
>>
>> The "document" is potentially infinitely long. The elements are
>> messages.
>>
>> The programmer would rather process the elements as DOM trees than
>> follow the meandering SAX parser.
> 
> In fact, the best thing would be if the DOM parser supported the use
> case out of the box: give the partial, whole or oversize document to the
> parser. If the document isn't complete, the parser should indicate the
> need for more input. If there are bytes after the document is
> successfully finished, the parser should leave the excess bytes in the
> pipeline.
> 
> IOW, if the DOM parser knows full well where the document ends, why
> must the application tell it to it?

In fact, XMPP traffic has a root element. And I agree that a tree is much
easier to handle than SAX events. ElementTree has gained a nice API in
Py3.4 that supports this in a much saner way than SAX, using iterators.
Basically, you just dump in some data that you received and get back an
iterator over the elements (and their subtrees) that it generated from it.
Intercept on the right top elements and you get your next subtree as soon
as it's ready.

https://docs.python.org/3.4/library/xml.etree.elementtree.html#pull-api-for-non-blocking-parsing

It's also supported by recent versions of lxml, which additionally has easy
to use support for the sending side with its xmlfile() tool.

http://lxml.de/parsing.html#incremental-event-parsing

http://lxml.de/api.html#incremental-xml-generation

Stefan