dynamic allocation file buffer

[Aaron "Castironpi" Brady]

On Sep 11, 2:40 am, Steven D'Aprano
<ste... at REMOVE.THIS.cybersource.com.au> wrote:
> On Wed, 10 Sep 2008 11:59:35 -0700, Aaron \"Castironpi\" Brady wrote:
> > On Sep 10, 5:24 am, Steven D'Aprano
> > <ste... at REMOVE.THIS.cybersource.com.au> wrote:
> >> On Wed, 10 Sep 2008 09:26:20 +0200, Fredrik Lundh wrote:
> >> > Steven D'Aprano wrote:
>
> >> >> You've created a solution to a problem which (probably) only affects
> >> >> a very small number of people, at least judging by your use-cases.
> >> >> Who has a 4GB XML file
>
> >> > Getting 4GB XML files from, say, logging processes or databases that
> >> > can render their output as XML is not that uncommon.  They're usually
> >> > record-oriented, and are intended to be processed as streams.  And
> >> > given the right tools, doing that is no harder than doing the same to
> >> > a 4GB text file.
>
> >> Fair enough, that's a good point.
>
> >> But would you expect random access to a 4GB XML file? If I've
> >> understood what Castironpi is trying for, his primary use case was for
> >> people wanting exactly that.
>
> >> --
> >> Steven
>
> > Steven,
>
> > Are you claiming that sequential storage is sufficient for small amounts
> > of data, and relational db.s are necessary for large amounts?
>
> I'm no longer *claiming* anything, I'm *asking* whether random access to
> a 4GB XML file is something that is credible or useful. It is my
> understanding that XML is particularly ill-suited to random access once
> the amount of data is too large to fit in RAM.
>
> I'm interested in what Fredrik has to say about this, as he's the author
> of ElementTree.
>
> --
> Steven

XML is the wrong word for the example I was thinking of (as was
already pointed out in another thread).  XML is by definition
sequential.  The use case pertained to a generic element hierarchy;
think of 4GB of hierarchical data.