dynamic allocation file buffer

[Paul Boddie]

On 11 Sep, 19:31, "Aaron \"Castironpi\" Brady" <castiro... at gmail.com>
wrote:
>
> An acquaintance suggests that defragmentation would be a useful
> service to provide along with memory management too, which also
> requires an index.

I presume that you mean efficient access to large amounts of data in
the sense that if all the data you want happens to be in the same page
or segment, then retrieving it is much more efficient than having to
seek around for all the different pieces. So the defragmentation would
be what they call clustering in a relational database context:

http://www.postgresql.org/docs/8.3/static/sql-cluster.html

I've seen similar phenomena outside the relational database world,
notably with big Lucene indexes which wouldn't fit in memory in their
entirety.

> I encourage overlap between a bare-bones alloc/free module and
> established database territory and I'm very aware of it.
>
> Databases already support both concurrency and persistence, but don't
> tell me you'd use a database for IPC.

Of course, databases are widely used in scalable systems to hold
central state, which is why there's a lot of effort put into to not
only scaling up database installations, but also into things like
caching which are supposed to save the database systems behind popular
Web applications from excessive load.

>                                        And don't tell me you've never
> wished you had a reference to a record in a table so that you could
> make an update just by changing one word of memory at the right
> place.  Sometimes databases are overkill where all you want is dynamic
> allocation.

I think that the challenge is to reduce an abstract operation (for
example, wanting to update a particular column in a particular record)
to its measurable effects (this word of memory/disk will change as a
consequence). It's easy for a human with a reasonable knowledge of,
say, a relational database system to anticipate such things, but to
actually collapse a number of layers through some kind of generic
optimisation process is a lot more difficult.

Paul