Data/File Structure and Algorithm for Retrieving Sorted Data Chunk Efficiently

[Alex Martelli]

Jane Austine <janeaustine50 at hotmail.com> wrote:
   ...
> Maybe I could keep the cursor position among sessions if I were using
> a standalone server. However, it is in CGI environment.

You can arrange things so that a daemon is always running in the
background, keeping whatever state you wish, and you CGI script
basically just send over the form data to the daemon and get the
response back from it.  Webware comes with this kind of arrangement as
one of the ways it can work, for example.


> What can I do instead? What data structure allows you to access
> nth..n+10th data from the sorted list in a constant time?

Assuming you can find a way to amortize DB connection time (by a
daemon-based arrangement), I suggest a MySQL database and the use of a
SELECT query with ORDER BY and LIMIT clauses.  I don't normally suggest
MySQL over other DB engines, but for your specific problem it appears to
me that it may be most appropriate.  PostgreSQL also has LIMIT and
OFFSET clauses that work similarly to MySQL's LIMIT clause (iff you also
have an ORDER BY), but standard SQL doesn't support such a "sliding
window on the results" concept, unfortunately.

This suggestion is partly based on your followup post where you mention
your dataset size as being typically of several 10s of 1000s of records.


Alex