writing large files quickly

[Scott David Daniels]

Jens Theisen wrote:
> Ivan wrote:
>> I read somewhere that it has a use in database software, but the only
>> thing I can imagine for this is when using heap queues
>> (http://python.active-venture.com/lib/node162.html).

I've used this feature eons ago where the file was essentially a single
large address space (memory mapped array) that was expected to never
fill all that full.  I was tracking data from a year of (thousands? of)
students seeing Drill-and-practice questions from a huge database of
questions.  The research criticism we got was that our analysis did not
rule out any kid seeing the same question more than once, and getting
"practice" that would improve performance w/o learning.  I built a
bit-filter and copied tapes dropping any repeats seen by students.
We then just ran the same analysis we had on the raw data, and found
no significant difference.

The nice thing is that file size grew over time, so (for a while) I
could run on the machine with other users.  By the last block of
tapes I was sitting alone in the machine room at 3:00 AM on Sat mornings
afraid to so much as fire up an editor.

-- 
-Scott David Daniels
scott.daniels at acm.org