simultaneous multiple requests to very simple database

[John Lenton]

On Tue, Jan 18, 2005 at 11:26:46AM -0500, Eric S. Johansson wrote:
> I have an application where I need a very simple database, effectively a 
> very large dictionary.  The very large dictionary must be accessed from 
> multiple processes simultaneously.  I need to be able to lock records 
> within the very large dictionary when records are written to.  Estimated 
> number of records will be in the ballpark of 50,000 to 100,000 in his 
> early phase and 10 times that in the future.  Each record will run about 
> 100 to 150 bytes.
> 
> speed is not a huge concern although I must complete processing in less 
> than 90 seconds.  The longer the delay however the greater number of 
> processes must be running parallel in order to keep the throughput up. 
> It's the usual trade-off we have all come to know and love.
> 
> it is not necessary for the dictionary to persist beyond the life of the 
> parent process although I have another project coming up in which this 
> would be a good idea.
> 
> at this point, I know they will be some kind souls suggesting various 
> SQL solutions.  While I appreciate the idea, unfortunately I do not have 
> time to puzzle out yet another component.  Someday I will figure it out 
> because I really liked what I see with SQL lite but unfortunately, today 
> is not that day (unless they will give me their work, home and cell 
> phone numbers so I can call when I am stuck. ;-)

I'm sure we could agree on a fee for me to do so :)

> So the solutions that come to mind are some form of dictionary in shared 
> memory with locking semaphore scoreboard or a multithreaded process 
> containing a single database (Python native dictionary, metakit, gdbm??) 
> and have all of my processes speak to it using xmlrpc which leaves me 
> with the question of how to make a multithreaded server using stock xmlrpc.

berkley db (at least version 3, http://pybsddb.sourceforge.net/)
supports multiple readers and writers, with fine-grained locking, it
looks like a dictionary, and it isn't sql. The use you have in mind is a
bit more complicated than the simple create-me-a-dictionary-in-a-file,
but is pretty straightforward. The documentation mostly refers you to
the C API, but fortunately it (the C API) is clear and well written.

HTH

-- 
John Lenton (john at grulic.org.ar) -- Random fortune:
Today is National Existential Ennui Awareness Day.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20050118/ed3c3966/attachment.sig>