a huge shared read-only data in parallel accesses -- How? multithreading? multiprocessing?

Aaron Watters aaron.watters at gmail.com
Wed Dec 9 16:06:13 EST 2009


On Dec 9, 9:58 am, Valery <khame... at gmail.com> wrote:
> Hi all,
>
> Q: how to organize parallel accesses to a huge common read-only Python
> data structure?

Use a BTree on disk in a file.  A good file system will keep most of
the
pages you need in RAM whenever the data is "warm".  This works
for Python or any other programming language.  Generally you can
always get to any piece of data in about 4 seeks at most anyway,
so if your disk is fast your app will be fast too.  The file can
be accessed concurrently without problems by any number of processes
or threads.

   -- Aaron Watters
      http://listtree.appspot.com
      http://whiffdoc.appspot.com
===
less is more



More information about the Python-list mailing list