Most efficient way to build very large dictionaries

[python at bdurham.com]

Hi Roger,

> you may want to consider using SQLite

Thank you for your suggestion about looking at SQLite. I haven't
compared the performance of SQLite to Python dictionaries, but I'm
skeptical that SQLite would be faster than in-memory Python dictionaries
for the type of analysis I'm doing. Prior to my use of Python, my
customer used a very expensive Oracle system to analyze their log files.
My simple Python scripts are 4-20x faster than the Oracle PL/SQL they
are replacing - and run on much cheaper hardware.

Note: Memory is currently not a concern for me so I don't need SQLite's
ability to work with data sets larger than my physical memory.

Regards,
Malcolm


----- Original message -----
From: "Roger Binns" <rogerb at rogerbinns.com>
To: python-list at python.org
Date: Wed, 24 Dec 2008 00:19:56 -0800
Subject: Re: Most efficient way to build very large dictionaries

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

python at bdurham.com wrote:
> I would appreciate your thoughts on whether there are advantages to
> working with a pre-built dictionary and if so, what are the best ways to
> create a pre-loaded dictionary.

Based on this and your other thread, you may want to consider using
SQLite (standard Python module is available for it).  SQL queries are
very similar to set operations and indices take care of performance (by
using more storage).  You also get transactions for free.  If you look
into SQLite pragmas you can get it to use more RAM to improve
performance.

And by using a SQL layer you can later switch to another database should
you need really hard core storage.  It also makes the data available to
other programs in a uniform way.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAklR8KgACgkQmOOfHg372QSrHQCfVJzueXVKme8QZcxoLf70BL4K
RL8AoM9QOFykOLrr5QXtpmZ5f7CFHm6e
=zAPG
-----END PGP SIGNATURE-----

--
http://mail.python.org/mailman/listinfo/python-list