Replacement for the shelve module?

[Steven D'Aprano]

Forafo San wrote:

> Folks,
> What might be a good replacement for the shelve module, but one that
> can handle a few gigs of data. I'm doing some calculations on daily
> stock prices and the result is a nested list like:
> 
> [[date_1, floating result 1],
>  [date_2, floating result 2],
> ...
>  [date_n, floating result n]]
> 
> However, there are about 5,000 lists like that, one for each stock
> symbol. 


You might save some memory by using tuples rather than lists:

>>> sys.getsizeof(["01/01/2000", 123.456])  # On a 32-bit system.
40
>>> sys.getsizeof(("01/01/2000", 123.456))
32


By the way, you know that you should never, ever use floats for currency,
right? 

http://vladzloteanu.wordpress.com/2010/01/11/why-you-shouldnt-use-float-for-currency-floating-point-issues-explained-for-ruby-and-ror/
http://stackoverflow.com/questions/3730019/why-not-use-double-or-float-to-represent-currency


> Using the shelve module I could easily save them to a file 
> ( myshelvefile['symbol_1') = symbol_1_list) and likewise retrieve the
> data. But shelve is deprecated 

It certainly is not.

http://docs.python.org/library/shelve.html
http://docs.python.org/py3k/library/shelve.html

Not a word about it being deprecated in either Python 2.x or 3.x.


> AND when a lot of data is written 
> shelve was acting weird (refusing to write, filesizes reported with an
> "ls" did not make sense, etc.).

I would like to see this replicated. If it is true, that's a bug in shelve,
but I expect you're probably doing something wrong.



-- 
Steven