[BangPypers] Reorder Dictionary Size in python

[Noufal Ibrahim]

Noufal Ibrahim <noufal at nibrahim.net.in> writes:

> Rahul R <rahul8590 at gmail.com> writes:
>
>>  I dont intent to pre optimize things , I am rather driven by curiosity on
>> how one could do that.
>
> I'm with Anand on recommendations but like you, I'm curious as to
> whether this is possible. Do you have some estimates of size for your
> dictionary and the average size for the values? 
>
> I'm trying some experiments. 

My experiments were not particularly revealing. However, 
take a look at
http://hg.python.org/cpython/file/3e02d70cd07b/Objects/dictnotes.txt

There are *compile time* parameters that can be tweaked in the
implementation which will affect behaviour. These things all affect
performance in subtle ways and the values that they've chosen are
probably based on varied data which best represent real life
cases. Tweaking them will probably be a micro optimisation that you'll
need only for specific cases.

The ones I see are dictionary size, load, growth rate, sparseness and
shrinkage rate. The last two are things you can tweak to get the
behaviour you want.

However, as it notes, shrinkage checks are never done when an item is
removed. So, it looks as though you have to actually add an item after
deleting everything else for the dictionary itself to shrink which is an
interesting quirk. 

Another thing that occurs to me is that the "size" of the dictionary is
probably dominated by the size of the keys and, perhaps more so, the
values. Unless these are super small, I don't think the extra overhead
of the dictionary itself will become a burden. In other words, the empty
slots you're worried about shouldn't be a problem unless your actual
data is so small that the free slots are the dominant data. 


[...]



-- 
Cordially,
Noufal
http://nibrahim.net.in