Python dictionary size/entry limit?

[intelliminer at gmail.com]

On Feb 21, 6:25 pm, Tino Wildenhain <t... at wildenhain.de> wrote:
> intellimi... at gmail.com wrote:
> > I wrote a script to process textual data and extract phrases from
> > them, storing these phrases in a dictionary. It encounters a
> > MemoryError when there are about 11.18M keys in the dictionary, and
> > the size is about 1.5GB. I tried multiple times, and the error occurs
> > everytime at exactly the same place (with the same number of keys in
> > the dict). I then split the dictionary into two using a simple
> > algorithm:
>
> > if str[0]<='m':
> >     dict=dict1
> > else:
> >     dict=dict2
>
> > #use dict...
>
> > And it worked fine. The total size of the two dictionaries well
> > exceeded 2GB yet no MemoryError occured.
>
> > I have 1GB of pysical memory and 3GB in pagefile. Is there a limit to
> > the size or number of entries that a single dictionary can possess? By
> > searching on the web I can't find a clue why this problem occurs.
>
>  From what can be deducted from the headers of your message:
> X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1;...
> you are using windows?
> It seems either python or windows memory management somehow prevent
> the use of continuous memory areas that large.
> We've got such an example somewhere down the list which was similar
> (iirc it was a large string in memory) which runned perfectly
> with linux. You can try yourself maybe by installing ubuntu
> on the same host. (If you feel fit you can even skip the install
> and run it off life CD but then you need to fiddle a little to
> get swap space on disk)
>
> Regards
> Tino
>
> > --
> >http://mail.python.org/mailman/listinfo/python-list
>
>
>
>  smime.p7s
> 4KViewDownload

Yes, it's winxp, I forgot to mention it.
Thanks for the reply.