MemoryError in data conversion

[Peter Otten]

Gregory Ewing wrote:

> Mok-Kong Shen wrote:
>> I have yet a question out of curiosity: Why is my 2nd list structure,
>> that apparently is too complex for handling by eval and json, seemingly
>> not a problem for pickle?
> 
> Pickle is intended for arbitrary data structures, so it
> is designed to be able to handle deeply-nested and/or
> recursive data. Eval only has to handle nesting to depths
> likely to be encountered in source code. Apparently the
> json parser also assumes you're not going to be using
> very deep nesting.

But pickle does have the same limitation:

>>> def check(load, dump):
...     items = []
...     try:
...             for i in range(10**6):
...                     assert load(dump(items)) == items
...                     items = [items]
...     except RuntimeError:
...             return i
... 
>>> check(json.loads, json.dumps)
994
>>> check(pickle.loads, pickle.dumps)
499

Mok-Kong Shen, for pickle and json you can increase the limit a bit: 

>>> import sys
>>> sys.setrecursionlimit(2000)
>>> check(json.loads, json.dumps)
1994
>>> check(pickle.loads, pickle.dumps)
999

But be careful, if you choose the limit too high you'll see Python react 
like any other C program:

>>> sys.setrecursionlimit(100000)
>>> items = []
>>> for i in range(100000):
...     items = [items]
... 
>>> s = pickle.dumps(items)
Segmentation fault

For literal_eval() the limit is unfortunately hard-coded in the C source.

Mok-Kong Shen wrote:

> What I need is to have my (complicated) list to be put into
> a bytearray, do some proceesing, transfer it to the recipient.
> The recipient reverses the processing, obtains a bytearray
> that is the same as my original one and gets from it the same
> list as mine. Using pickle I can manage to do it (in fact I
> did try and succeed with my 2nd list), but that's IMHO a
> rather unnatural/awkward work-around. (It means that I have
> to pickle out the list to a file and read in the content of
> the file in order to have it as a bytearray etc. etc.)
 
The limit has been increased before, see

http://bugs.python.org/issue1881

and maybe you can get the core developers to increase it again, but 
generally speaking the existence of a recursion limit is the price you pay 
for easy interfacing with C. 

literal_eval() could allocate its stack dynamically, but that would probably 
complicate the code so that return on investment is questionable.