[PyPy-issue] [issue689] RPython exception unpickling a coroutine whose creator's stack has shrunk since it was switch()'d

Thu Apr 21 17:51:35 CEST 2011

New submission from Chris Smowton <cs448 at cam.ac.uk>:

Here's an interesting bug that occurs when unpickling coroutines whose creator's
stack has grown shorter since that creator was switched-in.

That's a fairly tangled sentence -- here's a situation summary:

Coro MAIN creates coro A, then switches to A.
A enters f, then g (A's stack is [f,g])
A switches to MAIN
MAIN switches back to A
A returns from g (his stack is now [f])
A creates a coro B, bound to anything, and PICKLES B
We try to UNPICKLE B:
Result: RPython exception in reconstruct_frame_chain.

I don't have a patch to fix this, but I have a solid idea what's wrong from
instrumenting lib-python/modified-2.5.2/pickle.py. I stuck some prints into
__reduce_ex__  and __setstate__ calls, and it looks like the frame for g is
being pickled when we create B, even though g is *not* on the stack at the time
B is created or pickled. Looking at the series of __reduce_ex__ calls, it looks
like we're pickling A's coroutine *as it was when A was switched in* rather than
as it was at the time we created B.

Here's an example run, in which I carry out pretty much what I just described,
with logging prints in pickle.py active. The crash looks exactly the same with
or without the prints.

Python 2.5.2 (, Mar 10 2011, 13:06:15)
[PyPy 1.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``2009 is the year of reflection on
the desktop''
>>>> import pickle
>>>> import stackless
>>>> h = None
>>>> main_coro = stackless.coroutine.getcurrent()
>>>> 
>>>> def f():
....    g()
....    global h
....    new_coro = stackless.coroutine()
....    new_coro.bind(lambda x: x, 5)
....    h = pickle.dumps(new_coro)
.... 
>>>> def g():
....    main_coro.switch()
.... 
>>>> user_coro = stackless.coroutine()
>>>> user_coro.bind(f)
>>>> 
>>>> user_coro.switch()
>>>> user_coro.switch()
!!!!! REDUCE_EX <_stackless.coroutine object at 0x00007f8c20ff8770> TO (<type
'_stackless.coroutine'>, (), (0, (), (<function <lambda> at 0x00007f8c20fdbe60>,
(5,), {}), <_stackless.coroutine object at 0x00007f8c20ff86e0>))
!!!!! REDUCE_EX <code object <lambda>, file '<console>', line 5> TO (<built-in
function code_new>, (1, 1, 1, 595, '|\x00\x00S', (None,), (), ('x',),
'<console>', '<lambda>', 5, '', (), (), 168686271))
!!!!! REDUCE_EX <module '__main__' from ?> TO (<built-in function __import__>,
('__main__', None, None, ('',)))
!!!!! REDUCE_EX <_stackless.coroutine object at 0x00007f8c20ff86e0> TO (<type
'_stackless.coroutine'>, (), (0, (<frame object at 0x00007f8c20fffe60>, <frame
object at 0x00007f8c20ffff10>), None, <_stackless.coroutine object at
0x0000000002319000>))
!!!!! REDUCE_EX <frame object at 0x00007f8c20fffe60> TO (<built-in function
frame_new>, (), (None, <module '__builtin__' (built-in)>, <code object f, file
'<console>', line 1>, ((1,), <function dumps at 0x00007f8c210ab990>, None,
<_stackless.coroutine object at 0x00007f8c20ff8770>), (), None, None,
{'main_coro': <_stackless.coroutine object at 0x0000000002319000>, 'stackless':
<module 'stackless' from
'/local/scratch/cs448/pypy-1.4.1-src/lib_pypy/stackless.pyc'>, 'g': <function g
at 0x00007f8c20fdbd00>, 'h': None, '__builtins__': <module '__builtin__'
(built-in)>, '__name__': '__main__', 'user_coro': <_stackless.coroutine object
at 0x00007f8c20ff86e0>, 'f': <function f at 0x00007f8c20fdb830>, 'pickle':
<module 'pickle' from
'/local/scratch/cs448/pypy-1.4.1-src/lib-python/modified-2.5.2/pickle.pyc'>,
'__doc__': None}, 47, False, 6, ((), <_stackless.coroutine object at
0x00007f8c20ff8770>), None, None, 0, -1, -1, None))
!!!!! REDUCE_EX <module '__builtin__' (built-in)> TO (<built-in function
__import__>, ('__builtin__', None, None, ('',)))
!!!!! REDUCE_EX <code object f, file '<console>', line 1> TO (<built-in function
code_new>, (0, 1, 4, 2627,
't\x00\x00\x83\x00\x00\x01t\x01\x00\x91\x02\x00\x92\x00\x00}\x00\x00|\x00\x00\x91\x03\x00d\x01\x00\x84\x00\x00d\x02\x00\x92\x02\x00\x01t\x04\x00\x91\x05\x00|\x00\x00\x92\x01\x00a\x06\x00d\x00\x00S',
(None, <code object <lambda>, file '<console>', line 5>, 5), ('g', 'stackless',
'coroutine', 'bind', 'pickle', 'dumps', 'h'), ('new_coro',), '<console>', 'f',
1, '\x00\x01\x07\x02\x0c\x01\x13\x01', (), (), 168686271))
!!!!! REDUCE_EX <frame object at 0x00007f8c20ffff10> TO (<built-in function
frame_new>, (), (<frame object at 0x00007f8c20fffe60>, <module '__builtin__'
(built-in)>, <code object g, file '<console>', line 1>, ((),), (), None, None,
{'main_coro': <_stackless.coroutine object at 0x0000000002319000>, 'stackless':
<module 'stackless' from
'/local/scratch/cs448/pypy-1.4.1-src/lib_pypy/stackless.pyc'>, 'g': <function g
at 0x00007f8c20fdbd00>, 'h': None, '__builtins__': <module '__builtin__'
(built-in)>, '__name__': '__main__', 'user_coro': <_stackless.coroutine object
at 0x00007f8c20ff86e0>, 'f': <function f at 0x00007f8c20fdb830>, 'pickle':
<module 'pickle' from
'/local/scratch/cs448/pypy-1.4.1-src/lib-python/modified-2.5.2/pickle.pyc'>,
'__doc__': None}, 13, True, 2, ((),), None, None, 0, -1, -1, None))
!!!!! REDUCE_EX <code object g, file '<console>', line 1> TO (<built-in function
code_new>, (0, 0, 2, 2627, 't\x00\x00\x91\x01\x00\x92\x00\x00\x01d\x00\x00S',
(None,), ('main_coro', 'switch'), (), '<console>', 'g', 1, '\x00\x01', (), (),
168686271))
!!!!! REDUCE_EX <_stackless.coroutine object at 0x0000000002319000> TO
(<built-in function _return_main>, ())
>>>> print h
c_stackless
coroutine
p0
(tRp1
(I0
(t(c_pickle_support
func_new
p2
(tRp3
(S'<lambda>'
p4
Nc_pickle_support
code_new
p5
(I1
I1
I1
I595
S'|\x00\x00S'
p6
(Ntp7
(t(S'x'
p8
tp9
S'<console>'
p10
S'<lambda>'
p11
I5
S''
p12
(t(tI168686271
tp13
Rp14
c__builtin__
getattr
p15
(c__builtin__
__import__
p16
(S'__main__'
p17
NN(S''
p18
tp19
tp20
Rp21
S'__dict__'
p22
tp23
RN(t(dp24
g17
tp25
b(I5
tp26
(dp27
tp28
g0
(tRp29
(I0
(c_pickle_support
frame_new
p30
(tRp31
(Ng16
(S'__builtin__'
p32
NN(g18
tp33
tp34
Rp35
g5
(I0
I1
I4
I2627
S't\x00\x00\x83\x00\x00\x01t\x01\x00\x91\x02\x00\x92\x00\x00}\x00\x00|\x00\x00\x91\x03\x00d\x01\x00\x84\x00\x00d\x02\x00\x92\x02\x00\x01t\x04\x00\x91\x05\x00|\x00\x00\x92\x01\x00a\x06\x00d\x00\x00S'
p36
(Ng14
I5
tp37
(S'g'
p38
S'stackless'
p39
S'coroutine'
p40
S'bind'
p41
S'pickle'
p42
S'dumps'
p43
S'h'
p44
tp45
(S'new_coro'
p46
tp47
S'<console>'
p48
S'f'
p49
I1
S'\x00\x01\x07\x02\x0c\x01\x13\x01'
p50
(t(tI168686271
tp51
Rp52
((I1
tp53
cpickle
dumps
p54
Ng1
tp55
(tNNg15
(g21
g22
tp56
RI47
I00
I6
((tg1
tp57
NNI0
I-1
I-1
Ntp58
bg30
(tRp59
(g31
g35
g5
(I0
I0
I2
I2627
S't\x00\x00\x91\x01\x00\x92\x00\x00\x01d\x00\x00S'
p60
(Ntp61
(S'main_coro'
p62
S'switch'
p63
tp64
(tS'<console>'
p65
S'g'
p66
I1
S'\x00\x01'
p67
(t(tI168686271
tp68
Rp69
((ttp70
(tNNg15
(g21
g22
tp71
RI13
I01
I2
((ttp72
NNI0
I-1
I-1
Ntp73
btp74
Nc_stackless
_return_main
p75
(tRp76
tp77
btp78
b.
>>>> pickle.loads(h)
@@@SETSTATE <type 'function'> ('<lambda>', None, <code object <lambda>, file
'<console>', line 5>, {'main_coro': <_stackless.coroutine object at
0x0000000002319000>, 'stackless': <module 'stackless' from
'/local/scratch/cs448/pypy-1.4.1-src/lib_pypy/stackless.pyc'>, 'g': <function g
at 0x00007f8c20fdbd00>, 'h':
"c_stackless\ncoroutine\np0\n(tRp1\n(I0\n(t(c_pickle_support\nfunc_new\np2\n(tRp3\n(S'<lambda>'\np4\nNc_pickle_support\ncode_new\np5\n(I1\nI1\nI1\nI595\nS'|\\x00\\x00S'\np6\n(Ntp7\n(t(S'x'\np8\ntp9\nS'<console>'\np10\nS'<lambda>'\np11\nI5\nS''\np12\n(t(tI168686271\ntp13\nRp14\nc__builtin__\ngetattr\np15\n(c__builtin__\n__import__\np16\n(S'__main__'\np17\nNN(S''\np18\ntp19\ntp20\nRp21\nS'__dict__'\np22\ntp23\nRN(t(dp24\ng17\ntp25\nb(I5\ntp26\n(dp27\ntp28\ng0\n(tRp29\n(I0\n(c_pickle_support\nframe_new\np30\n(tRp31\n(Ng16\n(S'__builtin__'\np32\nNN(g18\ntp33\ntp34\nRp35\ng5\n(I0\nI1\nI4\nI2627\nS't\\x00\\x00\\x83\\x00\\x00\\x01t\\x01\\x00\\x91\\x02\\x00\\x92\\x00\\x00}\\x00\\x00|\\x00\\x00\\x91\\x03\\x00d\\x01\\x00\\x84\\x00\\x00d\\x02\\x00\\x92\\x02\\x00\\x01t\\x04\\x00\\x91\\x05\\x00|\\x00\\x00\\x92\\x01\\x00a\\x06\\x00d\\x00\\x00S'\np36\n(Ng14\nI5\ntp37\n(S'g'\np38\nS'stackless'\np39\nS'coroutine'\np40\nS'bind'\np41\nS'pickle'\np42\nS'dumps'\np43\nS'h'\np44\ntp45\n(S'new_coro'\np46\ntp47\nS'<console>'\np48\nS'f'\np49\nI1\nS'\\x00\\x01\\x07\\x02\\x0c\\x01\\x13\\x01'\np50\n(t(tI168686271\ntp51\nRp52\n((I1\ntp53\ncpickle\ndumps\np54\nNg1\ntp55\n(tNNg15\n(g21\ng22\ntp56\nRI47\nI00\nI6\n((tg1\ntp57\nNNI0\nI-1\nI-1\nNtp58\nbg30\n(tRp59\n(g31\ng35\ng5\n(I0\nI0\nI2\nI2627\nS't\\x00\\x00\\x91\\x01\\x00\\x92\\x00\\x00\\x01d\\x00\\x00S'\np60\n(Ntp61\n(S'main_coro'\np62\nS'switch'\np63\ntp64\n(tS'<console>'\np65\nS'g'\np66\nI1\nS'\\x00\\x01'\np67\n(t(tI168686271\ntp68\nRp69\n((ttp70\n(tNNg15\n(g21\ng22\ntp71\nRI13\nI01\nI2\n((ttp72\nNNI0\nI-1\nI-1\nNtp73\nbtp74\nNc_stackless\n_return_main\np75\n(tRp76\ntp77\nbtp78\nb.",
'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__',
'user_coro': <_stackless.coroutine object at 0x00007f8c20ff86e0>, 'f': <function
f at 0x00007f8c20fdb830>, 'pickle': <module 'pickle' from
'/local/scratch/cs448/pypy-1.4.1-src/lib-python/modified-2.5.2/pickle.pyc'>,
'__doc__': None}, None, (), {}, '__main__')
@@@SETSTATE <type 'frame'> (None, <module '__builtin__' (built-in)>, <code
object f, file '<console>', line 1>, ((1,), <function dumps at
0x00007f8c210ab990>, None, <_stackless.coroutine object at 0x00007f8c20ff8800>),
(), None, None, {'main_coro': <_stackless.coroutine object at
0x0000000002319000>, 'stackless': <module 'stackless' from
'/local/scratch/cs448/pypy-1.4.1-src/lib_pypy/stackless.pyc'>, 'g': <function g
at 0x00007f8c20fdbd00>, 'h':
"c_stackless\ncoroutine\np0\n(tRp1\n(I0\n(t(c_pickle_support\nfunc_new\np2\n(tRp3\n(S'<lambda>'\np4\nNc_pickle_support\ncode_new\np5\n(I1\nI1\nI1\nI595\nS'|\\x00\\x00S'\np6\n(Ntp7\n(t(S'x'\np8\ntp9\nS'<console>'\np10\nS'<lambda>'\np11\nI5\nS''\np12\n(t(tI168686271\ntp13\nRp14\nc__builtin__\ngetattr\np15\n(c__builtin__\n__import__\np16\n(S'__main__'\np17\nNN(S''\np18\ntp19\ntp20\nRp21\nS'__dict__'\np22\ntp23\nRN(t(dp24\ng17\ntp25\nb(I5\ntp26\n(dp27\ntp28\ng0\n(tRp29\n(I0\n(c_pickle_support\nframe_new\np30\n(tRp31\n(Ng16\n(S'__builtin__'\np32\nNN(g18\ntp33\ntp34\nRp35\ng5\n(I0\nI1\nI4\nI2627\nS't\\x00\\x00\\x83\\x00\\x00\\x01t\\x01\\x00\\x91\\x02\\x00\\x92\\x00\\x00}\\x00\\x00|\\x00\\x00\\x91\\x03\\x00d\\x01\\x00\\x84\\x00\\x00d\\x02\\x00\\x92\\x02\\x00\\x01t\\x04\\x00\\x91\\x05\\x00|\\x00\\x00\\x92\\x01\\x00a\\x06\\x00d\\x00\\x00S'\np36\n(Ng14\nI5\ntp37\n(S'g'\np38\nS'stackless'\np39\nS'coroutine'\np40\nS'bind'\np41\nS'pickle'\np42\nS'dumps'\np43\nS'h'\np44\ntp45\n(S'new_coro'\np46\ntp47\nS'<console>'\np48\nS'f'\np49\nI1\nS'\\x00\\x01\\x07\\x02\\x0c\\x01\\x13\\x01'\np50\n(t(tI168686271\ntp51\nRp52\n((I1\ntp53\ncpickle\ndumps\np54\nNg1\ntp55\n(tNNg15\n(g21\ng22\ntp56\nRI47\nI00\nI6\n((tg1\ntp57\nNNI0\nI-1\nI-1\nNtp58\nbg30\n(tRp59\n(g31\ng35\ng5\n(I0\nI0\nI2\nI2627\nS't\\x00\\x00\\x91\\x01\\x00\\x92\\x00\\x00\\x01d\\x00\\x00S'\np60\n(Ntp61\n(S'main_coro'\np62\nS'switch'\np63\ntp64\n(tS'<console>'\np65\nS'g'\np66\nI1\nS'\\x00\\x01'\np67\n(t(tI168686271\ntp68\nRp69\n((ttp70\n(tNNg15\n(g21\ng22\ntp71\nRI13\nI01\nI2\n((ttp72\nNNI0\nI-1\nI-1\nNtp73\nbtp74\nNc_stackless\n_return_main\np75\n(tRp76\ntp77\nbtp78\nb.",
'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__',
'user_coro': <_stackless.coroutine object at 0x00007f8c20ff86e0>, 'f': <function
f at 0x00007f8c20fdb830>, 'pickle': <module 'pickle' from
'/local/scratch/cs448/pypy-1.4.1-src/lib-python/modified-2.5.2/pickle.pyc'>,
'__doc__': None}, 47, False, 6, ((), <_stackless.coroutine object at
0x00007f8c20ff8800>), None, None, 0, -1, -1, None)
@@@SETSTATE <type 'frame'> (<frame object at 0x00007f8c21018c80>, <module
'__builtin__' (built-in)>, <code object g, file '<console>', line 1>, ((),), (),
None, None, {'main_coro': <_stackless.coroutine object at 0x0000000002319000>,
'stackless': <module 'stackless' from
'/local/scratch/cs448/pypy-1.4.1-src/lib_pypy/stackless.pyc'>, 'g': <function g
at 0x00007f8c20fdbd00>, 'h':
"c_stackless\ncoroutine\np0\n(tRp1\n(I0\n(t(c_pickle_support\nfunc_new\np2\n(tRp3\n(S'<lambda>'\np4\nNc_pickle_support\ncode_new\np5\n(I1\nI1\nI1\nI595\nS'|\\x00\\x00S'\np6\n(Ntp7\n(t(S'x'\np8\ntp9\nS'<console>'\np10\nS'<lambda>'\np11\nI5\nS''\np12\n(t(tI168686271\ntp13\nRp14\nc__builtin__\ngetattr\np15\n(c__builtin__\n__import__\np16\n(S'__main__'\np17\nNN(S''\np18\ntp19\ntp20\nRp21\nS'__dict__'\np22\ntp23\nRN(t(dp24\ng17\ntp25\nb(I5\ntp26\n(dp27\ntp28\ng0\n(tRp29\n(I0\n(c_pickle_support\nframe_new\np30\n(tRp31\n(Ng16\n(S'__builtin__'\np32\nNN(g18\ntp33\ntp34\nRp35\ng5\n(I0\nI1\nI4\nI2627\nS't\\x00\\x00\\x83\\x00\\x00\\x01t\\x01\\x00\\x91\\x02\\x00\\x92\\x00\\x00}\\x00\\x00|\\x00\\x00\\x91\\x03\\x00d\\x01\\x00\\x84\\x00\\x00d\\x02\\x00\\x92\\x02\\x00\\x01t\\x04\\x00\\x91\\x05\\x00|\\x00\\x00\\x92\\x01\\x00a\\x06\\x00d\\x00\\x00S'\np36\n(Ng14\nI5\ntp37\n(S'g'\np38\nS'stackless'\np39\nS'coroutine'\np40\nS'bind'\np41\nS'pickle'\np42\nS'dumps'\np43\nS'h'\np44\ntp45\n(S'new_coro'\np46\ntp47\nS'<console>'\np48\nS'f'\np49\nI1\nS'\\x00\\x01\\x07\\x02\\x0c\\x01\\x13\\x01'\np50\n(t(tI168686271\ntp51\nRp52\n((I1\ntp53\ncpickle\ndumps\np54\nNg1\ntp55\n(tNNg15\n(g21\ng22\ntp56\nRI47\nI00\nI6\n((tg1\ntp57\nNNI0\nI-1\nI-1\nNtp58\nbg30\n(tRp59\n(g31\ng35\ng5\n(I0\nI0\nI2\nI2627\nS't\\x00\\x00\\x91\\x01\\x00\\x92\\x00\\x00\\x01d\\x00\\x00S'\np60\n(Ntp61\n(S'main_coro'\np62\nS'switch'\np63\ntp64\n(tS'<console>'\np65\nS'g'\np66\nI1\nS'\\x00\\x01'\np67\n(t(tI168686271\ntp68\nRp69\n((ttp70\n(tNNg15\n(g21\ng22\ntp71\nRI13\nI01\nI2\n((ttp72\nNNI0\nI-1\nI-1\nNtp73\nbtp74\nNc_stackless\n_return_main\np75\n(tRp76\ntp77\nbtp78\nb.",
'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__',
'user_coro': <_stackless.coroutine object at 0x00007f8c20ff86e0>, 'f': <function
f at 0x00007f8c20fdb830>, 'pickle': <module 'pickle' from
'/local/scratch/cs448/pypy-1.4.1-src/lib-python/modified-2.5.2/pickle.pyc'>,
'__doc__': None}, 13, True, 2, ((),), None, None, 0, -1, -1, None)
@@@SETSTATE <type '_stackless.coroutine'> (0, (<frame object at
0x00007f8c21018c80>, <frame object at 0x00007f8c21018d30>), None,
<_stackless.coroutine object at 0x0000000002319000>)
RPython traceback:
  File "implement_26.c", line 197241, in AppCoroutine_descr__setstate__
  File "implement_28.c", line 20393, in AppCoroutine_reconstruct_framechain
Fatal RPython error: AssertionError
Aborted

----------
effort: ???
messages: 2396
nosy: pypy-issue, smowton
priority: bug
release: 1.4
status: unread
title: RPython exception unpickling a coroutine whose creator's stack has shrunk since it was switch()'d

_______________________________________________________
PyPy development tracker <pypy-dev-issue at codespeak.net>
<https://codespeak.net/issue/pypy-dev/issue689>
_______________________________________________________