[Python-bugs-list] [ python-Bugs-793826 ] using itertools.izip to mutate tuples

Fri Aug 29 14:22:43 EDT 2003

Bugs item #793826, was opened at 2003-08-23 12:24
Message generated for change (Comment added) made by rhettinger
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=793826&group_id=5470

Category: Python Interpreter Core
Group: Python 2.3
Status: Closed
Resolution: Wont Fix
Priority: 5
Submitted By: Armin Rigo (arigo)
Assigned to: Raymond Hettinger (rhettinger)
Summary: using itertools.izip to mutate tuples

Initial Comment:
Not sure how crucial this exactly is, but the
itertools.izip() can be abused to create a tuple that
can be mutated once. I couldn't find a way to crash
anything, however, but I wouldn't leave such a
possibility open in the Python interpreter nevertheless.

. from itertools import imap, izip
. 
. def mutatingtuple(tuple1, f, tuple2):
.     # this builds a tuple t which is a copy of tuple1,
.     # then calls f(t), then mutates t to be equal to
tuple2
.     # (needs len(tuple1) == len(tuple2)).
.     def g(value, first=[1]):
.         if first:
.             del first[:]
.             f(z.next())
.         return value
.     items = list(tuple2)
.     items[1:1] = list(tuple1)
.     gen = imap(g, items)
.     z = izip(*[gen]*len(tuple1))
.     z.next()
. 
. def f(t):
.     global T
.     T = t
.     print T
. 
. mutatingtuple((1,2,3), f, (4,5,6)) # print T -> (1, 2, 3)
. print T                            # print T -> (4, 5, 6)

----------------------------------------------------------------------

>Comment By: Raymond Hettinger (rhettinger)
Date: 2003-08-29 15:22

Message:
Logged In: YES 
user_id=80475

Please do submit a patch (attached to this report) and I'll 
take a look at it.

In your judgement, is this a problem that would negatively 
impact real code or does it only involve deliberate contortions 
and mischief?

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2003-08-28 08:13

Message:
Logged In: YES 
user_id=4771

Ah ! I knew there was a flaw ! Attached is a script
(iziptest4.py) that uses all the above plus yet another
minor hole to actually decrement the reference counter of an
arbitrary object. Have you already seen  Python dying on you
with 'Fatal Python error: deallocating None' ?  :-)

The original idea came from lurking at izip_next(), when I
was thinking about using gc.get_referrers() to grab the
internally stored tuple (see bug report #793822). Then I
realized that the recursion trick would be sufficient for
izip_next(). Finally, the attached script exploits the
common problem of DECREF'ing an object that is still in the
tuple *before* actually storing something new at that
position, which is only a problem if one can actually see
the tuple from the __del__ method...

I can make a patch if you wish.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2003-08-27 01:00

Message:
Logged In: YES 
user_id=80475

The bit about "only working when the tuple is not being 
used" relates to the dependency on the C code that checks 
that the refcount is one:
   d = {(1,2,3):value} # Create a tuple with one reference
   # now try to mutate it so that "value" becomes inaccessible

I'm curious about how the original idea arose and what was 
the thought process you used to develop the clever, 
exploitation script.

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2003-08-26 11:22

Message:
Logged In: YES 
user_id=4771

I agree with your general analysis : it doesn't seem
harmful. I don't understand the bit about "only works when
the tuple is not being used", however. In the example script
the function 'f' stores a copy of the tuple; any number of
references can be hold accross the mutation. Here is another
example:

g = {}
def f(t): global T; T=t; g[T] = 'a'
mutatingtuple((1,2,3), f, (4,5,6)) # print T -> (1, 2, 3)
g[T] = 'b'
print g

This prints {(4, 5, 6): 'a', (4, 5, 6): 'b'}.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2003-08-24 14:17

Message:
Logged In: YES 
user_id=80475

After more thought, I think this should be closed because the 
attached tuple mutation script only works when the tuple is 
not being used (there can be no other references to it) and in 
that situation, it doesn't matter if a deliberate contrived 
effort is made to mutate.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2003-08-23 18:30

Message:
Logged In: YES 
user_id=80475

I'm inclined to not worry about this one.  The tuple re-use 
technique appears elsewhere in the code base and has not 
had any problems in practice. 

The exploitation script is impressive and it makes clear that 
mutation requires combining a number of rare, deliberate 
steps that don't occur in practice, i.e. the intensional 
assignment to a tuple inside a self-referential iterator that 
refers to itself by the name in an enclosing scope.

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2003-08-23 12:28

Message:
Logged In: YES 
user_id=4771

Assigned to Raymond on the rationale that he's the author of
itertoolsmodule.c. I can propose a patch to
itertoolsmodule.c if you wish.

Also attached above example in a file.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=793826&group_id=5470