Bizarre additional calling overhead.

[Gabriel Genellina]

En Fri, 02 Nov 2007 21:07:19 -0300, Matimus <mccredie at gmail.com> escribió:

> On Nov 2, 3:08 pm, "Chris Mellon" <arka... at gmail.com> wrote:
>> >>> def test_func():
>>
>> ...     pass
>> ...>>> import new
>> >>> test_func2 = new.function(test_func.func_code, {}, "test_func2")
>> >>> test_func2
>>
>> <function test_func2 at 0x01B8C2F0>>>> test_func
>>
>> <function test_func at 0x01B8C270>>>> import timeit
>> >>> tf = timeit.Timer("test_func()", "from __main__ import test_func")
>> >>> tf.repeat()
>>
>> [0.2183461704377247, 0.18068215314489791, 0.17978585841498085]>>> tf2 =  
>> timeit.Timer("test_func2()", "from __main__ import test_func2")
>> >>> tf2.repeat()
>>
>> [0.40015390239890891, 0.35893452879396648, 0.36034628133737456]
>>
>> Why almost twice the calling overhead for a dynamic function?
>
> So, I don't have an official explanation for why it takes twice as
> long, but the only difference between the two functions I could find
> was that test_func.func_globals was set to globals() and
> test_func2.func_globals was an empty dict. When I re-created
> test_func2 with globals set to globals() it ran just as fast as
> test_func.

Yes - and that's a very important difference. Not because it's empty, nor  
because it's not the same as globals(), but because the builtins as seen  
by the function are not from the original __builtin__ module.

 From the Python Reference Manual, section 4.1:
	The built-in namespace associated with the execution of a code block is  
actually found by looking up the name __builtins__ in its global  
namespace; [...] __builtins__ can be set to a user-created dictionary to  
create a weak form of restricted execution.

 From the Library Reference, section 28, Restricted Execution:
	The Python run-time determines whether a particular code block is  
executing in restricted execution mode based on the identity of the  
__builtins__ object in its global variables: if this is (the dictionary  
of) the standard __builtin__ module, the code is deemed to be  
unrestricted, else it is deemed to be restricted.

Section 3.2, when describing frame objects:
	f_restricted is a flag indicating whether the function is executing in  
restricted execution mode

Let's try to see if this is the case. Getting the f_restricted flag is a  
bit hard with an empty globals(), so let's pass sys as an argument:

py> def test_func(sys):
...     print "restricted", sys._getframe().f_restricted
...
py> import sys, new
py> test_func2 = new.function(test_func.func_code, {}, "test_fun
c2")
py> test_func(sys)
restricted False
py> test_func2(sys)
restricted True

So test_func2 is running in restricted mode. That't the reason it is so  
slow. Even if the restricted mode implementation is now considered unsafe  
-because new style classes provide many holes to "escape" from the "jail"-  
the checks are still being done.

Ok, now let's try to avoid entering restricted mode:

py> import __builtin__
py> test_func3 = new.function(test_func.func_code, {'__builtins_
_':__builtin__}, "test_func3")
py> test_func3(sys)
restricted False

And now, repeating the timings when __builtins__ is correctly set to the  
builtin module __builtin__ (!), shows no difference with the original  
function. So this was the cause of the slow down.

The rule is: unless you *actually* want to execute code in restricted  
mode, pass globals() when building a new function object, or at least, set  
'__builtins__' to the __builtin__ module

-- 
Gabriel Genellina