Question about unreasonable slowness

[Steven D'Aprano]

On Thu, 16 Nov 2006 12:45:18 -0800, allenjo5 wrote:

> [ Warning: I'm new to Python.  Don't know it at all really yet, but had
> to examine some 3rd party code because of performance problems with it.
> ]
> 
> Here's a code snippet:
> 
> i = 0
> while (i < 20):
>   i = i + 1


You probably want to change that to:

for i in range(20):

If 20 is just a place-holder, and the real value is much bigger, change
the range() to xrange().


>   (shellIn, shellOut) = os.popen4("/bin/sh -c ':'")  # for testing, the
> spawned shell does nothing
>   print 'next'
> #  for line in shellOut:
> #       print line
> 
> On my system (AIX 5.1 if it matters, with Python 2.4.3), this simple
> loop spawning 20 subshells takes .75 sec.  Ok, that's reasonable.  Now,
> if I uncomment the two commented lines, which loop over the empty
> shellOut array, the progam now takes 11 secs.   That slowdown seems
> very hard to believe.  Why should it slow down so much?

What are you using to time the code?

Replacing print statements with "pass", I get these results:

>>> import timeit
>>>
>>> def test():
...     i = 0
...     while (i < 20):
...             i = i + 1
...             (shellIn, shellOut) = os.popen4("/bin/sh -c ':'")
...             pass # print 'next'
...             for line in shellOut:
...                     pass # print line
...
>>> timeit.Timer("test()", "from __main__ import test\nimport os").timeit(1)
0.49781703948974609
>>> timeit.Timer("test()", "from __main__ import test\nimport os").timeit(100)
54.894074201583862

About 0.5 second to open and dispose of 20 subshells, even with the "for
line in shellOut" loop.


I think you need some more fine-grained testing to determine whether the
slowdown is actually happening inside the "for line in shellOut" loop or
inside the while loop or when the while loop completes.



-- 
Steven.