[IronPython] Performance of IronPython 2 Beta 4 and IronPython 1

[Michael Foord]

Hello all,

I've ported Resolver One to run on IronPython 2 Beta 4 to check for any 
potential problems (we will only do a *proper* port once IP 2 is out of 
beta).

The basic porting was straightforward and several bugs have been fixed 
since IP 2 B3 - many thanks to the IronPython team.

The good news is that Resolver One is only 30-50% slower than Resolver 
One on IronPython 1! (It was 300 - 400% slower on top of IP 2 B3.) 
Resolver One is fairly heavily optimised around the performance hotspots 
of IronPython 1, so we expect to have to do a fair bit of profiling and 
refactoring to readjust to the performance profile of IP 2.

Having said that, there are a few oddities (and the areas that slow down 
vary tremendously depending on which spreadsheet we use to benchmark it 
- making it fairly difficult to track down the hotspots).

We have one particular phase of spreadsheet calculation that takes 
0.4seconds on IP1 and around 6 seconds on IP2, so I have been doing some 
micro-benchmarking to try and identify the hotspot. I've certainly found 
part of the problem.

For those that are interested I've attached the very basic 
microbenchmarks I've been using. The nice thing is that in *general* IP2 
does outperform IP1.

The results that stand out in the other direction are:

Using sets with custom classes (that define '__eq__', '__ne__' and 
'__hash__') seems to be 6 times slower in IronPython 2.

Adding lists together is about 50% slower.

Defining functions seems to be 25% slower and defining old style classes 
about 33% slower. (Creating instances of new style classes is massively 
faster though - thanks!)

The code I used to test sets (sets2.py) is as follows:

from System import DateTime

class Thing(object):
    def __init__(self, val):
        self.val = val
   
    def __eq__(self, other):
        return self.val == other.val

    def __neq__(self):
        return not self.__eq__(other)
       
    def __hash__(self):
        return hash(self.val)
       
       
def test(s):
    a = set()
    for i in xrange(100000):
        a.add(Thing(i))
        a.add(Thing(i+1))
        Thing(i) in a
        Thing(i+2) in a
    return (DateTime.Now -s).TotalMilliseconds
   
s = DateTime.Now
print test(s)


Interestingly the time taken is exactly the same if I remove the 
definition of '__hash__'.

The full set of results below:

Results in milliseconds with a granularity of about 15ms and so an 
accuracy of +/- ~60ms.
All testing with 10 000 000 operations unless otherwise stated.

Empty loop (overhead):
    IP1: 421.9
    IP2: 438
   
Create instance newstyle:
    IP1: 20360
    IP2: 1109
   
Create instance oldstyle:
    IP1: 3766
    IP2: 3359
   
Function call:
    IP1: 937
    IP2: 906
   
Create function: 25% slower
    IP1: 2828
    IP2: 3640
   
Define newstyle (1 000 000):
    IP1: 42047
    IP2: 20484
   
Define oldstyle (1 000 000): 33% slower
    IP1: 1781
    IP2: 2671

Comparing (== and !=):
    IP1: 278597
    IP2: 117662
   
Sets (with numbers):
    IP1: 37095
    IP2: 30860

Lists (10 000): 50% slower
    IP1: 10422
    IP2: 16109

Recursion (10 000):
    IP1: 1125
    IP2: 1000
  
Sets2 (100 000): 600% slower
    IP1: 4984
    IP2: 30547


I'll be doing more as the 600% slow down for sets and the 50% slow down 
for lists accounts for some of the dependency analysis problem but not 
all of it.

Many Thanks

Michael Foord
--
http://www.resolversystems.com
http://www.ironpythoninaction.com



-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: comparing.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: createfunction.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0001.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: createnewstyle.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0002.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: createoldstyle.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0003.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: definenewstyle.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0004.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: defineoldstyle.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0005.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: emptytest.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0006.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: functioncall.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0007.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: lists.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0008.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: recursion.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0009.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sets2.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0010.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sets.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20080814/0a7fab57/attachment-0011.ksh>