[IronPython] Performance of IronPython 2 Beta 4 and IronPython 1
Michael Foord
fuzzyman at voidspace.org.uk
Thu Aug 14 23:34:23 CEST 2008
Curt Hagenlocher wrote:
> Can you tell us about the environment you're running these tests in?
> Most interesting are the specific OS and the version of mscorwks.dll.
>
It's Win XP SP 3. What's the easiest way of telling you the specific
version of mscorwks.dll? I'll get you more details from work tomorrow.
Michael
> On Thu, Aug 14, 2008 at 9:41 AM, Michael Foord
> <fuzzyman at voidspace.org.uk <mailto:fuzzyman at voidspace.org.uk>> wrote:
>
> Just for fun I also compared with CPython. The results are
> interesting, I'll turn it into a blog post of course...
>
>
> Results in milliseconds with a granularity of about 15ms and so an
> accuracy of +/- ~60ms.
> All testing with 10 000 000 operations unless otherwise stated.
>
> The version of Python I compared against was Python 2.4.
>
> Empty loop (overhead):
> IP1: 422
> IP2: 438
> Py: 3578
>
> Create instance newstyle:
> IP1: 20360
> IP2: 1109
> Py: 4063
>
> Create instance oldstyle:
> IP1: 3766
> IP2: 3359
> Py: 4797
>
> Function call:
> IP1: 937
> IP2: 906
> Py: 3313
>
> Create function: 25% slower
> IP1: 2828
> IP2: 3640
> Py: 2766
>
> Define newstyle (1 000 000):
> IP1: 42047
> IP2: 20484
> Py: 23921
>
> Define oldstyle (1 000 000): 33% slower
> IP1: 1781
> IP2: 2671
> Py: 2108
>
>
> Comparing (== and !=):
> IP1: 278597
> IP2: 117662
> Py: 62423
> Sets:
> IP1: 37095
> IP2: 30860
> Py: 8047
>
>
> Lists (10 000): 50% slower
> IP1: 10422
> IP2: 16109
> Py: 6094
>
>
> Recursion (10 000):
> IP1: 1125
> IP2: 1000
> Py: 3609
>
> Sets2 (100 000): 500% slower
> IP1: 4984
> IP2: 30547
> Py: 1203
>
> func_with_args:
> IP1: 6312
> IP2: 5906
> Py: 11250
>
> method_with_args:
> IP1: 20594
> IP2: 11813
> Py: 14875
> method_with_kwargs:
> IP1: 27953
> IP2: 11187
> Py: 20032
> import: 15% slower
> IP1: 28469
> IP2: 32000
> Py: 25782
>
> global: 20% slower
> IP1: 1047
> IP2: 1203
> Py: 4141
>
> Exceptions (100000): 40% slower
> IP1: 4703
> IP2: 6125
> Py: 266
> Engine execution: 8000% slower!!
> IP1: 1600
> IP2: 115002
>
>
> Michael Foord wrote:
>
> Hello all,
>
> I've ported Resolver One to run on IronPython 2 Beta 4 to
> check for any potential problems (we will only do a *proper*
> port once IP 2 is out of beta).
>
> The basic porting was straightforward and several bugs have
> been fixed since IP 2 B3 - many thanks to the IronPython team.
>
> The good news is that Resolver One is only 30-50% slower than
> Resolver One on IronPython 1! (It was 300 - 400% slower on top
> of IP 2 B3.) Resolver One is fairly heavily optimised around
> the performance hotspots of IronPython 1, so we expect to have
> to do a fair bit of profiling and refactoring to readjust to
> the performance profile of IP 2.
>
> Having said that, there are a few oddities (and the areas that
> slow down vary tremendously depending on which spreadsheet we
> use to benchmark it - making it fairly difficult to track down
> the hotspots).
>
> We have one particular phase of spreadsheet calculation that
> takes 0.4seconds on IP1 and around 6 seconds on IP2, so I have
> been doing some micro-benchmarking to try and identify the
> hotspot. I've certainly found part of the problem.
>
> For those that are interested I've attached the very basic
> microbenchmarks I've been using. The nice thing is that in
> *general* IP2 does outperform IP1.
>
> The results that stand out in the other direction are:
>
> Using sets with custom classes (that define '__eq__', '__ne__'
> and '__hash__') seems to be 6 times slower in IronPython 2.
>
> Adding lists together is about 50% slower.
>
> Defining functions seems to be 25% slower and defining old
> style classes about 33% slower. (Creating instances of new
> style classes is massively faster though - thanks!)
>
> The code I used to test sets (sets2.py) is as follows:
>
> from System import DateTime
>
> class Thing(object):
> def __init__(self, val):
> self.val = val
> def __eq__(self, other):
> return self.val == other.val
>
> def __neq__(self):
> return not self.__eq__(other)
> def __hash__(self):
> return hash(self.val)
> def test(s):
> a = set()
> for i in xrange(100000):
> a.add(Thing(i))
> a.add(Thing(i+1))
> Thing(i) in a
> Thing(i+2) in a
> return (DateTime.Now -s).TotalMilliseconds
> s = DateTime.Now
> print test(s)
>
>
> Interestingly the time taken is exactly the same if I remove
> the definition of '__hash__'.
>
> The full set of results below:
>
> Results in milliseconds with a granularity of about 15ms and
> so an accuracy of +/- ~60ms.
> All testing with 10 000 000 operations unless otherwise stated.
>
> Empty loop (overhead):
> IP1: 421.9
> IP2: 438
> Create instance newstyle:
> IP1: 20360
> IP2: 1109
> Create instance oldstyle:
> IP1: 3766
> IP2: 3359
> Function call:
> IP1: 937
> IP2: 906
> Create function: 25% slower
> IP1: 2828
> IP2: 3640
> Define newstyle (1 000 000):
> IP1: 42047
> IP2: 20484
> Define oldstyle (1 000 000): 33% slower
> IP1: 1781
> IP2: 2671
>
> Comparing (== and !=):
> IP1: 278597
> IP2: 117662
> Sets (with numbers):
> IP1: 37095
> IP2: 30860
>
> Lists (10 000): 50% slower
> IP1: 10422
> IP2: 16109
>
> Recursion (10 000):
> IP1: 1125
> IP2: 1000
> Sets2 (100 000): 600% slower
> IP1: 4984
> IP2: 30547
>
>
> I'll be doing more as the 600% slow down for sets and the 50%
> slow down for lists accounts for some of the dependency
> analysis problem but not all of it.
>
> Many Thanks
>
> Michael Foord
> --
> http://www.resolversystems.com <http://www.resolversystems.com/>
> http://www.ironpythoninaction.com
> <http://www.ironpythoninaction.com/>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
>
> --
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/
> http://www.trypython.org/
> http://www.ironpython.info/
> http://www.resolverhacks.net/
> http://www.theotherdelia.co.uk/
>
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/
http://www.trypython.org/
http://www.ironpython.info/
http://www.theotherdelia.co.uk/
http://www.resolverhacks.net/
More information about the Ironpython-users
mailing list