[IronPython] pybench results for CPython and IronPython
M. David Peterson
xmlhacker at gmail.com
Sun Apr 22 10:08:22 CEST 2007
Hi Jim,
Your point regarding the performance difference between Mono and MS.NET is
an important one. It's one of those things that should have been a lot more
obvious than it was. None-the-less, thanks for bringing it to the surface!
One question: While I recognize the fact that for obvious reasons, ngen'ing
the IronPython assemblies as part of the distribution isn't an option (don't
worry, I'm not suggesting that it should or even could be an option ;-), has
anyone ran these same benchmarks against an ngen'd version of the assemblies
to compare the results? I realize that performance gains are no guarantee,
but I have found that more often than not the performance gains are fairly
significant, and worth the extra effort to ngen during the setup processes
in *most* cases. And in this case, given the fact that the CPython runtime
is compiled to native code and IronPython is still in all of its CIL glory
during the start-up process I wonder if its worth gaining a more
apples-to-apples comparison by running the same tests with ngen'd IP
assemblies?
I would suggest the same for Mono, but as of the last time I checked, there
is currently no support for AOT compiling .NET 2.0 assemblies (though I will
quickly check to verify, as its been a month or two since I last checked)
Worth pointing out: With these updated numbers, and given your statement,
IronPython just reached its 1.1 release and almost all of our recent work
> has been about compatibility and completeness.
I must admit, I am certainly looking forward to the result when "speed and
optimization" is added to the mix. :)
Thanks for all of your hard work!
On 4/22/07, Jim Hugunin <Jim.Hugunin at microsoft.com> wrote:
>
> I'd like to point out one fact about these benchmarks that seems to have
> been missed. Seo's numbers are for running IronPython 1.1 on Mono - not
> on the Microsoft .NET implementation. The Mono team has done excellent work
> with that project, but today the performance of the .NET implementation is
> still significantly better and is a much better platform for benchmarking
> IronPython's performance.
>
> I reran pybench on my laptop tonight on the RTM version of .NET that ships
> with Windows Vista - and has been available for XP for quite some time
> now. I compared with CPython-2.5.1 (the latest version).
>
> Running on .NET, I find that IronPython is faster on some tests and
> CPython is faster on others. We know that we still have performance work to
> do. After all, IronPython just reached its 1.1 release and almost all of
> our recent work has been about compatibility and completeness. However, the
> story is already quite interesting. Out of the 51 tests in pybench, CPython
> is more than 2x faster on 10 of the tests and IronPython is more than 2x
> faster on 9 of the tests. Depending on what your code does, either
> implementation could run faster.
>
> The most interesting cases to me are the 5 tests where CPython is more
> than 3x faster than IronPython and the other 5 tests where IronPython is
> more than 3x faster than CPython. CPython's strongest performance is in
> dictionaries with integer and string keys, list slicing, small tuples and
> code that actually throws and catches exceptions. IronPython's strongest
> performance is in calling builtin functions, if/then/else blocks, calling
> python functions, deep recursion, and try/except blocks that don't actually
> catch an exception.
>
> The places where IronPython is performing strongest are where it can use
> the .NET code generation and JIT optimization most effectively. The places
> where it is slowest are mainly in runtime library implementation of core
> datatypes. CPython's list and dictionary datatypes are written in C and
> have been hand-tuned for Python-style workloads over the past 10
> years. IronPython's datatypes are much newer and clearly need more tuning
> as the implementation matures.
>
> The only two tests that really stand out as showing a deep performance
> issue are the two exception handling ones. These are the only tests where
> either implementation is more than 4x faster than the other. IronPython is
> 10x faster on the try/catch without an exception and CPython is 30x faster
> when an exception is actually raised. This is a deliberate design decision
> within .NET to make code that doesn't throw exceptions run faster - even if
> that means slowing down code that does throw exceptions. I'm fairly
> confident this was the right decision - and even remember the day long ago
> when Guido was discussing Python's exception system and explained that he'd
> accept almost any slow-down to the exceptional case in return for removing a
> single instruction from the non-exceptional path.
>
> Thanks - Jim
>
> My results
>
> -------------------------------------------------------------------------------
> PYBENCH 2.0
>
> -------------------------------------------------------------------------------
> * using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32
> bit (Int
> el)]
> * disabled garbage collection
> * system check interval set to maximum: 2147483647
> * using timer: time.clock
>
>
> -------------------------------------------------------------------------------
> Benchmark: ipy11.pybench
>
> -------------------------------------------------------------------------------
>
> Rounds: 10
> Warp: 10
> Timer: time.time
>
> Machine Details:
> Platform ID: cli-32bit
> Processor:
>
> Python:
> Implementation: Python
> Executable: c:\ironpython-1.1\ipy.exe
> Version: 1.1.0
> Compiler: X
> Bits: 32bit
> Build: 0 0 (#0)
> Unicode: UCS2
>
>
>
> -------------------------------------------------------------------------------
> Comparing with: py25.pybench
>
> -------------------------------------------------------------------------------
>
> Rounds: 10
> Warp: 10
> Timer: time.clock
>
> Machine Details:
> Platform ID: Microsoft-Windows-32bit-WindowsPE
> Processor:
>
> Python:
> Implementation: Python
> Executable: c:\python25\python.exe
> Version: 2.5.1
> Compiler: MSC v.1310 32 bit (Intel)
> Bits: 32bit
> Build: Apr 18 2007 08:51:08 (#r251:54863)
> Unicode: UCS2
>
>
> Test minimum run-time average run-time
> this other diff this other
> diff
>
> -------------------------------------------------------------------------------
> BuiltinFunctionCalls: 44ms 181ms -75.7% 48ms 182ms -
> 73.7%
> BuiltinMethodLookup: 335ms 160ms +109.1% 344ms 162ms
> +112.0%
> CompareFloats: 99ms 111ms -11.3% 105ms 112ms
> -6.0%
> CompareFloatsIntegers: 63ms 125ms -49.7% 65ms 126ms -
> 48.7%
> CompareIntegers: 102ms 114ms -10.4% 105ms 115ms
> -8.9%
> CompareInternedStrings: 175ms 126ms +39.0% 180ms
> 127ms +41.6%
> CompareLongs: 111ms 105ms +5.3% 114ms 106ms
> +7.2%
> CompareStrings: 167ms 126ms +32.7% 172ms
> 127ms +35.1%
> CompareUnicode: 121ms 125ms -3.4% 123ms 126ms
> -2.0%
> ConcatStrings: 482ms 272ms +77.2% 533ms
> 275ms +93.4%
> ConcatUnicode: 305ms 216ms +41.0% 350ms
> 217ms +61.5%
> CreateInstances: 102ms 148ms -31.1% 106ms 149ms -
> 28.9%
> CreateNewInstances: 326ms 131ms +148.0% 335ms 133ms
> +151.9%
> CreateStringsWithConcat: 228ms 147ms +55.9% 238ms
> 148ms +61.0%
> CreateUnicodeWithConcat: 91ms 154ms -41.2% 94ms 157ms -
> 40.4%
> DictCreation: 137ms 105ms +30.0% 143ms
> 106ms +33.9%
> DictWithFloatKeys: 301ms 230ms +30.8% 306ms
> 232ms +32.0%
> DictWithIntegerKeys: 343ms 106ms +223.0% 351ms 108ms
> +224.7%
> DictWithStringKeys: 388ms 102ms +282.1% 395ms 102ms
> +285.1%
> ForLoops: 39ms 95ms -59.0% 40ms 97ms -
> 58.5%
> IfThenElse: 35ms 111ms -68.9% 37ms 112ms -
> 66.8%
> ListSlicing: 468ms 155ms +201.3% 477ms 157ms
> +204.4%
> NestedForLoops: 55ms 120ms -54.1% 57ms 122ms -
> 53.0%
> NormalClassAttribute: 276ms 127ms +117.7% 282ms 129ms
> +119.7%
> NormalInstanceAttribute: 106ms 119ms -11.0% 109ms 120ms
> -9.7%
> PythonFunctionCalls: 37ms 130ms -71.7% 40ms 131ms -
> 69.5%
> PythonMethodCalls: 127ms 158ms -19.6% 132ms 159ms -
> 16.8%
> Recursion: 49ms 177ms -72.5% 50ms 179ms -
> 71.8%
> SecondImport: 189ms 126ms +50.4% 195ms
> 127ms +53.6%
> SecondPackageImport: 196ms 134ms +45.8% 202ms
> 136ms +48.5%
> SecondSubmoduleImport: 266ms 177ms +50.6% 272ms
> 180ms +51.3%
> SimpleComplexArithmetic: 97ms 148ms -34.4% 101ms 149ms -
> 32.0%
> SimpleDictManipulation: 311ms 118ms +164.4% 318ms 118ms
> +168.0%
> SimpleFloatArithmetic: 64ms 119ms -46.6% 69ms 121ms -
> 43.2%
> SimpleIntFloatArithmetic: 41ms 100ms -59.4% 43ms 102ms -
> 58.2%
> SimpleIntegerArithmetic: 43ms 100ms -57.2% 44ms 101ms -
> 56.4%
> SimpleListManipulation: 121ms 104ms +16.6% 124ms
> 105ms +18.3%
> SimpleLongArithmetic: 126ms 113ms +11.2% 131ms
> 115ms +14.1%
> SmallLists: 223ms 154ms +45.0% 229ms
> 156ms +47.0%
> SmallTuples: 508ms 144ms +252.2% 523ms 145ms
> +259.7%
> SpecialClassAttribute: 275ms 124ms +121.2% 283ms 126ms
> +124.5%
> SpecialInstanceAttribute: 105ms 210ms -50.1% 109ms 212ms -
> 48.5%
> StringMappings: 342ms 633ms -46.0% 351ms 637ms -
> 44.9%
> StringPredicates: 222ms 217ms +2.3% 228ms 219ms
> +4.4%
> StringSlicing: 245ms 162ms +51.4% 260ms
> 163ms +59.4%
> TryExcept: 7ms 100ms -93.4% 11ms 102ms -
> 89.4%
> TryRaiseExcept: 3413ms 114ms +2896.2% 3453ms 115ms
> +2910.6%
> TupleSlicing: 227ms 150ms +51.6% 235ms
> 151ms +54.9%
> UnicodeMappings: 227ms 126ms +79.6% 233ms
> 129ms +81.4%
> UnicodePredicates: 218ms 129ms +68.8% 224ms
> 132ms +70.4%
> UnicodeSlicing: 211ms 182ms +15.8% 223ms
> 184ms +21.3%
>
> -------------------------------------------------------------------------------
> Totals: 12781ms 7659ms +66.9%
> 13191ms 7741ms +70.4%
>
> (this=ipy11.pybench, other=py25.pybench)
> _______________________________________________
> users mailing list
> users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
--
/M:D
M. David Peterson
http://mdavid.name | http://www.oreillynet.com/pub/au/2354 |
http://dev.aol.com/blog/3155
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20070422/d527d587/attachment.html>
More information about the Ironpython-users
mailing list