[IronPython] parallel importing
Michael Foord
fuzzyman at voidspace.org.uk
Mon Oct 13 19:16:49 CEST 2008
Curt Hagenlocher wrote:
> The" .NET people" might say they already have an answer which involves
> precompiling the IL with NGEN. And in fact, using DLR precompilation
> and NGEN with IronPython results in some pretty nice performance
> improvements.
>
> But using IronPython directly against .py files incurs several
> expenses that aren't part of normal .NET startup. We have to parse
> the .py file, build DLR-level expression trees from the language tree
> and build IL from the expression tree. Only then are we at the point
> where a C# starts, and can JIT the IL.
>
> It's worth noting that CPython has to do similar work when it first
> loads a .py file. It, too, has to parse into a tree structure and
> generate byte codes for the tree. And even though the CPython byte
> code is at a higher level than the IL generated by the DLR, CPython
> still has a performance issue -- which is addressed by storing the
> byte codes as a .pyc file.
The difference is still orders of magnitude smaller though...
We're looking forward to moving to precompiled binaries with IronPython
2 at Resolver Systems - but there we will still have situations where we
need to import from py files.
Michael
>
> I'm hoping that over the next year, we'll come up with a convention
> (like .pyc) that allows subsequent runs of a program to benefit from
> the work done in previous runs -- without having to go through the
> manual step of DLR precompilation.
>
> On Mon, Oct 13, 2008 at 9:57 AM, Vernon Cole <vernondcole at gmail.com
> <mailto:vernondcole at gmail.com>> wrote:
>
> Help me out here. I have seen many references to schemes to
> optimize import speeds on Iron Python. Isn't this an attempt to
> make up for the snail-like startup speeds suffered by *all* .net
> applications? It seems to me that when Iron Python finally starts
> running, it imports and performs very well. The slowdown is in
> getting the .net engine started. Try comparing startup times for
> "Hello World" in C Python and Iron.
> It is my experience that I can identify .net programs in
> general by the fact that I have double-clicked its desktop icon
> three times (thinking I had missed it somehow) before the first
> instance displays its splash screen, and then the next two display
> an "already running" dialog.
> Any chance of getting the .net people to help out with this?
> --
> Vernon Cole
>
>
> On Mon, Oct 13, 2008 at 10:00 AM, Dino Viehland
> <dinov at microsoft.com <mailto:dinov at microsoft.com>> wrote:
>
> This is all still on 1.x, right? It looks like #1 is fixed in
> 2.0 (we are locking but on the wrong object in 1.x).
>
> #2 is still broken in 2.x though as well.
>
> -----Original Message-----
> From: users-bounces at lists.ironpython.com
> <mailto:users-bounces at lists.ironpython.com>
> [mailto:users-bounces at lists.ironpython.com
> <mailto:users-bounces at lists.ironpython.com>] On Behalf Of
> Kamil Dworakowski
> Sent: Monday, October 13, 2008 4:45 AM
> To: users at lists.ironpython.com <mailto:users at lists.ironpython.com>
> Subject: [IronPython] parallel importing
>
> Importing time is such a pain that I wanted to do it in
> parallel on
> many threads. I figured that if the modules where imported in
> such a
> way that:
>
> no two threads import the same module at the same time
>
> everything would be fine. To ensure that condition, it is
> enough to
> build a dependency graph and import based on that.
>
> I did it: Resolver One start up time improved by 20% on a two-core
> machine. But it crashes, surprisingly on single core machines
> it is
> more often (6 crashes on 200 starts).
>
> So far I have identified two causes for crashes:
>
> 1. One thread imports a module with class B inside while
> another is
> importing a module with class C inside. If B and C are
> subclasses of
> A, it can result in IndexOutOfRangeException being raised,
> when, under
> the hood, IronPython.Runtime.Types.DynamicType.AddSubclass is
> being
> executed.
>
> 2. Attributes on .NET modules are loaded lazily, so importing
> namespaces only is not enough. Attribute getting from reflected
> packages is not thread safe. Looks like I would have to import
> every
> class explicitly (would that be enough?).
>
> Second cause would be pretty easy to address, but I'm not so sure
> about the first one. Are there any more potential points of
> problems?
> I am beginning to think I was to optimistic about all of this
> importing on multiple cores, but if these are the only ones it
> could
> probably be still fixed.
>
> If anyone is interested the code for it is on github:
> http://github.com/luntain/ipy-parallel-import.
>
> --
> Kamil Dworakowski
> Resolver Systems Ltd.
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com <mailto:Users at lists.ironpython.com>
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/
http://www.trypython.org/
http://www.ironpython.info/
http://www.resolverhacks.net/
http://wwww.theotherdelia.co.uk/
More information about the Ironpython-users
mailing list