[IronPython] parallel importing

[Kamil Dworakowski]

Thx, Dino. I didn't see this email when replying to your previous
message, Gmail is not yet perfect :(. At Resolver Systems we don't
want to ship customly modified IronPython dlls.

On Tue, Oct 14, 2008 at 12:10 AM, Dino Viehland <dinov at microsoft.com> wrote:
> Attached is a unified diff of the changes (supposedly equivalent to diff -u).  Let me know if some other diff format would be better.
>
> -----Original Message-----
> From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Kamil Dworakowski
> Sent: Monday, October 13, 2008 9:42 AM
> To: Discussion of IronPython
> Subject: Re: [IronPython] parallel importing
>
> Could you provide a patch?
>
> On Mon, Oct 13, 2008 at 5:31 PM, Dino Viehland <dinov at microsoft.com> wrote:
>> Making the code changes is easy - the hard part is really doing a new 1.x release and all of the sign off work that entails.  We haven't ruled it out but we sure would like to avoid it if possible.
>>
>> -----Original Message-----
>> From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Kamil Dworakowski
>> Sent: Monday, October 13, 2008 9:07 AM
>> To: Discussion of IronPython
>> Subject: Re: [IronPython] parallel importing
>>
>> Would that be easy to backport fix for #2 to 1.x branch?
>>
>> On Mon, Oct 13, 2008 at 5:00 PM, Dino Viehland <dinov at microsoft.com> wrote:
>>> This is all still on 1.x, right?  It looks like #1 is fixed in 2.0 (we are locking but on the wrong object in 1.x).
>>>
>>> #2 is still broken in 2.x though as well.
>>>
>>> -----Original Message-----
>>> From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Kamil Dworakowski
>>> Sent: Monday, October 13, 2008 4:45 AM
>>> To: users at lists.ironpython.com
>>> Subject: [IronPython] parallel importing
>>>
>>> Importing time is such a pain that I wanted to do it in parallel on
>>> many threads. I figured that if the modules where imported in such a
>>> way that:
>>>
>>> no two threads import the same module at the same time
>>>
>>> everything would be fine. To ensure that condition, it is enough to
>>> build a dependency graph and import based on that.
>>>
>>> I did it: Resolver One start up time improved by 20% on a two-core
>>> machine. But it crashes, surprisingly on single core machines it is
>>> more often (6 crashes on 200 starts).
>>>
>>> So far I have identified two causes for crashes:
>>>
>>> 1. One thread imports a module with class B inside while another is
>>> importing a module with class C inside. If B and C are subclasses of
>>> A, it can result in IndexOutOfRangeException being raised, when, under
>>> the hood, IronPython.Runtime.Types.DynamicType.AddSubclass is being
>>> executed.
>>>
>>> 2. Attributes on .NET modules are loaded lazily, so importing
>>> namespaces only is not enough. Attribute getting from reflected
>>> packages is not thread safe. Looks like I would have to import every
>>> class explicitly (would that be enough?).
>>>
>>> Second cause would be pretty easy to address, but I'm not so sure
>>> about the first one. Are there any more potential points of problems?
>>> I am beginning to think I was to optimistic about all of this
>>> importing on multiple cores, but if these are the only ones it could
>>> probably be still fixed.
>>>
>>> If anyone is interested the code for it is on github:
>>> http://github.com/luntain/ipy-parallel-import.
>>>
>>> --
>>> Kamil Dworakowski
>>> Resolver Systems Ltd.
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.ironpython.com
>>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.ironpython.com
>>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>> _______________________________________________
>> Users mailing list
>> Users at lists.ironpython.com
>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>
>