[IronPython] Jit leak in defaultdict for Ipy 2.6.2

[Idan Zaltzberg]

After further testing I find very weird behaviors of the Jit.

For instance, looks like if use some pattern too many times, The Jit counter
goes to infinity, while otherwise it stabilizes.

It would be very helpful to get printouts from ipy whenever it creates an a
object that requires jitting.

Are there a few places I can Console.writeline in the IronPython code that
should cover most of these cases?

Is there any cache mechanism you are aware of, that might cause this
behavior where "over usage" generates infinite Jitting?



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland
*Sent:* Monday, November 29, 2010 8:23 PM
*To:* Discussion of IronPython
*Subject:* Re: [IronPython] Jit leak in defaultdict for Ipy 2.6.2



The closest non-windbg solution I could think of might be CLR Profiler which
can probably report the number of dynamic method objects which are alive.
Windbg can also do this when you dump the heap and is probably over all
actually easier to use in this case as you can attach, do !DumpHeap –stat,
and then detach.



Just one more data point which might help you understand what’s going on –
the JITed methods in this case are probably adding 4k of memory to the cost
of each defaultdict instance as long as the defaultdict is alive.



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg
*Sent:* Monday, November 29, 2010 9:59 AM
*To:* Discussion of IronPython
*Subject:* Re: [IronPython] Jit leak in defaultdict for Ipy 2.6.2



Thanks.

How I suspect there is a leak in the JIT of my application, up until now I
thought that if the performance counter for "# of methods jitted" is
constantly rising then that means exactly that. From your reply I understand
that this is not the case.

Can you tell how can I know how many uncollectible JIT objects I have so
that I can trace them (preferably without using windbg)



Thanks again…



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland
*Sent:* Monday, November 29, 2010 7:41 PM
*To:* Discussion of IronPython
*Subject:* Re: [IronPython] Jit leak in defaultdict for Ipy 2.6.2



Defaultdict is creating a new invoke binder – it should be getting the
binder from PythonContext using the Invoke(CallSignature) method.  Because
it creates a new binder each time we are getting no caching of the rules
across defaultdict instances and it’ll end up generating a new method to
handle the missing call.  It is collectible (so not really a leak) but it is
really bad from a performance perspective.



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg
*Sent:* Monday, November 29, 2010 4:10 AM
*To:* Discussion of IronPython
*Subject:* [IronPython] Jit leak in defaultdict for Ipy 2.6.2



Hi,

I have noticed the following method always adds a jitted method (looking at
the ".NET CLR Jit" performance counter) when it is run:

def f():

                d = defaultdict(int)

                d[0]



I created my own implementation of defaultdict (in ipy):

class defaultdict(dict):

    def __init__(self, cls):

        super(defaultdict, self).__init__()

        self.cls = cls

    def __getitem__(self, key):

        if key not in self:

            self[key] = self.cls()

        return super(defaultdict, self).__getitem__(key)



And I noticed that it does not leak JIT and it works 200 times faster when
running the method f().

Can you please look why this happens in the current implementation?

Also I was wondering if there are any other utility methods that use similar
code and probably will have the same problem.



Thanks,

Idan zalzberg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20101202/e0cf2759/attachment.html>