[IronPython] HostCodeHeap leakage?

[Idan Zaltzberg]

Thanks for the elaborate answer, I'll look into it.



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland
*Sent:* Wednesday, October 13, 2010 10:01 PM
*To:* Discussion of IronPython
*Subject:* Re: [IronPython] HostCodeHeap leakage?



If you build from source you could set some breakpoints in AssemblyGen.cs in
the DefineType method.  You can also set one in DelegateUtils.cs
*and*DelegateHelpers.cs in DefineDelegateType.  I think those are all
the places
where we are creating uncollectible types.  If we’re continuously hitting
those breakpoints after you believe your app has reached steady state then
something is going wrong.



This code
http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads
me to believe the HostCodeHeap might also be used for DynamicMethods.
If that is the case then the other place to look would be if FunctionCode
objects are being re-created repeatedly.  That will happen if there’s
exec/eval/compile calls which are happening and if those objects are being
kept alive then we could be growing the heap over time.



There’s also some complicated code which deals with keeping a list of all
code that is alive.  We do cleanup this list, and the list is a list of weak
references so it shouldn’t actually keep the code alive, but you could put
some breakpoints at FunctionCode.RegisterFunctionCode and
FunctionCode.CodeCleanup to see if that list is growing boundlessly (which
it would be if something was keeping code objects alive after an
exec/eval/compile).



Another place where code generation could be occurring would be w/ regexes.
If you are dynamically generating reg-exes, or executing a huge different
variety of them over time, and they’re compiled, then the compiled regexes
could be staying in memory.  There is a regex cache and you can clear it by
calling re.purge().  But it should only cache up to 100 regexes.



A final possible thing to investigate might be what happens if you throw
away the entire ScriptEngine instance.  Here you could try re-cycling the
ScriptEngine say every 6 hours and see if the problem goes away.  If that
fixes the problem then it’s likely that it is one of the things I mentioned
(or some other cache that’s per-runtime).  At least that would start to
narrow it down vs. some potentially global state (like the subtype list
which is shared across ScriptEngines).



That’s a bunch of different things to look at – hopefully it’ll give some
insight into what’s going on and help track down the issue.



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg
*Sent:* Wednesday, October 13, 2010 12:12 AM
*To:* Discussion of IronPython
*Subject:* Re: [IronPython] HostCodeHeap leakage?



I tried what you suggested (changed setup.DebugMode = false;)

But still I get the same behavior:

The "Jit Code Heap" increases from about 17MB to 230MB in 2 days.

Is there a way to verify from the IronPython code that DebugMode is off?

Is there anything else I can do (other startup settings?) to
decrease/understand the increase in HostCodeHeap objects?

Thanks.



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland
*Sent:* Tuesday, October 05, 2010 6:59 PM
*To:* Discussion of IronPython
*Subject:* Re: [IronPython] HostCodeHeap leakage?



Yep, DebugMode is the same as –X:Debug.  In general I’d suggest making this
configurable somehow and only turn it on if you’re actually debugging.  It’s
unfortunate that we can’t offer both debugging & collectability but right
now that’s simply a limitation of the CLR and/or our lack of a separate VS
debug engine which can debug Python code.



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg
*Sent:* Tuesday, October 05, 2010 9:09 AM
*To:* Discussion of IronPython
*Subject:* Re: [IronPython] HostCodeHeap leakage?



Im running using the engine from a hosting app.

We have these lines in the startup:

ScriptRuntimeSetup setup = new ScriptRuntimeSetup();
setup.DebugMode = true;

ScriptRuntime runtime = Python.CreateRuntime(setup.Options);

engine = runtime.GetEngine("py");



Is this is the same like –X:Debug?

You reckon this could be the cause?



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland
*Sent:* Tuesday, October 05, 2010 5:53 PM
*To:* Discussion of IronPython
*Subject:* Re: [IronPython] HostCodeHeap leakage?



My guess is that’s code in the JIT heap that’s building up but I’m not 100%
certain.  How is your code being executed?  Do you have the debug option (-D
or –X:Debug) enabled?  To support debug mode we need to produce
uncollectible code which could be building up.



*From:* users-bounces at lists.ironpython.com [mailto:
users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg
*Sent:* Tuesday, October 05, 2010 2:26 AM
*To:* Discussion of IronPython
*Subject:* [IronPython] HostCodeHeap leakage?



I am trying to find a memory/"performance" leak in an Ipy application.

Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is
getting bigger (150MB increase per day). From the !eeheap output it seems
that the increase is due to HostCodeHeap (objects?).

As I understand these objects might be created by Ipy infra, is that right?

Is there anyway I can get more info on their content, or prevent them from
growing?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20101014/d1e1f260/attachment.html>