[IronPython] Please vote up this DLR memory leak bug
Dino Viehland
dinov at microsoft.com
Mon Mar 2 23:13:36 CET 2009
Sorry for the slow response there's nothing that jumped out me immediately. And the older versions make me inclined towards not investigating this too deeply. Instead I talked to Jimmy about getting a new release of the ASP.Net support for IronPython out so hopefully we can do that soon. That will be based off of IronPython 2.0.x which has the re-named namespaces for 3.5 compatability.
If you want to drill into this more I'd suggest looking at all the strings that exist:
!DumpHeap -mt 790fd8c4
(note this number may change between runs) and:
Dump all of the object arrays:
!DumpHeap -mt 7912d8f8
And then doing a:
!da <array addr>
On some of the individual arrays to see what they are holding onto. Depending on what's in those arrays it may be an IronPython, DLR, or someone else's bugs.
The presence of so many strings leads me to believe this could be a case of interned strings being a problem like in one of the repro cases, for example doing:
for i in xrange(10000000):
exec 'def f' + str(i) + '(): pass'
would be an example of this.
You can also do !GCRoot on individual objects such as the strings. If the strings are being kept alive by a Dictionary<string, int> or Dictionary<int, string> then it's our interning which is doing this.
Let me know what you get from here and maybe there'll be something obvious or maybe I can have some other suggestions.
> -----Original Message-----
> From: users-bounces at lists.ironpython.com [mailto:users-
> bounces at lists.ironpython.com] On Behalf Of Dody Gunawinata
> Sent: Thursday, February 26, 2009 12:07 AM
> To: Dino Viehland
> Cc: Discussion of IronPython
> Subject: Re: [IronPython] Please vote up this DLR memory leak bug
>
> Microsoft.Scripting2.Core is a namespace copy of the IP source code -
> based on IP 2.0 Beta 3 - just before one version that introduced
> incompatibility with .Net 3.5
>
> We are using ASP.Net with IronPython support based on July CTP 2007
> (which I think ran on IP 2.0 Alpha X). The latest drop of IP for
> ASP.Net was based on IP 2.0 Beta 4 which contained the .Net 3.5
> incompatibility which makes in unusable for our application.
>
> So my application has 2 copies of IP - one that runs the asp.net
> dynamic support and the other that runs the scripting support (with 2
> suffix in the namespace). It's a stupid situation but we rely a IP for
> ASP.Net and that piece of technology has not been brought up to date
> to a working state for .Net 3.5 framework- and our web productions
> people rely on the software so it's full speed ahead regardless of the
> technology limitation.
>
> This memory leak might be gone if we have an up to date IP for ASP.Net
> but one never know so here's the last 25 lines of the heap dump. I
> hope this is useful.
>
>
>
> 79104368 3939 94536 System.Collections.ArrayList
> 7911f030 1708 95648 System.Reflection.Emit.DynamicMethod
> 7911f400 2999 95968 System.Reflection.Emit.SignatureHelper
> 109cbc9c 2516 100640
> System.Linq.Expressions2.ConditionalExpression
> 1085c930 3181 101792 System.Scripting.SourceSpan
> 109c4594 3244 103808
> System.Linq.Expressions2.ParameterExpression
> 10f63bcc 1237 110188
> System.Collections.Generic.Dictionary`2+Entry[[System.Linq.Expressions2
> .Expression,
> Microsoft.Scripting2.Core],[System.Linq.Expressions2.CompilerScope+Stor
> age,
> Microsoft.Scripting2.Core]][]
> 79135014 3 114968
> System.Collections.Generic.Dictionary`2+Entry[[System.Int32,
> mscorlib],[System.String, mscorlib]][]
> 10f60a2c 2260 117520
> System.Collections.Generic.Dictionary`2[[System.Linq.Expressions2.Expre
> ssion,
> Microsoft.Scripting2.Core],[System.Object, mscorlib]]
> 109cfb1c 2736 120384
> System.Linq.Expressions2.BinaryExpression
> 10f6145c 1743 125700
> System.Collections.Generic.Dictionary`2+Entry[[System.Linq.Expressions2
> .Expression,
> Microsoft.Scripting2.Core],[System.Object, mscorlib]][]
> 109c4e4c 3164 126560 System.Linq.Expressions2.ScopeExpression
> 109c4274 4142 132544
> System.Linq.Expressions2.VariableExpression
> 109caf7c 3735 134460 System.Linq.Expressions2.UnaryExpression
> 10f5cd24 3762 135432
> System.Linq.Expressions2.StackSpiller+ChildRewriter
> 106fa09c 40 138784
> System.Collections.Generic.Dictionary`2+Entry[[Microsoft.Scripting.Symb
> olId,
> Microsoft.Scripting],[Microsoft.Scripting.DynamicMixin+SlotInfo,
> Microsoft.Scripting]][]
> 79111038 2585 144760 System.Reflection.RuntimePropertyInfo
> 79110680 3086 148128 System.Signature
> 106f63bc 4183 150588 Microsoft.Scripting.BuiltinFunction
> 109cbb3c 4327 155772
> System.Linq.Expressions2.MemberExpression
> 791336d8 37 162960
> System.Collections.Generic.Dictionary`2+Entry[[System.String,
> mscorlib],[System.Int32, mscorlib]][]
> 109c320c 4726 170136
> System.Linq.Expressions2.AssignmentExpression
> 7912f348 7171 172104
> System.Collections.Generic.List`1[[System.Reflection.MethodInfo,
> mscorlib]]
> 7911f538 1708 177632
> System.Reflection.Emit.DynamicILGenerator
> 7910ea30 15685 188220 System.RuntimeMethodHandle
> 109c76cc 6304 201728 System.Linq.Expressions2.Block
> 7912d9bc 889 204576 System.Collections.Hashtable+bucket[]
> 790ff734 10742 214840 System.RuntimeType
> 79130510 11613 223528 System.RuntimeTypeHandle[]
> 791208b4 16407 262512 System.Reflection.Emit.GenericFieldInfo
> 10f552cc 18594 297504
> System.Collections.ObjectModel.ReadOnlyCollection`1[[System.Linq.Expres
> sions2.Expression,
> Microsoft.Scripting2.Core]]
> 109c4f1c 9671 386840
> System.Linq.Expressions2.MethodCallExpression
> 7912dd40 646 432936 System.Char[]
> 109c7634 16161 517152
> System.Linq.Expressions2.ConstantExpression
> 79109778 12835 718760 System.Reflection.RuntimeMethodInfo
> 79135538 1494 1167624 System.Reflection.Emit.__FixupData[]
> 7910dd38 124487 1493844 System.RuntimeTypeHandle
> 7912d7c0 17031 1846316 System.Int32[]
> 7912dae8 11218 1877012 System.Byte[]
> 790fd8c4 46924 2987964 System.String
> 7912d8f8 134460 5074708 System.Object[]
>
>
> On Wed, Feb 25, 2009 at 10:51 PM, Dino Viehland <dinov at microsoft.com>
> wrote:
> > Your approach should be perfectly fine (it's basically what
> CompiledCode was designed for) and it's not clear to me what would be
> leaking in this scenario.
> >
> > One suggestion I'd have for you is to download and install windbg
> (http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx).
> >
> > Then attach to your process which is using all the memory and type
> in:
> >
> > .loadby sos mscorwks
> > !DumpHeap -stat
> >
> > And send back the about the last 25 lines of that output. That'll
> give us the most common objects which are using memory in the GC heap
> and we might be able to track it down from there. If that's not
> sufficient maybe we can iterate and figure out what's going on.
> >
> >> -----Original Message-----
> >> From: users-bounces at lists.ironpython.com [mailto:users-
> >> bounces at lists.ironpython.com] On Behalf Of Dody Gunawinata
> >> Sent: Wednesday, February 25, 2009 8:38 AM
> >> To: Dino Viehland
> >> Cc: Discussion of IronPython
> >> Subject: Re: [IronPython] Please vote up this DLR memory leak bug
> >>
> >> What is the impact of caching CompiledCode into this
> "collectability"
> >> issue?
> >>
> >> I am working on a CMS that expose functions implemented in
> IronPython
> >> to a template engine. So I read a directory of 40 or 50 xx.py files,
> >> load it up and compiled them. This is off course a costly operation
> to
> >> do it in every single HTTP request (and SLOW) so I compiled the
> source
> >> code to CompiledCode and cache them.
> >>
> >> Then for every HTTP request, I retrieve these CompiledCode's from
> the
> >> Cache and call Execute(scope) to make the frozen code to be a real
> >> function. So with X HTTP request, there will be X scope and (X *
> 40)
> >> functions being generated at runtime.
> >>
> >> Now our memory consumption is huge and is getting bigger until IIS
> >> gave up and recycle the process.
> >>
> >> Dody G.
> >>
> >>
> >>
> >>
> >> > Just some comments on this as there's a few issues intertwined
> here:
> >> >
> >> > 1. Optimized code - this is currently the default and part
> of
> >> the
> >> > original problem. If you execute code in "optimized" mode we
> >> generate an
> >> > uncollectible type. In 2.6 I'm switching the default to be non-
> >> optimized
> >> > and I also am trying out a change that should reduce the amount of
> >> leakage
> >> > when we do this. That will make the out of the box experience
> much
> >> better.
> >> >
> >>
> >>
> >> > 2. Multiple script runtimes - there's a bug in the DLR when
> you
> >> > repeatedly create script runtimes that the rules aren't getting
> >> collected.
> >> > This is supposed to be fixed in 2.6 but I'm still seeing some
> leaks
> >> in this
> >> > scenario that I'll look into and fix. Back-porting this to 2.0
> would
> >> be
> >> > quite difficult as a lot of things have changed but we could
> >> certainly look
> >> > at backporting a different fix to 2.0.
> >> >
> >>
> >> > 3. There's a repro that defines "TestScriptOldClass" + n
> which
> >> leaks.
> >> > All class / function / member names in code get interned and this
> is
> >> where
> >> > this leak is coming from. If you remove the + n then everything
> >> works. We
> >> > may get rid of this interning at some point in the future but it's
> a
> >> fairly
> >> > significant change. I'm inclined to believe this one isn't as
> much
> >> of an
> >> > issue as usually you aren't compiling code with new names every
> >> single
> >> > type.
> >> >
> >>
> >>
> >>
> >>
> >> >
> >> >
> >> > From: users-bounces at lists.ironpython.com
> >> > [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dody
> >> Gunawinata
> >> > Sent: Sunday, February 22, 2009 2:10 PM
> >> > To: Discussion of IronPython
> >> > Subject: [IronPython] Please vote up this DLR memory leak bug
> >> >
> >> >
> >> >
> >> > There's only 5 votes so far.
> >> >
> >> >
> >> >
> >> >
> >>
> http://www.codeplex.com/IronPython/WorkItem/View.aspx?WorkItemId=20399
> >> >
> >> >
> >> >
> >> > --
> >> > nomadlife.org
> >>
> >>
> >>
> >> --
> >> nomadlife.org
> >> _______________________________________________
> >> Users mailing list
> >> Users at lists.ironpython.com
> >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
> >
>
>
>
> --
> nomadlife.org
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
More information about the Ironpython-users
mailing list