[Ironpython-users] memory fragmentation oder leak in ironpython program / I apologize

[Markus Schaber]

Hi, Peter,

There is at least one known memory leak problem with IronPython. However, it only affects the case when you run lots of python scripts within the same process, not when a single python script is running (as an application):

http://ironpython.codeplex.com/workitem/31764

Recently, there was some progress in debugging of the problem, but it is not solved yet.

Usually, it is not good practice to call GC.Collect() explicitly, except in cases when you certainly know that there is a huge amount of memory which is to be released now.

There are certain knobs you can use to tweak the garbage collection algorithms. Especially the ServerGC setting brought a 200% speedup on a compute and memory allocation intense c# application (geodata processing) on a 64 Bit multi-CPU machine, on the cost of about 200MB more memory usage (of about 2.5G total).

Google will give you many helpful links like
http://msdn.microsoft.com/de-de/library/0xy59wtx%28v=vs.110%29.aspx
http://msdn.microsoft.com/en-us/library/ms229357%28v=vs.110%29.aspx
http://www.atalasoft.com/cs/blogs/rickm/archive/2008/08/20/changing-your-garbage-collector-settings-on-the-fly-net-memory-management-part-5.aspx
http://msdn.microsoft.com/en-us/library/x2tyfybc.aspx


Best regards

Markus Schaber

CODESYS(r) a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions
________________________________
3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber at codesys.com<mailto:m.schaber at codesys.com> | Web: codesys.com<http://www.codesys.com> | CODESYS store: store.codesys.com<http://store.codesys.com>
CODESYS forum: forum.codesys.com<http://forum.codesys.com>

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
Von: Ironpython-users [mailto:ironpython-users-bounces+m.schaber=codesys.com at python.org] Im Auftrag von Peter Schwalm
Gesendet: Dienstag, 10. Dezember 2013 05:29
An: Slide
Cc: ironpython mailList
Betreff: Re: [Ironpython-users] memory fragmentation oder leak in ironpython program / I apologize

Hallo Slide,

thank you for your answer. I don't knwo if iTextSharp uses IDisposable. I admit, I use it as a block box, but at least I suppose it's well written, because I think it is a proven tool.

... and I apologize: I have found the mistake in my own code now. It was a list which - unnecessarily - grew with every document and which was worked upon rather costly. So I caused the problem myself. The program has a constant throughput, and for now I'm happy.

But I still have no safe way to analyze such problems, and there are still questions which I don't have an answer for:

Is ipy kosher in it's memory handling?
Where does this ever growing number of GCHandles (shown by sysinternals procexp) come from? Can I simply ignore it?
Is it good practice to use explicit GC.Collect()-calls?

If anyone can share her/his experience in these fields, I would be happy.

Thanks in advance
Peter Schwalm





Am 10.12.2013 02:19, schrieb Slide:

Does itextsharp implement idisposable for any objects you are using?
On Dec 9, 2013 5:19 PM, "Peter Schwalm" <ps at peter-schwalm.de<mailto:ps at peter-schwalm.de>> wrote:
Hello,

I have an ipy script that manipulates several thousand .pdf documents in a single run via itextsharp. I observe that it is permanently getting slower. After starting the program it processes about 10 documents per second, after processing 5000 documents it does only process 3 oder 4 docs per second.

I have already in .net GC garbage collection calls which made the situation something better (before that it was still worse).

Does anyone know if this could a fragmention problem? And what I could against it?

Another peculiarity:

in sysinternals procexp I can see that the memory usage rises during the excecution, but - after I inserted the GC.Collect() calls - relative moderately. What rises constantly are the GC handles (about 170.000 after 8000 documents). If I read the docs it looks as if GCHandles are handles that make managed memory accessible for non-managed programs.

Does anyone know where these GC handles come from, if they could be the cause of my problems and what I could do against them?

Thanks in advance,
Peter Schwalm


_______________________________________________
Ironpython-users mailing list
Ironpython-users at python.org<mailto:Ironpython-users at python.org>
https://mail.python.org/mailman/listinfo/ironpython-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20131210/af428ebd/attachment.html>