From Thomas.Lundgaard at 3shape.com Fri Sep 6 12:01:00 2019 From: Thomas.Lundgaard at 3shape.com (Thomas Lundgaard Hansen) Date: Fri, 6 Sep 2019 16:01:00 +0000 Subject: [Python.NET] Memory management when embedding python. Message-ID: <15e391e5536344e3a006db8c5075a151@3shape.com> Hi. I am embedding Python into a .NET (C#) application. For the purpose of this discussion my code is like the example given here: https://github.com/pythonnet/pythonnet#example (except that I am working with arrays of a substantial size). Each of the "dynamic" variables created in that example are of the .NET type PyObject, which is a disposable type. The code in the example does not call Dispose() on these objects and it would indeed be very inconvenient to do so. Whats the best practice here and what considerations went into this design? Can we expect that the only unmanged resources held by PyObject's are unmanaged memory? (Unless, of course, the PyObject represents a system resource like a file or network stream). Has it been considered to use GC.AddMemoryPressure when PyObjects are backed by a large amount of unmanaged memory? Thanks, Thomas Lundgaard Hansen -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis.akhiyarov at gmail.com Sun Sep 8 11:43:10 2019 From: denis.akhiyarov at gmail.com (Denis Akhiyarov) Date: Sun, 8 Sep 2019 10:43:10 -0500 Subject: [Python.NET] Memory management when embedding python. In-Reply-To: <15e391e5536344e3a006db8c5075a151@3shape.com> References: <15e391e5536344e3a006db8c5075a151@3shape.com> Message-ID: There is a lot more going on this topic of GC in GitHub repo, but I'm not involved with the project anymore. Anyway if anyone is interested in moderating this mailing list, please let me know. Thanks, Denis On Sun, Sep 8, 2019, 10:35 AM Thomas Lundgaard Hansen < Thomas.Lundgaard at 3shape.com> wrote: > Hi. > > > > I am embedding Python into a .NET (C#) application. For the purpose of > this discussion my code is > > like the example given here: > > https://github.com/pythonnet/pythonnet#example > > (except that I am working with arrays of a substantial size). > > > > Each of the ?dynamic? variables created in that example are of the .NET > type PyObject, which is > > a disposable type. The code in the example does not call Dispose() on > these objects and it would > > indeed be very inconvenient to do so. > > > > Whats the best practice here and what considerations went into this > design? Can we expect that > > the only unmanged resources held by PyObject?s are unmanaged memory? > (Unless, of course, the > > PyObject represents a system resource like a file or network stream). > > > > Has it been considered to use GC.AddMemoryPressure when PyObjects are > backed by a large > > amount of unmanaged memory? > > > > > > Thanks, > > *Thomas Lundgaard Hansen* > > > _________________________________________________ > Python.NET mailing list - PythonDotNet at python.org > https://mail.python.org/mailman/listinfo/pythondotnet > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amos_402 at msn.com Sun Sep 8 14:12:01 2019 From: amos_402 at msn.com (Li Amos) Date: Sun, 8 Sep 2019 18:12:01 +0000 Subject: [Python.NET] Memory management when embedding python. In-Reply-To: References: <15e391e5536344e3a006db8c5075a151@3shape.com>, Message-ID: If you want to dispose the PyObject and reduce the reference count immediately you should call `Dispose` manually or use `using` statement. Otherwise, GC will take care of it although it would be a little delay, you can adjust ` Finalizer.Instance.Threshold ` for reducing the delay, it refers how many PyObjects will be collect at once after GC mark them as garbage. About `GC.AddMemoryPressure`, there?s no relate with it, PyObject only contains pointers of CPython, it doesn?t allocate any unmanaged memory, thus GC know the objects? size. From: Denis Akhiyarov Sent: Sunday, September 8, 2019 23:43 To: A list for users and developers of Python for .NET Subject: Re: [Python.NET] Memory management when embedding python. There is a lot more going on this topic of GC in GitHub repo, but I'm not involved with the project anymore. Anyway if anyone is interested in moderating this mailing list, please let me know. Thanks, Denis On Sun, Sep 8, 2019, 10:35 AM Thomas Lundgaard Hansen > wrote: Hi. I am embedding Python into a .NET (C#) application. For the purpose of this discussion my code is like the example given here: https://github.com/pythonnet/pythonnet#example (except that I am working with arrays of a substantial size). Each of the ?dynamic? variables created in that example are of the .NET type PyObject, which is a disposable type. The code in the example does not call Dispose() on these objects and it would indeed be very inconvenient to do so. Whats the best practice here and what considerations went into this design? Can we expect that the only unmanged resources held by PyObject?s are unmanaged memory? (Unless, of course, the PyObject represents a system resource like a file or network stream). Has it been considered to use GC.AddMemoryPressure when PyObjects are backed by a large amount of unmanaged memory? Thanks, Thomas Lundgaard Hansen _________________________________________________ Python.NET mailing list - PythonDotNet at python.org https://mail.python.org/mailman/listinfo/pythondotnet -------------- next part -------------- An HTML attachment was scrubbed... URL: From Thomas.Lundgaard at 3shape.com Wed Sep 11 03:58:34 2019 From: Thomas.Lundgaard at 3shape.com (Thomas Lundgaard Hansen) Date: Wed, 11 Sep 2019 07:58:34 +0000 Subject: [Python.NET] Memory management when embedding python. In-Reply-To: References: <15e391e5536344e3a006db8c5075a151@3shape.com>, Message-ID: Thank you. I am fine with a little delay before the objects are collected, I just wanted to make sure there are no other side effects by not calling Dispose() on the objects. On a side node, regarding GC.AddMemoryPressure: PyObjects have some (possibly large) amount of backing unmanaged memory. The .NET garbage collector does not know about this memory and thus cannot take it into account when deciding when the GC should be triggered. AddMemoryPressure allows you to tell the .NET GC about the unmanaged backing memory. However, PythonNET may not know the amount of memory that has been allocated when constructing python objects. Best regards, Thomas Lundgaard Hansen From: PythonDotNet On Behalf Of Li Amos Sent: Sunday, September 8, 2019 8:12 PM To: A list for users and developers of Python for .NET Subject: Re: [Python.NET] Memory management when embedding python. If you want to dispose the PyObject and reduce the reference count immediately you should call `Dispose` manually or use `using` statement. Otherwise, GC will take care of it although it would be a little delay, you can adjust ` Finalizer.Instance.Threshold ` for reducing the delay, it refers how many PyObjects will be collect at once after GC mark them as garbage. About `GC.AddMemoryPressure`, there's no relate with it, PyObject only contains pointers of CPython, it doesn't allocate any unmanaged memory, thus GC know the objects' size. From: Denis Akhiyarov Sent: Sunday, September 8, 2019 23:43 To: A list for users and developers of Python for .NET Subject: Re: [Python.NET] Memory management when embedding python. There is a lot more going on this topic of GC in GitHub repo, but I'm not involved with the project anymore. Anyway if anyone is interested in moderating this mailing list, please let me know. Thanks, Denis On Sun, Sep 8, 2019, 10:35 AM Thomas Lundgaard Hansen > wrote: Hi. I am embedding Python into a .NET (C#) application. For the purpose of this discussion my code is like the example given here: https://github.com/pythonnet/pythonnet#example (except that I am working with arrays of a substantial size). Each of the "dynamic" variables created in that example are of the .NET type PyObject, which is a disposable type. The code in the example does not call Dispose() on these objects and it would indeed be very inconvenient to do so. Whats the best practice here and what considerations went into this design? Can we expect that the only unmanged resources held by PyObject's are unmanaged memory? (Unless, of course, the PyObject represents a system resource like a file or network stream). Has it been considered to use GC.AddMemoryPressure when PyObjects are backed by a large amount of unmanaged memory? Thanks, Thomas Lundgaard Hansen _________________________________________________ Python.NET mailing list - PythonDotNet at python.org https://mail.python.org/mailman/listinfo/pythondotnet -------------- next part -------------- An HTML attachment was scrubbed... URL: From markv at unity3d.com Thu Sep 12 15:44:01 2019 From: markv at unity3d.com (Mark Visser) Date: Thu, 12 Sep 2019 15:44:01 -0400 Subject: [Python.NET] Working group meeting Tuesday September 10th Message-ID: <00A80EA2-7B6B-4E9C-AD77-85C2EA3CC871@unity3d.com> Hi everyone, Here are the minutes from Tuesday's meeting. Present: Victor Milovanov, Amos Li, Mark Visser Mark mentioned that his team has found a solution to C# domain reload that doesn't crash the Unity Editor. Benoit Hudson will join the next meeting to discuss with the group. Victor has been working on the ability to inherit C# classes from Python classes and override Python methods in C# using attributes. The next meeting will held on Tuesday, September 24th at 3pm EST. If you have not received an invite, please e-mail me directly and I'll add you to the event. Also please note that I went ahead and switched to Zoom for hosting meetings as Google Hangouts is blocked in the PRC. I'll ask the list for agenda items the week before the meeting. Hope to see you all online next time. cheers, -Mark Mark Visser DCC Team Lead Unity Technologies - www.unity3d.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From markv at unity3d.com Tue Sep 17 11:59:41 2019 From: markv at unity3d.com (Mark Visser) Date: Tue, 17 Sep 2019 11:59:41 -0400 Subject: [Python.NET] Call for discussion items for next week's working group meeting (September 24th) Message-ID: <35041A49-BEFB-413F-A5B8-8D7FEA8ED208@unity3d.com> Hi all, The next Python for .NET working group call will be held on Tuesday, September 24th, at 3pm EST / 12pm PST. Please reply to this e-mail with any discussion items you would like to add to the agenda, or if you would like to be added to the call. Mine: 1. I suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. 2. Benedikt (filmor at gmail.com ) can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation cheers, -Mark Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From markv at unity3d.com Tue Sep 17 12:31:07 2019 From: markv at unity3d.com (Mark Visser) Date: Tue, 17 Sep 2019 12:31:07 -0400 Subject: [Python.NET] Call for discussion items for next week's working group meeting (September 24th) In-Reply-To: <35041A49-BEFB-413F-A5B8-8D7FEA8ED208@unity3d.com> References: <35041A49-BEFB-413F-A5B8-8D7FEA8ED208@unity3d.com> Message-ID: <4D9C8BB1-E4D0-4E0E-BE9F-1C0212802349@unity3d.com> Another discussion item from us (Unity): https://github.com/pythonnet/pythonnet/issues/957 Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com > On Sep 17, 2019, at 11:59 AM, Mark Visser wrote: > > Hi all, > > The next Python for .NET working group call will be held on Tuesday, September 24th, at 3pm EST / 12pm PST. > > Please reply to this e-mail with any discussion items you would like to add to the agenda, or if you would like to be added to the call. > > Mine: > 1. I suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. > 2. Benedikt (filmor at gmail.com ) can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation > > cheers, > -Mark > > Mark Visser > Tooling Dev Manager > Unity Technologies - www.unity3d.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amos_402 at msn.com Tue Sep 17 12:54:47 2019 From: amos_402 at msn.com (Li Amos) Date: Tue, 17 Sep 2019 16:54:47 +0000 Subject: [Python.NET] Call for discussion items for next week's working group meeting (September 24th) In-Reply-To: <4D9C8BB1-E4D0-4E0E-BE9F-1C0212802349@unity3d.com> References: <35041A49-BEFB-413F-A5B8-8D7FEA8ED208@unity3d.com>, <4D9C8BB1-E4D0-4E0E-BE9F-1C0212802349@unity3d.com> Message-ID: ?About how to reset the Python, that?s what I exactly do it at last weekends. https://github.com/pythonnet/pythonnet/pull/958 ________________________________ From: PythonDotNet on behalf of Mark Visser Sent: Wednesday, September 18, 2019 12:31:07 AM To: PythonDotNet at python.org Subject: Re: [Python.NET] Call for discussion items for next week's working group meeting (September 24th) Another discussion item from us (Unity): https://github.com/pythonnet/pythonnet/issues/957 Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com On Sep 17, 2019, at 11:59 AM, Mark Visser > wrote: Hi all, The next Python for .NET working group call will be held on Tuesday, September 24th, at 3pm EST / 12pm PST. Please reply to this e-mail with any discussion items you would like to add to the agenda, or if you would like to be added to the call. Mine: 1. I suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. 2. Benedikt (filmor at gmail.com) can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation cheers, -Mark Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From amos_402 at msn.com Tue Sep 24 15:10:21 2019 From: amos_402 at msn.com (Li Amos) Date: Tue, 24 Sep 2019 19:10:21 +0000 Subject: [Python.NET] Call for discussion items for next week's working group meeting (September 24th) In-Reply-To: References: <35041A49-BEFB-413F-A5B8-8D7FEA8ED208@unity3d.com>, <4D9C8BB1-E4D0-4E0E-BE9F-1C0212802349@unity3d.com>, Message-ID: Still having the meeting today? I can?t find the entry. ________________________________ From: PythonDotNet on behalf of Li Amos Sent: Wednesday, September 18, 2019 12:54:47 AM To: A list for users and developers of Python for .NET Subject: Re: [Python.NET] Call for discussion items for next week's working group meeting (September 24th) ?About how to reset the Python, that?s what I exactly do it at last weekends. https://github.com/pythonnet/pythonnet/pull/958 ________________________________ From: PythonDotNet on behalf of Mark Visser Sent: Wednesday, September 18, 2019 12:31:07 AM To: PythonDotNet at python.org Subject: Re: [Python.NET] Call for discussion items for next week's working group meeting (September 24th) Another discussion item from us (Unity): https://github.com/pythonnet/pythonnet/issues/957 Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com On Sep 17, 2019, at 11:59 AM, Mark Visser > wrote: Hi all, The next Python for .NET working group call will be held on Tuesday, September 24th, at 3pm EST / 12pm PST. Please reply to this e-mail with any discussion items you would like to add to the agenda, or if you would like to be added to the call. Mine: 1. I suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. 2. Benedikt (filmor at gmail.com) can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation cheers, -Mark Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From lostfreeman at gmail.com Tue Sep 24 15:12:01 2019 From: lostfreeman at gmail.com (=?utf-8?Q?Victor_=E2=80=9CLOST=E2=80=9D_Milovanov?=) Date: Tue, 24 Sep 2019 12:12:01 -0700 Subject: [Python.NET] Call for discussion items for next week's workinggroup meeting (September 24th) In-Reply-To: References: <35041A49-BEFB-413F-A5B8-8D7FEA8ED208@unity3d.com>, <4D9C8BB1-E4D0-4E0E-BE9F-1C0212802349@unity3d.com>, Message-ID: <5d8a6a80.1c69fb81.46d3e.6649@mx.google.com> https://unity3d.zoom.us/j/346931511 Sent from Mail for Windows 10 From: Li Amos Sent: Tuesday, September 24, 2019 12:10 PM To: A list for users and developers of Python for .NET Subject: Re: [Python.NET] Call for discussion items for next week's workinggroup meeting (September 24th) Still having the meeting today? I can?t find the entry. ? From: PythonDotNet on behalf of Li Amos Sent: Wednesday, September 18, 2019 12:54:47 AM To: A list for users and developers of Python for .NET Subject: Re: [Python.NET] Call for discussion items for next week's working group meeting (September 24th) ? ?About how to reset the Python, that?s what I exactly do it at last weekends. https://github.com/pythonnet/pythonnet/pull/958 ? From: PythonDotNet on behalf of Mark Visser Sent: Wednesday, September 18, 2019 12:31:07 AM To: PythonDotNet at python.org Subject: Re: [Python.NET] Call for discussion items for next week's working group meeting (September 24th) ? Another discussion item from us (Unity):?https://github.com/pythonnet/pythonnet/issues/957 Mark Visser Tooling Dev Manager Unity Technologies?-?www.unity3d.com On Sep 17, 2019, at 11:59 AM, Mark Visser wrote: Hi all,? The next Python for .NET working group call will be held on Tuesday, September 24th, at 3pm EST / 12pm PST.? Please reply to this e-mail with any discussion items you would like to add to the agenda, or if you would like to be added to the call. Mine: 1. I suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. 2. Benedikt (filmor at gmail.com) can we get a status update on joining the .NET Foundation? Github issue:?Joining the .NET Foundation cheers, -Mark Mark Visser Tooling Dev Manager Unity Technologies?-?www.unity3d.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 7FEC19DE415D44FD83DBB345E030D2F2.png Type: image/png Size: 159 bytes Desc: not available URL: From markv at unity3d.com Tue Sep 24 16:04:18 2019 From: markv at unity3d.com (Mark Visser) Date: Tue, 24 Sep 2019 16:04:18 -0400 Subject: [Python.NET] Python for .NET working group meeting notes from Tuesday, September 24, 2019 Message-ID: <25E41A58-EB3A-4A83-842B-7F7603B49349@unity3d.com> Attendees: Mark Visser (Unity), Benoit Hudson & Felix Bourbonnais (ImgSpc), Victor Milovanov, Mohamed Koubaa (ANSYS Inc), Amos Li Agenda 1. (Mark) Suggest we use a "needs discussion" label on issues and PRs for next time. Then we can just go through them on the call. 2. (Mark) Can we get a status update on joining the .NET Foundation? Github issue: Joining the .NET Foundation 3. (Benoit) Python can't reliably re-initialize; new proposal for Unity's domain reload 4. (Benoit) What's the policy with the GIL? Main thread appears to be grabbing the lock and not releasing it. 5. (Mohamed) Can I get clarification on IWrapper vs. reflection for PR #808? Notes Welcome to new contributor Mohamed Koubaa (ANSYS), who is investigating moving his company's platform from IronPython to CPython. 1 & 2) need input from Benedikt, Mark will follow up with Benedikt. 4) Benoit: GIL - Should we expect the GIL to be held after initialization? Victor: I wouldn't expect it to be held. [PyGIL] attribute is informational. 3) Benoit: Python can't reliably re-initialize. last fall's work - force PyFinalize to be called unfortunately doesn't work for native extensions - can't reliably reinitialize proposal instead of calling PyFinalize, delete all links from Python to C# Felix prototyped a solution that tracks all links during proxy generation and cuts all links during domain shutdown to prevent dereference of dead pointers result is just AttributeError instead of crash Amos proposed a different solution which seems to be targeting memory leaks Victor: what about memory leaks? C# side deletes all of its memory If C# holds a reference to Python, it will still exist when C# shuts down - no problem Amos: has a branch that solves one of the problems Benoit & Felix will investigate his branch and see where it intersects with their solution the reason we're using pythonnet is to enable PySide/PyQt and numpy among others 5) Mohamed: Working on a small change to implement the wrapper function for console output via __repr__ https://github.com/pythonnet/pythonnet/pull/808 inheritance over reflection, IWrapper vs. method with a signature Pythonnet can reflect what are the pros and cons? should we go with a convention? Victor: you can take the reflection approach, but need to cache the delegates more of an implementation detail, not actually necessary (note: any errors in the above are mine, feel free to correct any gross misrepresentations I've inadvertently made!) Action Items Mark follow up with Benedikt about Python foundation and using issues to track discussion items Benoit & Felix investigate why GIL is held indefinitely test out Amos's branch and connect over gitter to discuss what parts of the problem each approach is solving Thank you all for attending! The next call will be held at 3pm EST on Tuesday, October 8th, 2019, see you then. Please e-mail me if you're not on the attendee list and I will add you. Mark Visser Tooling Dev Manager Unity Technologies - www.unity3d.com -------------- next part -------------- An HTML attachment was scrubbed... URL: