From bruce.bromberek at gmail.com Fri Oct 1 13:34:25 2010 From: bruce.bromberek at gmail.com (Bruce Bromberek) Date: Fri, 1 Oct 2010 06:34:25 -0500 Subject: [IronPython] Visual Studio Tools Startup Error In-Reply-To: References: Message-ID: Thanks Dino! This was the culprit. Deleted the cruft, reinstall 2.7 and everything works now. > Such as IronStudio or IronPython Tools for Visual Studio? ?If so can you delete > those directories and try re-installing? ?It also might be good to check > %LOCALAPPDATA%\Microsoft\VisualStudio\10.0\Extensions\Microsoft but > there really shouldn't be anything there. Bruce From merllab at microsoft.com Fri Oct 1 17:59:34 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Fri, 1 Oct 2010 08:59:34 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: <446ab2f1-697c-4dde-aaaf-1f4697a439e5@tk5-exsmh-c101.redmond.corp.microsoft.com> This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/77717. MODIFIED SOURCES $/IronPython/IronPython_Main/External.LCA_RESTRICTED/Languages/IronRuby/yaml/IronRuby.Libraries.Yaml/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Hosts/MerlinWeb/Microsoft.Scripting.AspNet/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Hosts/Silverlight/Chiron/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/IronPython/AssemblyVersion.cs $/IronPython/IronPython_Main/Languages/IronPython/Samples/BadPaint/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/IronPython/Samples/DynamicWebServiceHelpers/sources/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/IronPython/Samples/tests/Direct3D/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/IronPython/Samples/tests/FMsynth/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/IronPython/Samples/tests/Puzzle/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/IronPython/Samples/tests/UIRunner/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/Ruby/ClassInitGenerator/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/Ruby/Console/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/Ruby/IronRuby.Tests/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/Ruby/Samples/Tutorial/app/design/Tutorial/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/Ruby/Samples/Tutorial/app/design/TutorialSL/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Languages/Ruby/Tests/Tools/ParseOnly/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Microsoft.Dynamic/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Microsoft.Scripting.Core/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Microsoft.Scripting.Metadata/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Microsoft.Scripting/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Samples/ExpressionTree/ETSample1_CS/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Samples/ExpressionTree/UESamples/ConsoleApplication1/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Samples/Hosting/ShapeScript/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Samples/LibraryAuthors/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Samples/Silverlight/Hosting/Python/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Samples/Silverlight/Hosting/Ruby/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Samples/sympl/csharp-cponly/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Samples/sympl/csharp-cponly/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Samples/sympl/csharp/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/AstTest/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/ComTest/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/DLRTestHost/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/ETScenarios/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/ETScenariosCSLinq/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/HostingTest/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/HostingTestRunner/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/Metadata/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/NoPia/NoPiaHelper2/NoPiaHelper2/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/NoPia/NoPiaHelperClass/NoPiaHelperClass/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/NoPia/Tests/Tests/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/Perf/PerfV2Nodes/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/PerfHost/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/PerfV1Nodes/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/RowanTest.Common/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/SecurityTests/DynamicTest/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/SecurityTests/SecurityTests/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/SiteTest/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/TestAst/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/TestHost/Microsoft.Silverlight.TestHostCritical/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/TestHost/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Runtime/Tests/TestInternalDLR/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Test/TestRunner/TestRunner/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Tools/IronStudio/AnalysisTest/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Tools/IronStudio/IronPythonTools/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Tools/IronStudio/IronPythonToolsCore/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Tools/IronStudio/IronStudio/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Tools/IronStudio/IronStudioCore/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Tools/IronStudio/RemoteScriptFactory/Properties/AssemblyInfo.cs $/IronPython/IronPython_Main/Tools/IronStudio/UnitTests/Properties/AssemblyInfo.cs From charles.medcoff at rcmt.com Sun Oct 3 14:15:38 2010 From: charles.medcoff at rcmt.com (Medcoff, Charles) Date: Sun, 3 Oct 2010 08:15:38 -0400 Subject: [IronPython] Embedding IronPython in ASP.NET/SharePoint2010/Commerce Server Message-ID: <4EFFA72C340FF54A9A2FA84786378C6DC4F6211593@RCMTMAIL.rcmt.com> Hello, I have a client for which I am developing a SharePoint 2010/Microsoft Commerce Server 2009 applications. The client has a fairly complex set of business rules for their ordering process which may change frequently. Rules engines are pretty expensive so I'm considering the approach of embedding Iron Python into the app as an alternative to making it a bit easier to make rules updates via scripting. My approach is to launch an Iron Python script from within an Operation Sequence Component (http://msdn.microsoft.com/en-us/library/dd464335(CS.90).aspx) which is a standard extensibility point within Commerce Server. In nutshell an Operation Sequence Component is a .NET assembly containing a class which implements a particular interface, and is loaded via reflection by the Commerce Server runtime. The class is then loaded/executed when the relevant Commerce Server API is used. I've successfully written a simple Operation Sequence Component which executes a script. The only catch to getting it to work is to register all of the related Iron Python DLL in the GAC. Here is an example of what I'm doing: namespace IPythonOperationalSequenceComponent { public class IronPythonOperationSequenceComponent : Microsoft.Commerce.Providers.Components.OperationSequenceComponent { public override void ExecuteQuery( Microsoft.Commerce.Contracts.Messages.CommerceQueryOperation queryOperation, Microsoft.Commerce.Broker.OperationCacheDictionary operationCache, Microsoft.Commerce.Contracts.Messages.CommerceQueryOperationResponse response) { try { ScriptEngine pyEngine = Python.CreateEngine(); ScriptScope pyScope = pyEngine.CreateScope(); pyScope.SetVariable("request", queryOperation); pyScope.SetVariable("cache", operationCache); pyScope.SetVariable("response", response); ScriptSource source = pyEngine.CreateScriptSourceFromFile(@"rules.py"); CompiledCode compiled = source.Compile(); compiled.Execute(pyScope); } catch (Exception ex) { // error handling elided } } } } Now this is a "toy" approach as I shouldn't need to create an engine, scope, compile the script, etc. for every post back. I'm thinking a better approach is to have the script engine, scope and compiled script live at the HttpWebApplication level and be shared across all threads, requests, users. My concern is that will this approach work. What are the threading, concurrency, and performance issues involved? Has anyone done anything like this in ASP, let alone SharePoint and can share their experiences with regards to what works, what does not work, best approach, etc. --chuck Best Regards, Chuck Charles Medcoff Principal Consultant | Enterprise Integration Solutions [cid:image001.jpg at 01CB62D1.7C4366B0] Tel: (248) 687-5623 Cell: (248) 884-2854 www.rcmt.com/eis -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 3657 bytes Desc: image001.jpg URL: From jdhardy at gmail.com Sun Oct 3 17:45:44 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Sun, 3 Oct 2010 09:45:44 -0600 Subject: [IronPython] Embedding IronPython in ASP.NET/SharePoint2010/Commerce Server In-Reply-To: <4EFFA72C340FF54A9A2FA84786378C6DC4F6211593@RCMTMAIL.rcmt.com> References: <4EFFA72C340FF54A9A2FA84786378C6DC4F6211593@RCMTMAIL.rcmt.com> Message-ID: Hi Charles, The approach I've used for web applications in the past is to have the runtime and engine be application-wide and create a new scope for each script on every request. Scopes are very cheap to create and will keep the scripts isolated so that they don't affect each other. The engine and runtime are thread-safe, so you can safely get away with only having one per application. I also use a cache of CompiledCode objects for each script to avoid recompiling them each time. - Jeff On Sun, Oct 3, 2010 at 6:15 AM, Medcoff, Charles wrote: > Hello, > > > > I have a client for which I am developing a SharePoint 2010/Microsoft > Commerce Server 2009 applications. The client has a fairly complex set of > business rules for their ordering process which may change frequently. > Rules engines are pretty expensive so I?m considering the approach of > embedding Iron Python into the app as an alternative to making it a bit > easier to make rules updates via scripting. > > > > My approach is to launch an Iron Python script from within an Operation > Sequence Component ( > http://msdn.microsoft.com/en-us/library/dd464335(CS.90).aspx) which is a > standard extensibility point within Commerce Server. In nutshell an > Operation Sequence Component is a .NET assembly containing a class which > implements a particular interface, and is loaded via reflection by the > Commerce Server runtime. The class is then loaded/executed when the > relevant Commerce Server API is used. > > > > I?ve successfully written a simple Operation Sequence Component which > executes a script. The only catch to getting it to work is to register all > of the related Iron Python DLL in the GAC. Here is an example of what I?m > doing: > > > > namespace IPythonOperationalSequenceComponent > { > public class IronPythonOperationSequenceComponent > : Microsoft.Commerce.Providers.Components.OperationSequenceComponent > { > public override void ExecuteQuery( > Microsoft.Commerce.Contracts.Messages.CommerceQueryOperation > queryOperation, > Microsoft.Commerce.Broker.OperationCacheDictionary > operationCache, > Microsoft.Commerce.Contracts.Messages. > CommerceQueryOperationResponse response) > { > try > { > ScriptEngine pyEngine = Python.CreateEngine(); > ScriptScope pyScope = pyEngine.CreateScope(); > pyScope.SetVariable("request", queryOperation); > pyScope.SetVariable("cache", operationCache); > pyScope.SetVariable("response", response); > ScriptSource source = pyEngine.CreateScriptSourceFromFile( > @"rules.py"); > CompiledCode compiled = source.Compile(); > compiled.Execute(pyScope); > } > catch (Exception ex) > { > // error handling elided > } > } > } > } > > > > > > Now this is a ?toy? approach as I shouldn?t need to create an engine, > scope, compile the script, etc. for every post back. I?m thinking a better > approach is to have the script engine, scope and compiled script live at the > HttpWebApplication level and be shared across all threads, requests, users. > My concern is that will this approach work. What are the threading, > concurrency, and performance issues involved? Has anyone done anything like > this in ASP, let alone SharePoint and can share their experiences with > regards to what works, what does not work, best approach, etc. > > > > --chuck > > > > > > Best Regards, > > Chuck > > > > > > Charles Medcoff > > Principal Consultant* *|* *Enterprise Integration Solutions* * > > [image: Description: Description: mailsiglogo] > > Tel: (248) 687-5623 > > Cell: (248) 884-2854 > > www.rcmt.com/eis > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charles.c.strahan at gmail.com Mon Oct 4 06:11:09 2010 From: charles.c.strahan at gmail.com (Charles Strahan) Date: Sun, 3 Oct 2010 23:11:09 -0500 Subject: [IronPython] Question about CriticalFinalizerObject usage. Message-ID: Hello, I'm about to begin work on implementing the FFI gemfor IronRuby, and I was referred to IronPython's CTypes for inspiration. I noticed that the MemoryHolder inherits from CriticalFinalizerObject, and I was curious why that is. I'm actually not familiar with Constrained Execution Regions in general, so a quick high level description would be awesome. I've briefly looked at the following MSDN docs, but they're a little too fine-grained for me to follow: CriticalFinalizerObject Class: http://msdn.microsoft.com/en-us/library/system.runtime.constrainedexecution.criticalfinalizerobject.aspx System.Runtime.ConstrainedExecution Namespace: http://msdn.microsoft.com/en-us/library/system.runtime.constrainedexecution.aspx Constrained Execution Regions: http://msdn.microsoft.com/en-us/library/ms228973.aspx Reliability Best Practices: http://msdn.microsoft.com/en-us/library/ms228970.aspx Any help in understanding the purpose of CriticalFinalizerObject and why it is used as the base class for MemoryHolder would be greatly appreciated. Thanks! -Charles -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Mon Oct 4 07:48:54 2010 From: dinov at microsoft.com (Dino Viehland) Date: Mon, 4 Oct 2010 05:48:54 +0000 Subject: [IronPython] Question about CriticalFinalizerObject usage. In-Reply-To: References: Message-ID: Objects which inherit from CriticalFinalizerObject basically have their finalize method turned into a CER. The finalizer method is also JITed before any instances are created so the finalizer is guaranteed to be runnable. In generally CERs are all about ensuring that the VM won't cause any unexpected failure points in your code. These can be introduced because the VM does something lazily (including JITing methods) and doing the work might fail due to insufficient resources. Or it can also be due to thread abort. Because these objects are responsible for freeing up resources we don't want any unexpected failures to be injected otherwise the resources would leak. So for example MemoryHolder also has a CER in the constructor - this ensures that we don't take a ThreadAbort between the CallocCall, storing the value in _data, or assigning to _ownsData. This will all complete or not complete so that our state is consistent when the finalizer is run. It also makes sure that any work the CLR needs to perform to call NativeFunctions.Calloc is all performed before we enter the CER so that we don't get an out of memory exception while calling or returning from it. For most environments it's not super important that this is gotten right - but if you run in a process which needs long uptime, is resource constrained, and/or uses thread abort a lot (SQL server being an example of all 3) it's important that this is correct. I happened to work on this feature when I was on the CLR team so it came rather naturally to me to get it right :) From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Charles Strahan Sent: Sunday, October 03, 2010 9:11 PM To: users at lists.ironpython.com Subject: [IronPython] Question about CriticalFinalizerObject usage. Hello, I'm about to begin work on implementing the FFI gem for IronRuby, and I was referred to IronPython's CTypes for inspiration. I noticed that the MemoryHolder inherits from CriticalFinalizerObject, and I was curious why that is. I'm actually not familiar with Constrained Execution Regions in general, so a quick high level description would be awesome. I've briefly looked at the following MSDN docs, but they're a little too fine-grained for me to follow: CriticalFinalizerObject Class: http://msdn.microsoft.com/en-us/library/system.runtime.constrainedexecution.criticalfinalizerobject.aspx System.Runtime.ConstrainedExecution Namespace: http://msdn.microsoft.com/en-us/library/system.runtime.constrainedexecution.aspx Constrained Execution Regions: http://msdn.microsoft.com/en-us/library/ms228973.aspx Reliability Best Practices: http://msdn.microsoft.com/en-us/library/ms228970.aspx Any help in understanding the purpose of CriticalFinalizerObject and why it is used as the base class for MemoryHolder would be greatly appreciated. Thanks! -Charles -------------- next part -------------- An HTML attachment was scrubbed... URL: From charles.c.strahan at gmail.com Mon Oct 4 08:51:09 2010 From: charles.c.strahan at gmail.com (Charles Strahan) Date: Mon, 4 Oct 2010 01:51:09 -0500 Subject: [IronPython] Question about CriticalFinalizerObject usage. In-Reply-To: References: Message-ID: Awesome - that makes sense! That's something I'll definitely want to consider for my FFI implementation. As you mentioned, it won't be a problem in most cases, but for those that do wish to use my FFI implementation *and*require reliable resource management, it's something I'll need to account for sooner or later. Thanks Dino, -Charles On Mon, Oct 4, 2010 at 12:48 AM, Dino Viehland wrote: > Objects which inherit from CriticalFinalizerObject basically have their > finalize method turned into a CER. The finalizer method is also JITed > before any instances are created so the finalizer is guaranteed to be > runnable. > > > > In generally CERs are all about ensuring that the VM won?t cause any > unexpected failure points in your code. These can be introduced because the > VM does something lazily (including JITing methods) and doing the work might > fail due to insufficient resources. Or it can also be due to thread abort. > Because these objects are responsible for freeing up resources we don?t want > any unexpected failures to be injected otherwise the resources would leak. > > > > So for example MemoryHolder also has a CER in the constructor ? this > ensures that we don?t take a ThreadAbort between the CallocCall, storing the > value in _data, or assigning to _ownsData. This will all complete or not > complete so that our state is consistent when the finalizer is run. It also > makes sure that any work the CLR needs to perform to call > NativeFunctions.Calloc is all performed before we enter the CER so that we > don?t get an out of memory exception while calling or returning from it. > > > > For most environments it?s not super important that this is gotten right ? > but if you run in a process which needs long uptime, is resource > constrained, and/or uses thread abort a lot (SQL server being an example of > all 3) it?s important that this is correct. I happened to work on this > feature when I was on the CLR team so it came rather naturally to me to get > it right J > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Charles Strahan > *Sent:* Sunday, October 03, 2010 9:11 PM > *To:* users at lists.ironpython.com > *Subject:* [IronPython] Question about CriticalFinalizerObject usage. > > > > Hello, > > > > I'm about to begin work on implementing the FFI gemfor IronRuby, and I was referred to IronPython's CTypes for inspiration. I > noticed that the MemoryHolder inherits from CriticalFinalizerObject, and I > was curious why that is. I'm actually not familiar with Constrained > Execution Regions in general, so a quick high level description would be > awesome. I've briefly looked at the following MSDN docs, but they're a > little too fine-grained for me to follow: > > > > CriticalFinalizerObject Class: > http://msdn.microsoft.com/en-us/library/system.runtime.constrainedexecution.criticalfinalizerobject.aspx > > System.Runtime.ConstrainedExecution Namespace: > http://msdn.microsoft.com/en-us/library/system.runtime.constrainedexecution.aspx > > Constrained Execution Regions: > http://msdn.microsoft.com/en-us/library/ms228973.aspx > > Reliability Best Practices: > http://msdn.microsoft.com/en-us/library/ms228970.aspx > > > > Any help in understanding the purpose of CriticalFinalizerObject and why it > is used as the base class for MemoryHolder would be greatly appreciated. > > > > Thanks! > > -Charles > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Mon Oct 4 12:07:02 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Mon, 4 Oct 2010 12:07:02 +0200 Subject: [IronPython] Using builtin id in Ipy 2.6.1 Message-ID: <4cc9cdbfb70f1d28ff11276b14d8393e@mail.gmail.com> Hi, I have a very big application based on Ipy 2.6.1 We recently encountered a problem that the memory gets bigger and the system gets sluggish over time. Using windbg we have found that there are 1.6M objects of type IdDispenser+Wrapper. As I understand it, these are created whenever someone calls the builtin id method on an object. Except for assigning ids this also has the effect of maintaining a weakref to the object inorder to allow a back reference from an id to an object. We suspect this might cause some of our problems, and were wondering if there is a way to use a cheaper (one-way) id method, since the back-reference is rarely needed. We could define one ourselves but that will not work with lib methods that use id like copy.deepcopy and pickle.dumps. Is there something we can do to simplify the call to id? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Mon Oct 4 16:55:35 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Mon, 4 Oct 2010 10:55:35 -0400 Subject: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? In-Reply-To: <4C93BF31.7070900@bakalari.cz> References: <4C93BF31.7070900@bakalari.cz> Message-ID: Hi, We tried windbg, but weren't able to get anywhere with it. We have pretty much given up on trying to get IP 2.6.1 to work with .NET 4.0 as an .exe (works WONDERFULLY in 'ipy foo.py' mode), and we are now going to try running this with the 2.7 Alpha of IP. I've just installed this, and skimmed through the entire docset, and while 2.7 supposedly has integration with Visual Studio, there is no mention of it in the docs. Anybody out there tried this and gotten it to work? Also, is the full VS 2010 product needed or can this be run with VS Express? Thanks, Ken On Fri, Sep 17, 2010 at 3:19 PM, Lukas Cenovsky wrote: > I would try to run it via WinDbg to see all exceptions raised. > > -- > -- Luk?? > > > > On 17.9.2010 19:43, Ken MacDonald wrote: > > We're trying to get a WPF/IPY project converted from .NET 3.5 to 4.0 using > IPY 2.6 for .NET 4.0. I got it running fine yesterday using IPY > interactively: > > ipy etms.py > > bring up all dialogs, connects to DB, works great. > > but I try to compile it into an executable using the 'pyc.py' tool: > > C:\hs\tally\etms>ipy.exe "c:\Program Files (x86)\IronPython 2.6 for .NET > 4.0\Tools\Scripts\pyc.py" /main:etms.py /platform:x86 /target:winexe etms.py > main_window.py model.py presenter.py wpf_helpers.py pyevent.py dialogs.py > login.py kitchen_ticket.py hs_quantity.py hssecdll.py debug_settings.py > version_info.py > > and it completes 'successfully' but then if I attempt to run it, the > command prompt returns immediately and not even the login dialog appears. > There are no error messages, or indications of a problem. This worked fine > (with path changes to reference 3.5) in .NET 3.5, but is now a mystery. I've > tried a variety of pyc.py flags that seemed possibly relevant, but nothing > makes it work. Suggestions for what this could be appreciated. > Ken > > > _______________________________________________ > Users mailing listUsers at lists.ironpython.comhttp://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Mon Oct 4 18:02:13 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Mon, 4 Oct 2010 18:02:13 +0200 Subject: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Message-ID: Hi, I'm investigating memory leakage in a Ipy 2.6.1 application. Using WinDbg I found a WeakDictionary that is only increasing in size, though most of the instances are already collected. This WeakDicitionary is used by a DelegationInfo object. Looking at the code, I think I found the problem: The method CheckCleanup is called the method "public void Add(TKey key, TValue value)" *But it NOT called* from the indexer: public TValue this[TKey key] { get { return dict[key]; } set { // If the WeakHash already holds this value as a key, it will lead to a circular-reference and result // in the objects being kept alive forever. The caller needs to ensure that this cannot happen. Debug.Assert(!dict.ContainsKey(value)); dict[new WeakObject(key)] = value; } } Since the dictionary is populated using the indexer in DelegateInfo: _constantMap[target] = new WeakReference(clone = (object [])_constants.Clone()); This means that the "version" field is never updated, and the dictionary never gets cleaned up. Have I got something wrong here? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Mon Oct 4 19:22:24 2010 From: dinov at microsoft.com (Dino Viehland) Date: Mon, 4 Oct 2010 17:22:24 +0000 Subject: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary In-Reply-To: References: Message-ID: Nope, you have it right - this is fixed in the Main branch and internally it's fixed in the 2.6 branch as well (There was a report about this from some internal uses of IronPython). Unfortunately it looks like the fix hasn't propagated out to the 2.6 branch on CodePlex for some reason. I'll try and figure out what's going on there but either way it'll be fixed in the next set of releases. If you want to build from source you can just add a call to CheckCleanup(); in there before the dict[new WeakObject(...)] = ... assignment. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Monday, October 04, 2010 9:02 AM To: Discussion of IronPython Subject: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Hi, I'm investigating memory leakage in a Ipy 2.6.1 application. Using WinDbg I found a WeakDictionary that is only increasing in size, though most of the instances are already collected. This WeakDicitionary is used by a DelegationInfo object. Looking at the code, I think I found the problem: The method CheckCleanup is called the method "public void Add(TKey key, TValue value)" But it NOT called from the indexer: public TValue this[TKey key] { get { return dict[key]; } set { // If the WeakHash already holds this value as a key, it will lead to a circular-reference and result // in the objects being kept alive forever. The caller needs to ensure that this cannot happen. Debug.Assert(!dict.ContainsKey(value)); dict[new WeakObject(key)] = value; } } Since the dictionary is populated using the indexer in DelegateInfo: _constantMap[target] = new WeakReference(clone = (object[])_constants.Clone()); This means that the "version" field is never updated, and the dictionary never gets cleaned up. Have I got something wrong here? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Mon Oct 4 19:27:02 2010 From: dinov at microsoft.com (Dino Viehland) Date: Mon, 4 Oct 2010 17:27:02 +0000 Subject: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? In-Reply-To: References: <4C93BF31.7070900@bakalari.cz> Message-ID: When running w/ windbg a couple of tips would be to first issue the command: .loadby sos clr Then do: sxe clr And then: g And when you break on each exception do: !pe Those will load the SOS debugging extension for the CLR, enable breaking on 1st chance CLR exceptions, run the program, and then !pe will print the current exception. There?s almost certainly an exception in there somewhere which is causing things not to work. For the VS integration ? it doesn?t work w/ the express SKUs simply because the express SKUs don?t allow extensions. But it does work w/ the free integrated shell which you can download here http://www.microsoft.com/downloads/en/details.aspx?FamilyID=8e5aa7b6-8436-43f0-b778-00c3bca733d3&displaylang=en There?s information on the tools over here http://www.ironpython.net/tools/ including a walk through the spec. The documentation for it isn?t yet integrated into the CHM which comes w/ 2.7. I think lots of people have gotten it to work. You?ll need to have VS installed before you install 2.7 or you?ll need to re-run the 2.7 installer after installing to install the tools. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Monday, October 04, 2010 7:56 AM To: Discussion of IronPython Subject: Re: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? Hi, We tried windbg, but weren't able to get anywhere with it. We have pretty much given up on trying to get IP 2.6.1 to work with .NET 4.0 as an .exe (works WONDERFULLY in 'ipy foo.py' mode), and we are now going to try running this with the 2.7 Alpha of IP. I've just installed this, and skimmed through the entire docset, and while 2.7 supposedly has integration with Visual Studio, there is no mention of it in the docs. Anybody out there tried this and gotten it to work? Also, is the full VS 2010 product needed or can this be run with VS Express? Thanks, Ken On Fri, Sep 17, 2010 at 3:19 PM, Lukas Cenovsky > wrote: I would try to run it via WinDbg to see all exceptions raised. -- -- Luk?? On 17.9.2010 19:43, Ken MacDonald wrote: We're trying to get a WPF/IPY project converted from .NET 3.5 to 4.0 using IPY 2.6 for .NET 4.0. I got it running fine yesterday using IPY interactively: ipy etms.py bring up all dialogs, connects to DB, works great. but I try to compile it into an executable using the 'pyc.py' tool: C:\hs\tally\etms>ipy.exe "c:\Program Files (x86)\IronPython 2.6 for .NET 4.0\Tools\Scripts\pyc.py" /main:etms.py /platform:x86 /target:winexe etms.py main_window.py model.py presenter.py wpf_helpers.py pyevent.py dialogs.py login.py kitchen_ticket.py hs_quantity.py hssecdll.py debug_settings.py version_info.py and it completes 'successfully' but then if I attempt to run it, the command prompt returns immediately and not even the login dialog appears. There are no error messages, or indications of a problem. This worked fine (with path changes to reference 3.5) in .NET 3.5, but is now a mystery. I've tried a variety of pyc.py flags that seemed possibly relevant, but nothing makes it work. Suggestions for what this could be appreciated. Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Mon Oct 4 22:06:37 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Mon, 4 Oct 2010 16:06:37 -0400 Subject: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? In-Reply-To: References: <4C93BF31.7070900@bakalari.cz> Message-ID: Hi Dino, Thanks for the info / pointers on VS integration. I looked at the schedule in the spec, and it appears that "build to .exe" support is not going to be integrated for some time, whenever "P2" is; is there a schedule for that? I got that stuff installed, and was able to get a very simple app (hello, world-ish) up and running in VS/IronPython, but I'm still unable to do a build-out from VS and get a real .exe out of the deal, which was what I was hoping to be able to do with the VS integration. We are sort of at a crossroads here where we want to see if we can stay w/IP as our dev platform, or if we need to port the whole beast to C#, which seems like a large pain. Ken On Mon, Oct 4, 2010 at 1:27 PM, Dino Viehland wrote: > > For the VS integration ? it doesn?t work w/ the express SKUs simply because > the express SKUs don?t allow extensions. But it does work w/ the free > integrated shell which you can download here > http://www.microsoft.com/downloads/en/details.aspx?FamilyID=8e5aa7b6-8436-43f0-b778-00c3bca733d3&displaylang=en > > > > There?s information on the tools over here > http://www.ironpython.net/tools/ including a walk through the spec. The > documentation for it isn?t yet integrated into the CHM which comes w/ 2.7. > I think lots of people have gotten it to work. You?ll need to have VS > installed before you install 2.7 or you?ll need to re-run the 2.7 installer > after installing to install the tools. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Monday, October 04, 2010 7:56 AM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] difference in compiling ipy in 3.5 vs. .NET > 4.0? > > > > Hi, > We tried windbg, but weren't able to get anywhere with it. We have pretty > much given up on trying to get IP 2.6.1 to work with .NET 4.0 as an .exe > (works WONDERFULLY in 'ipy foo.py' mode), and we are now going to try > running this with the 2.7 Alpha of IP. I've just installed this, and skimmed > through the entire docset, and while 2.7 supposedly has integration with > Visual Studio, there is no mention of it in the docs. Anybody out there > tried this and gotten it to work? Also, is the full VS 2010 product needed > or can this be run with VS Express? > Thanks, > Ken > > On Fri, Sep 17, 2010 at 3:19 PM, Lukas Cenovsky > wrote: > > I would try to run it via WinDbg to see all exceptions raised. > > -- > -- Luk?? > > > > > On 17.9.2010 19:43, Ken MacDonald wrote: > > We're trying to get a WPF/IPY project converted from .NET 3.5 to 4.0 > using IPY 2.6 for .NET 4.0. I got it running fine yesterday using IPY > interactively: > > ipy etms.py > > bring up all dialogs, connects to DB, works great. > > but I try to compile it into an executable using the 'pyc.py' tool: > > C:\hs\tally\etms>ipy.exe "c:\Program Files (x86)\IronPython 2.6 for .NET > 4.0\Tools\Scripts\pyc.py" /main:etms.py /platform:x86 /target:winexe etms.py > main_window.py model.py presenter.py wpf_helpers.py pyevent.py dialogs.py > login.py kitchen_ticket.py hs_quantity.py hssecdll.py debug_settings.py > version_info.py > > and it completes 'successfully' but then if I attempt to run it, the > command prompt returns immediately and not even the login dialog appears. > There are no error messages, or indications of a problem. This worked fine > (with path changes to reference 3.5) in .NET 3.5, but is now a mystery. I've > tried a variety of pyc.py flags that seemed possibly relevant, but nothing > makes it work. Suggestions for what this could be appreciated. > Ken > > > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Mon Oct 4 23:24:12 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Mon, 4 Oct 2010 17:24:12 -0400 Subject: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? In-Reply-To: References: <4C93BF31.7070900@bakalari.cz> Message-ID: Hm, well whatever happened in 2.7, it now seems like I can get a working .exe as output from pyc.py. All I have is a really simple test app, but it's doing .NET 4.0 stuff, and running from the .exe. I have no idea what changed from 2.6.1 to make it workable again. Anyway, hopefully I can move over the whole app tomorrow and have some better luck with that. Ken On Mon, Oct 4, 2010 at 4:06 PM, Ken MacDonald wrote: > Hi Dino, > Thanks for the info / pointers on VS integration. I looked at the schedule > in the spec, and it appears that "build to .exe" support is not going to be > integrated for some time, whenever "P2" is; is there a schedule for that? > > I got that stuff installed, and was able to get a very simple app (hello, > world-ish) up and running in VS/IronPython, but I'm still unable to do a > build-out from VS and get a real .exe out of the deal, which was what I was > hoping to be able to do with the VS integration. We are sort of at a > crossroads here where we want to see if we can stay w/IP as our dev > platform, or if we need to port the whole beast to C#, which seems like a > large pain. > Ken > > > On Mon, Oct 4, 2010 at 1:27 PM, Dino Viehland wrote: > >> >> For the VS integration ? it doesn?t work w/ the express SKUs simply >> because the express SKUs don?t allow extensions. But it does work w/ the >> free integrated shell which you can download here >> http://www.microsoft.com/downloads/en/details.aspx?FamilyID=8e5aa7b6-8436-43f0-b778-00c3bca733d3&displaylang=en >> >> >> >> There?s information on the tools over here >> http://www.ironpython.net/tools/ including a walk through the spec. The >> documentation for it isn?t yet integrated into the CHM which comes w/ 2.7. >> I think lots of people have gotten it to work. You?ll need to have VS >> installed before you install 2.7 or you?ll need to re-run the 2.7 installer >> after installing to install the tools. >> >> >> >> *From:* users-bounces at lists.ironpython.com [mailto: >> users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald >> *Sent:* Monday, October 04, 2010 7:56 AM >> *To:* Discussion of IronPython >> *Subject:* Re: [IronPython] difference in compiling ipy in 3.5 vs. .NET >> 4.0? >> >> >> >> Hi, >> We tried windbg, but weren't able to get anywhere with it. We have pretty >> much given up on trying to get IP 2.6.1 to work with .NET 4.0 as an .exe >> (works WONDERFULLY in 'ipy foo.py' mode), and we are now going to try >> running this with the 2.7 Alpha of IP. I've just installed this, and skimmed >> through the entire docset, and while 2.7 supposedly has integration with >> Visual Studio, there is no mention of it in the docs. Anybody out there >> tried this and gotten it to work? Also, is the full VS 2010 product needed >> or can this be run with VS Express? >> Thanks, >> Ken >> >> On Fri, Sep 17, 2010 at 3:19 PM, Lukas Cenovsky >> wrote: >> >> I would try to run it via WinDbg to see all exceptions raised. >> >> -- >> -- Luk?? >> >> >> >> >> On 17.9.2010 19:43, Ken MacDonald wrote: >> >> We're trying to get a WPF/IPY project converted from .NET 3.5 to 4.0 >> using IPY 2.6 for .NET 4.0. I got it running fine yesterday using IPY >> interactively: >> >> ipy etms.py >> >> bring up all dialogs, connects to DB, works great. >> >> but I try to compile it into an executable using the 'pyc.py' tool: >> >> C:\hs\tally\etms>ipy.exe "c:\Program Files (x86)\IronPython 2.6 for .NET >> 4.0\Tools\Scripts\pyc.py" /main:etms.py /platform:x86 /target:winexe etms.py >> main_window.py model.py presenter.py wpf_helpers.py pyevent.py dialogs.py >> login.py kitchen_ticket.py hs_quantity.py hssecdll.py debug_settings.py >> version_info.py >> >> and it completes 'successfully' but then if I attempt to run it, the >> command prompt returns immediately and not even the login dialog appears. >> There are no error messages, or indications of a problem. This worked fine >> (with path changes to reference 3.5) in .NET 3.5, but is now a mystery. I've >> tried a variety of pyc.py flags that seemed possibly relevant, but nothing >> makes it work. Suggestions for what this could be appreciated. >> Ken >> >> >> >> _______________________________________________ >> >> Users mailing list >> >> Users at lists.ironpython.com >> >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Mon Oct 4 23:30:48 2010 From: dinov at microsoft.com (Dino Viehland) Date: Mon, 4 Oct 2010 21:30:48 +0000 Subject: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? In-Reply-To: References: <4C93BF31.7070900@bakalari.cz> Message-ID: Cool, I?ll try and see if I can repro the issue w/ 2.6.1 w/ a simple hello world app. Regarding when we?ll have the build to EXE support in tools ? I doubt it?ll happen for v1 of the tools unfortunately. In general P2 features mean we?ll do them in a future version. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Monday, October 04, 2010 2:24 PM To: Discussion of IronPython Subject: Re: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? Hm, well whatever happened in 2.7, it now seems like I can get a working .exe as output from pyc.py. All I have is a really simple test app, but it's doing .NET 4.0 stuff, and running from the .exe. I have no idea what changed from 2.6.1 to make it workable again. Anyway, hopefully I can move over the whole app tomorrow and have some better luck with that. Ken On Mon, Oct 4, 2010 at 4:06 PM, Ken MacDonald > wrote: Hi Dino, Thanks for the info / pointers on VS integration. I looked at the schedule in the spec, and it appears that "build to .exe" support is not going to be integrated for some time, whenever "P2" is; is there a schedule for that? I got that stuff installed, and was able to get a very simple app (hello, world-ish) up and running in VS/IronPython, but I'm still unable to do a build-out from VS and get a real .exe out of the deal, which was what I was hoping to be able to do with the VS integration. We are sort of at a crossroads here where we want to see if we can stay w/IP as our dev platform, or if we need to port the whole beast to C#, which seems like a large pain. Ken On Mon, Oct 4, 2010 at 1:27 PM, Dino Viehland > wrote: For the VS integration ? it doesn?t work w/ the express SKUs simply because the express SKUs don?t allow extensions. But it does work w/ the free integrated shell which you can download here http://www.microsoft.com/downloads/en/details.aspx?FamilyID=8e5aa7b6-8436-43f0-b778-00c3bca733d3&displaylang=en There?s information on the tools over here http://www.ironpython.net/tools/ including a walk through the spec. The documentation for it isn?t yet integrated into the CHM which comes w/ 2.7. I think lots of people have gotten it to work. You?ll need to have VS installed before you install 2.7 or you?ll need to re-run the 2.7 installer after installing to install the tools. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Monday, October 04, 2010 7:56 AM To: Discussion of IronPython Subject: Re: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? Hi, We tried windbg, but weren't able to get anywhere with it. We have pretty much given up on trying to get IP 2.6.1 to work with .NET 4.0 as an .exe (works WONDERFULLY in 'ipy foo.py' mode), and we are now going to try running this with the 2.7 Alpha of IP. I've just installed this, and skimmed through the entire docset, and while 2.7 supposedly has integration with Visual Studio, there is no mention of it in the docs. Anybody out there tried this and gotten it to work? Also, is the full VS 2010 product needed or can this be run with VS Express? Thanks, Ken On Fri, Sep 17, 2010 at 3:19 PM, Lukas Cenovsky > wrote: I would try to run it via WinDbg to see all exceptions raised. -- -- Luk?? On 17.9.2010 19:43, Ken MacDonald wrote: We're trying to get a WPF/IPY project converted from .NET 3.5 to 4.0 using IPY 2.6 for .NET 4.0. I got it running fine yesterday using IPY interactively: ipy etms.py bring up all dialogs, connects to DB, works great. but I try to compile it into an executable using the 'pyc.py' tool: C:\hs\tally\etms>ipy.exe "c:\Program Files (x86)\IronPython 2.6 for .NET 4.0\Tools\Scripts\pyc.py" /main:etms.py /platform:x86 /target:winexe etms.py main_window.py model.py presenter.py wpf_helpers.py pyevent.py dialogs.py login.py kitchen_ticket.py hs_quantity.py hssecdll.py debug_settings.py version_info.py and it completes 'successfully' but then if I attempt to run it, the command prompt returns immediately and not even the login dialog appears. There are no error messages, or indications of a problem. This worked fine (with path changes to reference 3.5) in .NET 3.5, but is now a mystery. I've tried a variety of pyc.py flags that seemed possibly relevant, but nothing makes it work. Suggestions for what this could be appreciated. Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From merllab at microsoft.com Mon Oct 4 23:32:20 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Mon, 4 Oct 2010 14:32:20 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/77894. ADDED SOURCES $/IronPython/IronPython_2_6/Src/IronPython4.sln DELETED SOURCES $/IronPython/IronPython_2_6/Src/Hosts $/IronPython/IronPython_2_6/Src/IronPython.sln $/IronPython/IronPython_2_6/Src/Runtime/Microsoft.Scripting.Core $/IronPython/IronPython_2_6/Src/Tests/ClrAssembly $/IronPython/IronPython_2_6/Src/Tests/DlrComLibrary MODIFIED SOURCES $/IronPython/IronPython_2_6/Src/AssemblyVersion.cs $/IronPython/IronPython_2_6/Src/IronPython4.sln $/IronPython/IronPython_2_6/Src/IronPython.Modules/IronPython.Modules.csproj $/IronPython/IronPython_2_6/Src/IronPython/IronPython.csproj $/IronPython/IronPython_2_6/Src/IronPython/Compiler/Parser.cs $/IronPython/IronPython_2_6/Src/IronPython/Compiler/Ast/AstMethods.cs $/IronPython/IronPython_2_6/Src/IronPython/Compiler/Ast/CallExpression.cs $/IronPython/IronPython_2_6/Src/IronPython/Compiler/Ast/IfStatement.cs $/IronPython/IronPython_2_6/Src/IronPython/Runtime/PythonContext.cs $/IronPython/IronPython_2_6/Src/IronPython/Runtime/PythonOptions.cs $/IronPython/IronPython_2_6/Src/IronPython/Runtime/Operations/PythonOps.cs $/IronPython/IronPython_2_6/Src/IronPythonTest/IronPythonTest.csproj $/IronPython/IronPython_2_6/Src/Runtime/Microsoft.Dynamic/Microsoft.Dynamic.csproj $/IronPython/IronPython_2_6/Src/Runtime/Microsoft.Dynamic/Properties/AssemblyInfo.cs $/IronPython/IronPython_2_6/Src/Runtime/Microsoft.Dynamic/Utils/WeakDictionary.cs $/IronPython/IronPython_2_6/Src/Runtime/Microsoft.Scripting.Debugging/AssemblyInfo.cs $/IronPython/IronPython_2_6/Src/Runtime/Microsoft.Scripting.Debugging/Microsoft.Scripting.Debugging.csproj $/IronPython/IronPython_2_6/Src/Runtime/Microsoft.Scripting/Microsoft.Scripting.csproj $/IronPython/IronPython_2_6/Src/Runtime/Microsoft.Scripting/Properties/AssemblyInfo.cs $/IronPython/IronPython_2_6/Src/Tests/test_class.py $/IronPython/IronPython_2_6/Src/Tests/test_conditional.py $/IronPython/IronPython_2_6/Src/Tests/test_imp.py $/IronPython/IronPython_2_6/Src/Tests/test_ipye.py CHECKIN COMMENTS -------------------------------------------------------------------------------- Changeset Id: 1928757 Date: 7/21/2010 2:09:44 AM unicode pre-compile stack overflow on nested if's 27247 objcet.__delattr__ (tested) 27547 Problem with ScriptSource.GetCodeProperties() We need to pass empty parameters if we fail to parse them (tested) 26940 Wrong line numbers in traceback when encoding is specified We should always seek back to the beginning of the file so line numbers remain correct (tested) (Shelveset: 262fixesfinal;REDMOND\dinov | SNAP CheckinId: 11060) From dinov at microsoft.com Mon Oct 4 23:35:18 2010 From: dinov at microsoft.com (Dino Viehland) Date: Mon, 4 Oct 2010 21:35:18 +0000 Subject: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary In-Reply-To: References: Message-ID: The 2.6.1 sources have successfully been pushed now so this fix is publicly available in that branch now too. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Monday, October 04, 2010 10:22 AM To: Discussion of IronPython Subject: Re: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Nope, you have it right - this is fixed in the Main branch and internally it's fixed in the 2.6 branch as well (There was a report about this from some internal uses of IronPython). Unfortunately it looks like the fix hasn't propagated out to the 2.6 branch on CodePlex for some reason. I'll try and figure out what's going on there but either way it'll be fixed in the next set of releases. If you want to build from source you can just add a call to CheckCleanup(); in there before the dict[new WeakObject(...)] = ... assignment. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Monday, October 04, 2010 9:02 AM To: Discussion of IronPython Subject: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Hi, I'm investigating memory leakage in a Ipy 2.6.1 application. Using WinDbg I found a WeakDictionary that is only increasing in size, though most of the instances are already collected. This WeakDicitionary is used by a DelegationInfo object. Looking at the code, I think I found the problem: The method CheckCleanup is called the method "public void Add(TKey key, TValue value)" But it NOT called from the indexer: public TValue this[TKey key] { get { return dict[key]; } set { // If the WeakHash already holds this value as a key, it will lead to a circular-reference and result // in the objects being kept alive forever. The caller needs to ensure that this cannot happen. Debug.Assert(!dict.ContainsKey(value)); dict[new WeakObject(key)] = value; } } Since the dictionary is populated using the indexer in DelegateInfo: _constantMap[target] = new WeakReference(clone = (object[])_constants.Clone()); This means that the "version" field is never updated, and the dictionary never gets cleaned up. Have I got something wrong here? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Tue Oct 5 08:14:38 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Tue, 5 Oct 2010 08:14:38 +0200 Subject: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary In-Reply-To: References: Message-ID: Great, Thanks! *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Monday, October 04, 2010 11:35 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary The 2.6.1 sources have successfully been pushed now so this fix is publicly available in that branch now too. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Monday, October 04, 2010 10:22 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Nope, you have it right ? this is fixed in the Main branch and internally it?s fixed in the 2.6 branch as well (There was a report about this from some internal uses of IronPython). Unfortunately it looks like the fix hasn?t propagated out to the 2.6 branch on CodePlex for some reason. I?ll try and figure out what?s going on there but either way it?ll be fixed in the next set of releases. If you want to build from source you can just add a call to CheckCleanup(); in there before the dict[new WeakObject(?)] = ? assignment. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Monday, October 04, 2010 9:02 AM *To:* Discussion of IronPython *Subject:* [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Hi, I'm investigating memory leakage in a Ipy 2.6.1 application. Using WinDbg I found a WeakDictionary that is only increasing in size, though most of the instances are already collected. This WeakDicitionary is used by a DelegationInfo object. Looking at the code, I think I found the problem: The method CheckCleanup is called the method "public void Add(TKey key, TValue value)" *But it NOT called* from the indexer: public TValue this[TKey key] { get { return dict[key]; } set { // If the WeakHash already holds this value as a key, it will lead to a circular-reference and result // in the objects being kept alive forever. The caller needs to ensure that this cannot happen. Debug.Assert(!dict.ContainsKey(value)); dict[new WeakObject(key)] = value; } } Since the dictionary is populated using the indexer in DelegateInfo: _constantMap[target] = new WeakReference(clone = (object [])_constants.Clone()); This means that the "version" field is never updated, and the dictionary never gets cleaned up. Have I got something wrong here? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Tue Oct 5 11:25:42 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Tue, 5 Oct 2010 11:25:42 +0200 Subject: [IronPython] HostCodeHeap leakage? Message-ID: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Tue Oct 5 11:46:12 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Tue, 5 Oct 2010 11:46:12 +0200 Subject: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary In-Reply-To: References: Message-ID: <9e9dac4aeb5019d33ccad7b42c384beb@mail.gmail.com> I want to try to build from source. We are using the 2.6.1 release from April on .Net 2.0, so I guess I need to download: IronPython-2.6.1-Src-Net20SP1.zip After that, can you tell which configuration to use in Visual Studio in order to get the same result as d/ling the binaries directly from site? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Monday, October 04, 2010 7:22 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Nope, you have it right ? this is fixed in the Main branch and internally it?s fixed in the 2.6 branch as well (There was a report about this from some internal uses of IronPython). Unfortunately it looks like the fix hasn?t propagated out to the 2.6 branch on CodePlex for some reason. I?ll try and figure out what?s going on there but either way it?ll be fixed in the next set of releases. If you want to build from source you can just add a call to CheckCleanup(); in there before the dict[new WeakObject(?)] = ? assignment. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Monday, October 04, 2010 9:02 AM *To:* Discussion of IronPython *Subject:* [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Hi, I'm investigating memory leakage in a Ipy 2.6.1 application. Using WinDbg I found a WeakDictionary that is only increasing in size, though most of the instances are already collected. This WeakDicitionary is used by a DelegationInfo object. Looking at the code, I think I found the problem: The method CheckCleanup is called the method "public void Add(TKey key, TValue value)" *But it NOT called* from the indexer: public TValue this[TKey key] { get { return dict[key]; } set { // If the WeakHash already holds this value as a key, it will lead to a circular-reference and result // in the objects being kept alive forever. The caller needs to ensure that this cannot happen. Debug.Assert(!dict.ContainsKey(value)); dict[new WeakObject(key)] = value; } } Since the dictionary is populated using the indexer in DelegateInfo: _constantMap[target] = new WeakReference(clone = (object [])_constants.Clone()); This means that the "version" field is never updated, and the dictionary never gets cleaned up. Have I got something wrong here? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Tue Oct 5 17:51:47 2010 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 5 Oct 2010 15:51:47 +0000 Subject: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary In-Reply-To: <9e9dac4aeb5019d33ccad7b42c384beb@mail.gmail.com> References: <9e9dac4aeb5019d33ccad7b42c384beb@mail.gmail.com> Message-ID: Release is what we build for final released binaries. By default we produce unsigned binaries unless the strong name key is present. If you rename "MSSharedLibDelaySigned.snk" to "MSSharedLibKey.snk" then you will produce delay signed binaries w/ the same key that we use. You unfortunately cannot produce really signed binaries because it requires submitting the binaries to a service inside of Microsoft which signs them w/ the real key. You can disable verification of the assemblies with "sn -Vr *,31bf3856ad364e35" (or individually like sn -Vr IronPython,31bf3856ad364e35 which is the more secure thing to do). Or you can just use the unsigned ones which might be all around easier. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 2:46 AM To: Discussion of IronPython Subject: Re: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary I want to try to build from source. We are using the 2.6.1 release from April on .Net 2.0, so I guess I need to download: IronPython-2.6.1-Src-Net20SP1.zip After that, can you tell which configuration to use in Visual Studio in order to get the same result as d/ling the binaries directly from site? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Monday, October 04, 2010 7:22 PM To: Discussion of IronPython Subject: Re: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Nope, you have it right - this is fixed in the Main branch and internally it's fixed in the 2.6 branch as well (There was a report about this from some internal uses of IronPython). Unfortunately it looks like the fix hasn't propagated out to the 2.6 branch on CodePlex for some reason. I'll try and figure out what's going on there but either way it'll be fixed in the next set of releases. If you want to build from source you can just add a call to CheckCleanup(); in there before the dict[new WeakObject(...)] = ... assignment. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Monday, October 04, 2010 9:02 AM To: Discussion of IronPython Subject: [IronPython] Possible bug in Ipy 2.6.1 WeakDictionary Hi, I'm investigating memory leakage in a Ipy 2.6.1 application. Using WinDbg I found a WeakDictionary that is only increasing in size, though most of the instances are already collected. This WeakDicitionary is used by a DelegationInfo object. Looking at the code, I think I found the problem: The method CheckCleanup is called the method "public void Add(TKey key, TValue value)" But it NOT called from the indexer: public TValue this[TKey key] { get { return dict[key]; } set { // If the WeakHash already holds this value as a key, it will lead to a circular-reference and result // in the objects being kept alive forever. The caller needs to ensure that this cannot happen. Debug.Assert(!dict.ContainsKey(value)); dict[new WeakObject(key)] = value; } } Since the dictionary is populated using the indexer in DelegateInfo: _constantMap[target] = new WeakReference(clone = (object[])_constants.Clone()); This means that the "version" field is never updated, and the dictionary never gets cleaned up. Have I got something wrong here? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Tue Oct 5 17:53:29 2010 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 5 Oct 2010 15:53:29 +0000 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> Message-ID: My guess is that's code in the JIT heap that's building up but I'm not 100% certain. How is your code being executed? Do you have the debug option (-D or -X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 2:26 AM To: Discussion of IronPython Subject: [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Tue Oct 5 17:59:58 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Tue, 5 Oct 2010 11:59:58 -0400 Subject: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? In-Reply-To: References: <4C93BF31.7070900@bakalari.cz> Message-ID: I am getting much farther with my app now that I can actually run it as an .exe; and running under VS is a HUGE help. Currently, the app is dying on the following statement: import os When I run it in VS or from the ipy command prompt, "import os" seems to work fine, "dir(os)" yields a bunch of symbols, for instance. When I run it as a compiled .exe, it runs up to that point, and dies with an exception "no module named os". This seems like it must be some sort of path issue or similar. Where is the assembly for "os", and how do I ensure it gets found from my .exe - is there a specific env. variable, or the Windows %PATH% e.v., or something I haven't AddReference'd to???? Thanks, Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Tue Oct 5 18:09:04 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Tue, 5 Oct 2010 18:09:04 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> Message-ID: Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at voidspace.org.uk Tue Oct 5 18:27:03 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Tue, 05 Oct 2010 17:27:03 +0100 Subject: [IronPython] difference in compiling ipy in 3.5 vs. .NET 4.0? In-Reply-To: References: <4C93BF31.7070900@bakalari.cz> Message-ID: <4CAB51D7.3070805@voidspace.org.uk> On 05/10/2010 16:59, Ken MacDonald wrote: > I am getting much farther with my app now that I can actually run it > as an .exe; and running under VS is a HUGE help. Currently, the app is > dying on the following statement: > > import os > > When I run it in VS or from the ipy command prompt, "import os" seems > to work fine, "dir(os)" yields a bunch of symbols, for instance. > > When I run it as a compiled .exe, it runs up to that point, and dies > with an exception "no module named os". > > This seems like it must be some sort of path issue or similar. Where > is the assembly for "os", "os" is not an assembly but a Python module from the standard library. You need to ensure that the Python standard library (or the parts that you use and their dependencies) is on the path. All the best, Michael Foord > and how do I ensure it gets found from my .exe - is there a specific > env. variable, or the Windows %PATH% e.v., or something I haven't > AddReference'd to???? > Thanks, > Ken > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Tue Oct 5 18:58:57 2010 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 5 Oct 2010 16:58:57 +0000 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> Message-ID: Yep, DebugMode is the same as -X:Debug. In general I'd suggest making this configurable somehow and only turn it on if you're actually debugging. It's unfortunate that we can't offer both debugging & collectability but right now that's simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 9:09 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like -X:Debug? You reckon this could be the cause? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 5:53 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? My guess is that's code in the JIT heap that's building up but I'm not 100% certain. How is your code being executed? Do you have the debug option (-D or -X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 2:26 AM To: Discussion of IronPython Subject: [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Tue Oct 5 19:01:48 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Tue, 5 Oct 2010 19:01:48 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> Message-ID: <08edc35e6b4f6c9469d7654b12be325c@mail.gmail.com> OK, Ill try to remove it and see if that helps. Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 6:59 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this configurable somehow and only turn it on if you?re actually debugging. It?s unfortunate that we can?t offer both debugging & collectability but right now that?s simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 9:09 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Tue Oct 5 21:45:46 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Tue, 5 Oct 2010 15:45:46 -0400 Subject: [IronPython] Ironpython 2.7 missing _weakrefset Message-ID: I'm getting an error in my 2.7 Alpha app - "no module named _weakrefset". Distribution is apparently missing this - I saw some discussion from a while back and it sounds like it will be fixed in a future release? Is there anywhere to get a copy of this module without having to install svn and jump though a lot of hoops? Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Tue Oct 5 21:48:14 2010 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 5 Oct 2010 19:48:14 +0000 Subject: [IronPython] Ironpython 2.7 missing _weakrefset In-Reply-To: References: Message-ID: It's not fixed yet and I don't think I was aware of it. Most likely you can modify weakref.py to just not import this as a work around. Could you open a bug to open it? That shouldn't be too difficult given that we already have weak dictionaries and sets. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Tuesday, October 05, 2010 12:46 PM To: Discussion of IronPython Subject: [IronPython] Ironpython 2.7 missing _weakrefset I'm getting an error in my 2.7 Alpha app - "no module named _weakrefset". Distribution is apparently missing this - I saw some discussion from a while back and it sounds like it will be fixed in a future release? Is there anywhere to get a copy of this module without having to install svn and jump though a lot of hoops? Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Tue Oct 5 22:14:24 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Tue, 5 Oct 2010 16:14:24 -0400 Subject: [IronPython] Ironpython 2.7 missing _weakrefset In-Reply-To: References: Message-ID: Oooops. I think I just had an incomplete copy; re-did it and _weakrefset.py showed up. Sorry for the alarm.... Ken On Tue, Oct 5, 2010 at 3:48 PM, Dino Viehland wrote: > It?s not fixed yet and I don?t think I was aware of it. Most likely you > can modify weakref.py to just not import this as a work around. Could you > open a bug to open it? That shouldn?t be too difficult given that we > already have weak dictionaries and sets. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Tuesday, October 05, 2010 12:46 PM > *To:* Discussion of IronPython > *Subject:* [IronPython] Ironpython 2.7 missing _weakrefset > > > > I'm getting an error in my 2.7 Alpha app - "no module named _weakrefset". > Distribution is apparently missing this - I saw some discussion from a while > back and it sounds like it will be fixed in a future release? Is there > anywhere to get a copy of this module without having to install svn and jump > though a lot of hoops? > Ken > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Tue Oct 5 22:27:27 2010 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 5 Oct 2010 20:27:27 +0000 Subject: [IronPython] Ironpython 2.7 missing _weakrefset In-Reply-To: References: Message-ID: Cool, usually those _ modules are built-ins that we need to implement. I'm glad to hear it's in Python instead :) From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Tuesday, October 05, 2010 1:14 PM To: Discussion of IronPython Subject: Re: [IronPython] Ironpython 2.7 missing _weakrefset Oooops. I think I just had an incomplete copy; re-did it and _weakrefset.py showed up. Sorry for the alarm.... Ken On Tue, Oct 5, 2010 at 3:48 PM, Dino Viehland > wrote: It's not fixed yet and I don't think I was aware of it. Most likely you can modify weakref.py to just not import this as a work around. Could you open a bug to open it? That shouldn't be too difficult given that we already have weak dictionaries and sets. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Tuesday, October 05, 2010 12:46 PM To: Discussion of IronPython Subject: [IronPython] Ironpython 2.7 missing _weakrefset I'm getting an error in my 2.7 Alpha app - "no module named _weakrefset". Distribution is apparently missing this - I saw some discussion from a while back and it sounds like it will be fixed in a future release? Is there anywhere to get a copy of this module without having to install svn and jump though a lot of hoops? Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Tue Oct 5 23:27:21 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Tue, 5 Oct 2010 17:27:21 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? Message-ID: I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like "os") are not being built into the .exe/.dll, but instead are required to be imported in source form, e.g. "os.py" must be somewhere on sys.path. In the IP 2.5 .exe's we had been building, they would run fine on machines without the IP standard library installed at all, in other words, with "os.py" not present on the machine at all. We did notice that the .exe in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 version, and the newer version requires that the source code for the IP standard library be on the path. Is this a deliberate change in behavior? We never had to package the standard library source when we sent out .exe's to customers before >"os" is not an assembly but a Python module from the standard library. You need to ensure >that the Python standard library (or the parts that you use and their dependencies) is on the >path. > > All the best, > > Michael Foord > > and how do I ensure it gets found from my .exe - is there a specific env. > variable, or the Windows %PATH% e.v., or something I haven't AddReference'd > to???? > Thanks, > Ken > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.comhttp://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > -- http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, > on behalf of your employer, to release me from all obligations > and waivers arising from any and all NON-NEGOTIATED agreements, > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > confidentiality, non-disclosure, non-compete and acceptable use > policies (?BOGUS AGREEMENTS?) that I have entered into with your > employer, its partners, licensors, agents and assigns, in > perpetuity, without prejudice to my ongoing rights and privileges. > You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cenovsky at bakalari.cz Tue Oct 5 23:50:16 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Tue, 05 Oct 2010 23:50:16 +0200 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: Message-ID: <4CAB9D98.6080900@bakalari.cz> It's easy - just add the os.py to the build command with the correct path: ipy.exe Tools\Scripts\pyc.py /target:exe /main:myprg.py /out:myprg myprg.py lib\os.py You will probably need to add more references. -- -- Luk?? On 5.10.2010 23:27, Ken MacDonald wrote: > I've been looking at the .exe's we built - using pyc.py - with IP > 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that > the imports (like "os") are not being built into the .exe/.dll, but > instead are required to be imported in source form, e.g. "os.py" must > be somewhere on sys.path. In the IP 2.5 .exe's we had been building, > they would run fine on machines without the IP standard library > installed at all, in other words, with "os.py" not present on the > machine at all. We did notice that the .exe in question went from > being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 > version, and the newer version requires that the source code for the > IP standard library be on the path. Is this a deliberate change in > behavior? We never had to package the standard library source when we > sent out .exe's to customers before > > >"os" is not an assembly but a Python module from the standard > library. You need to ensure >that the Python standard library (or the > parts that you use and their dependencies) is on the >path. > > > All the best, > > Michael Foord > >> and how do I ensure it gets found from my .exe - is there a >> specific env. variable, or the Windows %PATH% e.v., or something >> I haven't AddReference'd to???? >> Thanks, >> Ken >> >> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > -- > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, > on behalf of your employer, to release me from all obligations > and waivers arising from any and all NON-NEGOTIATED agreements, > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > confidentiality, non-disclosure, non-compete and acceptable use > policies (?BOGUS AGREEMENTS?) that I have entered into with your > employer, its partners, licensors, agents and assigns, in > perpetuity, without prejudice to my ongoing rights and privileges. > You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at voidspace.org.uk Wed Oct 6 13:08:44 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Wed, 06 Oct 2010 12:08:44 +0100 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: Message-ID: <4CAC58BC.206@voidspace.org.uk> On 05/10/2010 22:27, Ken MacDonald wrote: > I've been looking at the .exe's we built - using pyc.py - with IP > 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that > the imports (like "os") are not being built into the .exe/.dll, but > instead are required to be imported in source form, e.g. "os.py" must > be somewhere on sys.path. In the IP 2.5 .exe's we had been building, > they would run fine on machines without the IP standard library > installed at all, in other words, with "os.py" not present on the > machine at all. We did notice that the .exe in question went from > being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 > version, and the newer version requires that the source code for the > IP standard library be on the path. Is this a deliberate change in > behavior? We never had to package the standard library source when we > sent out .exe's to customers before Hmmm... I'm pretty sure I always had to explicitly compile and bundle the standard library with previous versions of Python. Odd. Anyway, the simple solution is to ensure that you add any standard library modules you use to the set you compile and ship. All the best, Michael Foord > > >"os" is not an assembly but a Python module from the standard > library. You need to ensure >that the Python standard library (or the > parts that you use and their dependencies) is on the >path. > > > All the best, > > Michael Foord > >> and how do I ensure it gets found from my .exe - is there a >> specific env. variable, or the Windows %PATH% e.v., or something >> I haven't AddReference'd to???? >> Thanks, >> Ken >> >> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > -- > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, > on behalf of your employer, to release me from all obligations > and waivers arising from any and all NON-NEGOTIATED agreements, > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > confidentiality, non-disclosure, non-compete and acceptable use > policies (?BOGUS AGREEMENTS?) that I have entered into with your > employer, its partners, licensors, agents and assigns, in > perpetuity, without prejudice to my ongoing rights and privileges. > You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From merllab at microsoft.com Wed Oct 6 18:02:21 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Wed, 6 Oct 2010 09:02:21 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: <0fd29f21-fb00-4256-9956-a1c84c673d83@tk5-exsmh-c101.redmond.corp.microsoft.com> This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/78026. MODIFIED SOURCES $/IronPython/IronPython_Main/Languages/Ruby/IronRuby.Tests/Runtime/DlrInteropTests.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Initializers.Generated.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Builtins/Enumerable.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Extensions/IListOps.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/RubyStackTraceBuilder.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/Calls/InteropBinder.cs $/IronPython/IronPython_Main/Languages/Ruby/Tests/Scripts/irtests.rb From drken567 at gmail.com Wed Oct 6 20:42:03 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Wed, 6 Oct 2010 14:42:03 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: <4CAC58BC.206@voidspace.org.uk> References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Hi Michael, I started out on implementing this, but I am importing maybe a dozen of the std. library modules, which then import others, and so on. It appears that eventually, most of the std modules would have to be imported explicitly (perhaps 400 or so files) which might make for a somewhat cumbersome command line, incidentally also about 20K characters too long :-). I'm hoping to find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one of us can suggest something if we have an understanding of what you're trying to do. Ken On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord wrote: > On 05/10/2010 22:27, Ken MacDonald wrote: > > I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net > 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like > "os") are not being built into the .exe/.dll, but instead are required to be > imported in source form, e.g. "os.py" must be somewhere on sys.path. In the > IP 2.5 .exe's we had been building, they would run fine on machines without > the IP standard library installed at all, in other words, with "os.py" not > present on the machine at all. We did notice that the .exe in question went > from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 > version, and the newer version requires that the source code for the IP > standard library be on the path. Is this a deliberate change in behavior? We > never had to package the standard library source when we sent out .exe's to > customers before > > > Hmmm... I'm pretty sure I always had to explicitly compile and bundle the > standard library with previous versions of Python. Odd. Anyway, the simple > solution is to ensure that you add any standard library modules you use to > the set you compile and ship. > > > All the best, > > Michael Foord > > > > >"os" is not an assembly but a Python module from the standard library. You > need to ensure >that the Python standard library (or the parts that you use > and their dependencies) is on the >path. > >> >> All the best, >> >> Michael Foord >> >> and how do I ensure it gets found from my .exe - is there a specific >> env. variable, or the Windows %PATH% e.v., or something I haven't >> AddReference'd to???? >> Thanks, >> Ken >> >> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.comhttp://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> >> >> -- http://www.voidspace.org.uk/blog >> >> READ CAREFULLY. By accepting and reading this email you agree, >> on behalf of your employer, to release me from all obligations >> and waivers arising from any and all NON-NEGOTIATED agreements, >> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, >> confidentiality, non-disclosure, non-compete and acceptable use >> policies (?BOGUS AGREEMENTS?) that I have entered into with your >> employer, its partners, licensors, agents and assigns, in >> perpetuity, without prejudice to my ongoing rights and privileges. >> You further represent that you have the authority to release me >> from any BOGUS AGREEMENTS on behalf of your employer. >> >> > > > -- http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, > on behalf of your employer, to release me from all obligations > and waivers arising from any and all NON-NEGOTIATED agreements, > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > confidentiality, non-disclosure, non-compete and acceptable use > policies (?BOGUS AGREEMENTS?) that I have entered into with your > employer, its partners, licensors, agents and assigns, in > perpetuity, without prejudice to my ongoing rights and privileges. > You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Wed Oct 6 20:57:38 2010 From: dinov at microsoft.com (Dino Viehland) Date: Wed, 6 Oct 2010 18:57:38 +0000 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: How are you distributing your app? I'm assuming you're going to have something like: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... You should be able to also distribute the standard library and just drop it into a Lib directory next to IronPython.dll: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... Lib\ os.py That lib dir should be on sys.path at startup and so it should be available for importing. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 11:42 AM To: Michael Foord Cc: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Michael, I started out on implementing this, but I am importing maybe a dozen of the std. library modules, which then import others, and so on. It appears that eventually, most of the std modules would have to be imported explicitly (perhaps 400 or so files) which might make for a somewhat cumbersome command line, incidentally also about 20K characters too long :-). I'm hoping to find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one of us can suggest something if we have an understanding of what you're trying to do. Ken On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: On 05/10/2010 22:27, Ken MacDonald wrote: I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like "os") are not being built into the .exe/.dll, but instead are required to be imported in source form, e.g. "os.py" must be somewhere on sys.path. In the IP 2.5 .exe's we had been building, they would run fine on machines without the IP standard library installed at all, in other words, with "os.py" not present on the machine at all. We did notice that the .exe in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 version, and the newer version requires that the source code for the IP standard library be on the path. Is this a deliberate change in behavior? We never had to package the standard library source when we sent out .exe's to customers before Hmmm... I'm pretty sure I always had to explicitly compile and bundle the standard library with previous versions of Python. Odd. Anyway, the simple solution is to ensure that you add any standard library modules you use to the set you compile and ship. All the best, Michael Foord >"os" is not an assembly but a Python module from the standard library. You need to ensure >that the Python standard library (or the parts that you use and their dependencies) is on the >path. All the best, Michael Foord and how do I ensure it gets found from my .exe - is there a specific env. variable, or the Windows %PATH% e.v., or something I haven't AddReference'd to???? Thanks, Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Wed Oct 6 22:04:40 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Wed, 6 Oct 2010 16:04:40 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Hi Dino, On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland wrote: > How are you distributing your app? I?m assuming you?re going to have > something like: > > > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > > This is exactly how we have been distributing our app up to IP 2.6 with .NET 3.5. We also have another DLL with our resources in it; XAML, icons, image files, but no code per se. We have been distributing it this way to customer systems that do NOT have IronPython installed, or any sign of the std. library, or "os.py" specifically, and it's been working really, really well. We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that we should have to care about distributing the std. library now. The way it was before was quite simple and robust; deliver a small handful of files and it just worked, very easy to keep track of. Now instead of distributing 5 files, we suddenly need to distribute 500??? We've figured out that it certainly COULD work as described below, but it seems like a giant step backwards on several fronts, including the potential for folks to maliciously or accidentally tamper with the std lib. sources and affect the functioning of our app. So, how do we get back to the old/better functionality? On a *slightly *related note, our app imports some package directories in addition, e.g. "import ctypes". When python encounters a directory import, it looks for __init__.py in the directory, and derives the package import directions from there, as I understand it. However, I can't specify the "ctypes" directory as an argument to the pyc.py compile app; just causes it to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and the other files in the ctypes subdirectory, it seems like "import ctypes" would have no clue that the __init__.py that was compiled in had anything to do with the "ctypes" package, as the path names are presumably irrelevant to the compiler as long as they specify a python file. I'm considering mod'ing pyc.py to be able to incorporate a list of std lib modules to compile in: simple enough for the standalone files like "os.py", but the compile modules don't seem to be able to grok what to do with a package subdirectory. Ken > > > > > You should be able to also distribute the standard library and just drop it > into a Lib directory next to IronPython.dll: > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > Lib\ > > os.py > > > > That lib dir should be on sys.path at startup and so it should be available > for importing. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 11:42 AM > *To:* Michael Foord > > *Cc:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Hi Michael, > > I started out on implementing this, but I am importing maybe a dozen of the > std. library modules, which then import others, and so on. It appears that > eventually, most of the std modules would have to be imported explicitly > (perhaps 400 or so files) which might make for a somewhat cumbersome command > line, incidentally also about 20K characters too long :-). I'm hoping to > find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. > > Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one > of us can suggest something if we have an understanding of what you're > trying to do. > Ken > > On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: > > On 05/10/2010 22:27, Ken MacDonald wrote: > > I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net > 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like > "os") are not being built into the .exe/.dll, but instead are required to be > imported in source form, e.g. "os.py" must be somewhere on sys.path. In the > IP 2.5 .exe's we had been building, they would run fine on machines without > the IP standard library installed at all, in other words, with "os.py" not > present on the machine at all. We did notice that the .exe in question went > from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 > version, and the newer version requires that the source code for the IP > standard library be on the path. Is this a deliberate change in behavior? We > never had to package the standard library source when we sent out .exe's to > customers before > > > > Hmmm... I'm pretty sure I always had to explicitly compile and bundle the > standard library with previous versions of Python. Odd. Anyway, the simple > solution is to ensure that you add any standard library modules you use to > the set you compile and ship. > > > > All the best, > > Michael Foord > > > > > >"os" is not an assembly but a Python module from the standard library. You > need to ensure >that the Python standard library (or the parts that you use > and their dependencies) is on the >path. > > > All the best, > > Michael Foord > > > and how do I ensure it gets found from my .exe - is there a specific env. > variable, or the Windows %PATH% e.v., or something I haven't AddReference'd > to???? > Thanks, > Ken > > > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Wed Oct 6 22:26:05 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Wed, 6 Oct 2010 16:26:05 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: As an FYI, the following change was made from the IP 2.6 version of pyc.py to the IP 2.7 version (newer version shown first): < # get the ScriptCode assembly... < gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) < gen.EmitCall(OpCodes.Callvirt, clr.GetClrType(Assembly).GetMethod("get_Location"), ()) --- > # get the ScriptCode assembly... > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( (str, ) )); > gen.EmitCall(OpCodes.Call, clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); Don't know what these things do at this point, but wondering if the changes have to do with compiling the entire code tree into the application .DLL??? Ken On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald wrote: > Hi Dino, > > On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland wrote: > >> How are you distributing your app? I?m assuming you?re going to have >> something like: >> >> >> >> MyApp\ >> >> MyApp.exe >> >> MyApp.dll >> >> IronPython.dll >> >> IronPython.Modules.DLL >> >> ? >> >> >> > This is exactly how we have been distributing our app up to IP 2.6 with > .NET 3.5. We also have another DLL with our resources in it; XAML, icons, > image files, but no code per se. We have been distributing it this way to > customer systems that do NOT have IronPython installed, or any sign of the > std. library, or "os.py" specifically, and it's been working really, really > well. > > We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that > we should have to care about distributing the std. library now. The way it > was before was quite simple and robust; deliver a small handful of files and > it just worked, very easy to keep track of. Now instead of distributing 5 > files, we suddenly need to distribute 500??? We've figured out that it > certainly COULD work as described below, but it seems like a giant step > backwards on several fronts, including the potential for folks to > maliciously or accidentally tamper with the std lib. sources and affect the > functioning of our app. So, how do we get back to the old/better > functionality? > > On a *slightly *related note, our app imports some package directories in > addition, e.g. "import ctypes". When python encounters a directory import, > it looks for __init__.py in the directory, and derives the package import > directions from there, as I understand it. However, I can't specify the > "ctypes" directory as an argument to the pyc.py compile app; just causes it > to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and > the other files in the ctypes subdirectory, it seems like "import ctypes" > would have no clue that the __init__.py that was compiled in had anything to > do with the "ctypes" package, as the path names are presumably irrelevant to > the compiler as long as they specify a python file. I'm considering mod'ing > pyc.py to be able to incorporate a list of std lib modules to compile in: > simple enough for the standalone files like "os.py", but the compile modules > don't seem to be able to grok what to do with a package subdirectory. > Ken > > >> >> >> >> >> You should be able to also distribute the standard library and just drop >> it into a Lib directory next to IronPython.dll: >> >> MyApp\ >> >> MyApp.exe >> >> MyApp.dll >> >> IronPython.dll >> >> IronPython.Modules.DLL >> >> ? >> >> Lib\ >> >> os.py >> >> >> >> That lib dir should be on sys.path at startup and so it should be >> available for importing. >> >> >> >> *From:* users-bounces at lists.ironpython.com [mailto: >> users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald >> *Sent:* Wednesday, October 06, 2010 11:42 AM >> *To:* Michael Foord >> >> *Cc:* Discussion of IronPython >> *Subject:* Re: [IronPython] change in standard library behavior for >> compiled .exe/.dll??? >> >> >> >> Hi Michael, >> >> I started out on implementing this, but I am importing maybe a dozen of >> the std. library modules, which then import others, and so on. It appears >> that eventually, most of the std modules would have to be imported >> explicitly (perhaps 400 or so files) which might make for a somewhat >> cumbersome command line, incidentally also about 20K characters too long >> :-). I'm hoping to find a way to get this to work as well as it did under IP >> 2.5 / .NET 3.5. >> >> Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe >> one of us can suggest something if we have an understanding of what you're >> trying to do. >> Ken >> >> On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord >> wrote: >> >> On 05/10/2010 22:27, Ken MacDonald wrote: >> >> I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net >> 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like >> "os") are not being built into the .exe/.dll, but instead are required to be >> imported in source form, e.g. "os.py" must be somewhere on sys.path. In the >> IP 2.5 .exe's we had been building, they would run fine on machines without >> the IP standard library installed at all, in other words, with "os.py" not >> present on the machine at all. We did notice that the .exe in question went >> from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 >> version, and the newer version requires that the source code for the IP >> standard library be on the path. Is this a deliberate change in behavior? We >> never had to package the standard library source when we sent out .exe's to >> customers before >> >> >> >> Hmmm... I'm pretty sure I always had to explicitly compile and bundle the >> standard library with previous versions of Python. Odd. Anyway, the simple >> solution is to ensure that you add any standard library modules you use to >> the set you compile and ship. >> >> >> >> All the best, >> >> Michael Foord >> >> >> >> >> >"os" is not an assembly but a Python module from the standard library. >> You need to ensure >that the Python standard library (or the parts that you >> use and their dependencies) is on the >path. >> >> >> All the best, >> >> Michael Foord >> >> >> and how do I ensure it gets found from my .exe - is there a specific >> env. variable, or the Windows %PATH% e.v., or something I haven't >> AddReference'd to???? >> Thanks, >> Ken >> >> >> >> _______________________________________________ >> >> Users mailing list >> >> Users at lists.ironpython.com >> >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> >> >> >> -- >> >> http://www.voidspace.org.uk/blog >> >> >> >> READ CAREFULLY. By accepting and reading this email you agree, >> >> on behalf of your employer, to release me from all obligations >> >> and waivers arising from any and all NON-NEGOTIATED agreements, >> >> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, >> >> confidentiality, non-disclosure, non-compete and acceptable use >> >> policies (?BOGUS AGREEMENTS?) that I have entered into with your >> >> employer, its partners, licensors, agents and assigns, in >> >> perpetuity, without prejudice to my ongoing rights and privileges. >> >> You further represent that you have the authority to release me >> >> from any BOGUS AGREEMENTS on behalf of your employer. >> >> >> >> >> >> >> -- >> >> http://www.voidspace.org.uk/blog >> >> >> >> READ CAREFULLY. By accepting and reading this email you agree, >> >> on behalf of your employer, to release me from all obligations >> >> and waivers arising from any and all NON-NEGOTIATED agreements, >> >> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, >> >> confidentiality, non-disclosure, non-compete and acceptable use >> >> policies (?BOGUS AGREEMENTS?) that I have entered into with your >> >> employer, its partners, licensors, agents and assigns, in >> >> perpetuity, without prejudice to my ongoing rights and privileges. >> >> You further represent that you have the authority to release me >> >> from any BOGUS AGREEMENTS on behalf of your employer. >> >> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Wed Oct 6 23:15:39 2010 From: dinov at microsoft.com (Dino Viehland) Date: Wed, 6 Oct 2010 21:15:39 +0000 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: AFAIK nothing has changed between 2.7 and 2.6 w/ how we'd go about finding the standard library - and we certainly never had anything which would automatically include it. Do you have something which works on 2.6 where you could print out os.__file__ (or some other module you're importing from the std lib) and sys.path? That'd at least tell us where/how we're picking these up from in the previous version of the app and maybe we can work out what's going on from there. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 1:05 PM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Dino, On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland > wrote: How are you distributing your app? I'm assuming you're going to have something like: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... This is exactly how we have been distributing our app up to IP 2.6 with .NET 3.5. We also have another DLL with our resources in it; XAML, icons, image files, but no code per se. We have been distributing it this way to customer systems that do NOT have IronPython installed, or any sign of the std. library, or "os.py" specifically, and it's been working really, really well. We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that we should have to care about distributing the std. library now. The way it was before was quite simple and robust; deliver a small handful of files and it just worked, very easy to keep track of. Now instead of distributing 5 files, we suddenly need to distribute 500??? We've figured out that it certainly COULD work as described below, but it seems like a giant step backwards on several fronts, including the potential for folks to maliciously or accidentally tamper with the std lib. sources and affect the functioning of our app. So, how do we get back to the old/better functionality? On a slightly related note, our app imports some package directories in addition, e.g. "import ctypes". When python encounters a directory import, it looks for __init__.py in the directory, and derives the package import directions from there, as I understand it. However, I can't specify the "ctypes" directory as an argument to the pyc.py compile app; just causes it to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and the other files in the ctypes subdirectory, it seems like "import ctypes" would have no clue that the __init__.py that was compiled in had anything to do with the "ctypes" package, as the path names are presumably irrelevant to the compiler as long as they specify a python file. I'm considering mod'ing pyc.py to be able to incorporate a list of std lib modules to compile in: simple enough for the standalone files like "os.py", but the compile modules don't seem to be able to grok what to do with a package subdirectory. Ken You should be able to also distribute the standard library and just drop it into a Lib directory next to IronPython.dll: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... Lib\ os.py That lib dir should be on sys.path at startup and so it should be available for importing. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 11:42 AM To: Michael Foord Cc: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Michael, I started out on implementing this, but I am importing maybe a dozen of the std. library modules, which then import others, and so on. It appears that eventually, most of the std modules would have to be imported explicitly (perhaps 400 or so files) which might make for a somewhat cumbersome command line, incidentally also about 20K characters too long :-). I'm hoping to find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one of us can suggest something if we have an understanding of what you're trying to do. Ken On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: On 05/10/2010 22:27, Ken MacDonald wrote: I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like "os") are not being built into the .exe/.dll, but instead are required to be imported in source form, e.g. "os.py" must be somewhere on sys.path. In the IP 2.5 .exe's we had been building, they would run fine on machines without the IP standard library installed at all, in other words, with "os.py" not present on the machine at all. We did notice that the .exe in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 version, and the newer version requires that the source code for the IP standard library be on the path. Is this a deliberate change in behavior? We never had to package the standard library source when we sent out .exe's to customers before Hmmm... I'm pretty sure I always had to explicitly compile and bundle the standard library with previous versions of Python. Odd. Anyway, the simple solution is to ensure that you add any standard library modules you use to the set you compile and ship. All the best, Michael Foord >"os" is not an assembly but a Python module from the standard library. You need to ensure >that the Python standard library (or the parts that you use and their dependencies) is on the >path. All the best, Michael Foord and how do I ensure it gets found from my .exe - is there a specific env. variable, or the Windows %PATH% e.v., or something I haven't AddReference'd to???? Thanks, Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Wed Oct 6 23:28:04 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Wed, 6 Oct 2010 17:28:04 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Funny, we did exactly that yesterday. When we run the .exe on the new version (2.7), we get a sys.path of [".", "...\current_dir\DLLs", "...\current_dir\Lib"], and the os.__file__ was "...\current_dir\os.py" - where I happened to have put os.py. We rebuilt our 2.6 app, and sys.path was the same, but os.__file__ was just plain "os" - no file path listed. Almost as if it was built into the application DLL, which is where we think it used to be.... Ken On Wed, Oct 6, 2010 at 5:15 PM, Dino Viehland wrote: > AFAIK nothing has changed between 2.7 and 2.6 w/ how we?d go about > finding the standard library ? and we certainly never had anything which > would automatically include it. Do you have something which works on 2.6 > where you could print out os.__file__ (or some other module you?re importing > from the std lib) and sys.path? That?d at least tell us where/how we?re > picking these up from in the previous version of the app and maybe we can > work out what?s going on from there. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 1:05 PM > *To:* Discussion of IronPython > > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Hi Dino, > > On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland wrote: > > How are you distributing your app? I?m assuming you?re going to have > something like: > > > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > > > This is exactly how we have been distributing our app up to IP 2.6 with > .NET 3.5. We also have another DLL with our resources in it; XAML, icons, > image files, but no code per se. We have been distributing it this way to > customer systems that do NOT have IronPython installed, or any sign of the > std. library, or "os.py" specifically, and it's been working really, really > well. > > We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that > we should have to care about distributing the std. library now. The way it > was before was quite simple and robust; deliver a small handful of files and > it just worked, very easy to keep track of. Now instead of distributing 5 > files, we suddenly need to distribute 500??? We've figured out that it > certainly COULD work as described below, but it seems like a giant step > backwards on several fronts, including the potential for folks to > maliciously or accidentally tamper with the std lib. sources and affect the > functioning of our app. So, how do we get back to the old/better > functionality? > > On a *slightly *related note, our app imports some package directories in > addition, e.g. "import ctypes". When python encounters a directory import, > it looks for __init__.py in the directory, and derives the package import > directions from there, as I understand it. However, I can't specify the > "ctypes" directory as an argument to the pyc.py compile app; just causes it > to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and > the other files in the ctypes subdirectory, it seems like "import ctypes" > would have no clue that the __init__.py that was compiled in had anything to > do with the "ctypes" package, as the path names are presumably irrelevant to > the compiler as long as they specify a python file. I'm considering mod'ing > pyc.py to be able to incorporate a list of std lib modules to compile in: > simple enough for the standalone files like "os.py", but the compile modules > don't seem to be able to grok what to do with a package subdirectory. > Ken > > > > > > > You should be able to also distribute the standard library and just drop it > into a Lib directory next to IronPython.dll: > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > Lib\ > > os.py > > > > That lib dir should be on sys.path at startup and so it should be available > for importing. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 11:42 AM > *To:* Michael Foord > > > *Cc:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Hi Michael, > > > I started out on implementing this, but I am importing maybe a dozen of the > std. library modules, which then import others, and so on. It appears that > eventually, most of the std modules would have to be imported explicitly > (perhaps 400 or so files) which might make for a somewhat cumbersome command > line, incidentally also about 20K characters too long :-). I'm hoping to > find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. > > Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one > of us can suggest something if we have an understanding of what you're > trying to do. > Ken > > On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: > > On 05/10/2010 22:27, Ken MacDonald wrote: > > I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net > 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like > "os") are not being built into the .exe/.dll, but instead are required to be > imported in source form, e.g. "os.py" must be somewhere on sys.path. In the > IP 2.5 .exe's we had been building, they would run fine on machines without > the IP standard library installed at all, in other words, with "os.py" not > present on the machine at all. We did notice that the .exe in question went > from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 > version, and the newer version requires that the source code for the IP > standard library be on the path. Is this a deliberate change in behavior? We > never had to package the standard library source when we sent out .exe's to > customers before > > > > Hmmm... I'm pretty sure I always had to explicitly compile and bundle the > standard library with previous versions of Python. Odd. Anyway, the simple > solution is to ensure that you add any standard library modules you use to > the set you compile and ship. > > > > All the best, > > Michael Foord > > > > >"os" is not an assembly but a Python module from the standard library. You > need to ensure >that the Python standard library (or the parts that you use > and their dependencies) is on the >path. > > > All the best, > > Michael Foord > > and how do I ensure it gets found from my .exe - is there a specific env. > variable, or the Windows %PATH% e.v., or something I haven't AddReference'd > to???? > Thanks, > Ken > > > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Wed Oct 6 23:59:23 2010 From: dinov at microsoft.com (Dino Viehland) Date: Wed, 6 Oct 2010 21:59:23 +0000 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Hurm, will __file__ will still be set even for pre-compiled DLLs so it's not picking up a pre-compiled version. What about sys.builtin_module_names on 2.6? Does it have os listed in there somehow? And does os have the same members at the nt module or does it have significantly more? For example if I do "set(dir(os)) - set(dir(nt))" there's 40 more members in os (os just builds on top of the underlying nt/unix/other platform built-in module so if these are getting aliased somehow then os would have the same members). From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 2:28 PM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Funny, we did exactly that yesterday. When we run the .exe on the new version (2.7), we get a sys.path of [".", "...\current_dir\DLLs", "...\current_dir\Lib"], and the os.__file__ was "...\current_dir\os.py" - where I happened to have put os.py. We rebuilt our 2.6 app, and sys.path was the same, but os.__file__ was just plain "os" - no file path listed. Almost as if it was built into the application DLL, which is where we think it used to be.... Ken On Wed, Oct 6, 2010 at 5:15 PM, Dino Viehland > wrote: AFAIK nothing has changed between 2.7 and 2.6 w/ how we'd go about finding the standard library - and we certainly never had anything which would automatically include it. Do you have something which works on 2.6 where you could print out os.__file__ (or some other module you're importing from the std lib) and sys.path? That'd at least tell us where/how we're picking these up from in the previous version of the app and maybe we can work out what's going on from there. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 1:05 PM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Dino, On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland > wrote: How are you distributing your app? I'm assuming you're going to have something like: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... This is exactly how we have been distributing our app up to IP 2.6 with .NET 3.5. We also have another DLL with our resources in it; XAML, icons, image files, but no code per se. We have been distributing it this way to customer systems that do NOT have IronPython installed, or any sign of the std. library, or "os.py" specifically, and it's been working really, really well. We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that we should have to care about distributing the std. library now. The way it was before was quite simple and robust; deliver a small handful of files and it just worked, very easy to keep track of. Now instead of distributing 5 files, we suddenly need to distribute 500??? We've figured out that it certainly COULD work as described below, but it seems like a giant step backwards on several fronts, including the potential for folks to maliciously or accidentally tamper with the std lib. sources and affect the functioning of our app. So, how do we get back to the old/better functionality? On a slightly related note, our app imports some package directories in addition, e.g. "import ctypes". When python encounters a directory import, it looks for __init__.py in the directory, and derives the package import directions from there, as I understand it. However, I can't specify the "ctypes" directory as an argument to the pyc.py compile app; just causes it to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and the other files in the ctypes subdirectory, it seems like "import ctypes" would have no clue that the __init__.py that was compiled in had anything to do with the "ctypes" package, as the path names are presumably irrelevant to the compiler as long as they specify a python file. I'm considering mod'ing pyc.py to be able to incorporate a list of std lib modules to compile in: simple enough for the standalone files like "os.py", but the compile modules don't seem to be able to grok what to do with a package subdirectory. Ken You should be able to also distribute the standard library and just drop it into a Lib directory next to IronPython.dll: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... Lib\ os.py That lib dir should be on sys.path at startup and so it should be available for importing. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 11:42 AM To: Michael Foord Cc: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Michael, I started out on implementing this, but I am importing maybe a dozen of the std. library modules, which then import others, and so on. It appears that eventually, most of the std modules would have to be imported explicitly (perhaps 400 or so files) which might make for a somewhat cumbersome command line, incidentally also about 20K characters too long :-). I'm hoping to find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one of us can suggest something if we have an understanding of what you're trying to do. Ken On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: On 05/10/2010 22:27, Ken MacDonald wrote: I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like "os") are not being built into the .exe/.dll, but instead are required to be imported in source form, e.g. "os.py" must be somewhere on sys.path. In the IP 2.5 .exe's we had been building, they would run fine on machines without the IP standard library installed at all, in other words, with "os.py" not present on the machine at all. We did notice that the .exe in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 version, and the newer version requires that the source code for the IP standard library be on the path. Is this a deliberate change in behavior? We never had to package the standard library source when we sent out .exe's to customers before Hmmm... I'm pretty sure I always had to explicitly compile and bundle the standard library with previous versions of Python. Odd. Anyway, the simple solution is to ensure that you add any standard library modules you use to the set you compile and ship. All the best, Michael Foord >"os" is not an assembly but a Python module from the standard library. You need to ensure >that the Python standard library (or the parts that you use and their dependencies) is on the >path. All the best, Michael Foord and how do I ensure it gets found from my .exe - is there a specific env. variable, or the Windows %PATH% e.v., or something I haven't AddReference'd to???? Thanks, Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From calmasy at gmail.com Thu Oct 7 00:29:36 2010 From: calmasy at gmail.com (=?ISO-8859-1?Q?Count_L=E1szl=F3_de_Alm=E1sy?=) Date: Wed, 6 Oct 2010 18:29:36 -0400 Subject: [IronPython] Mono 2.8 is out Message-ID: FYI. This includes a new generational GC, the LLVM JIT, ASP.NET 4.0, a new faster thread pool, ASP.NET 4.0, ASP.NET MVC2, MEF, the DLR, etc. http://www.mono-project.com/Release_Notes_Mono_2.8 -- Cheers, L?szl? From drken567 at gmail.com Thu Oct 7 17:05:40 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Thu, 7 Oct 2010 11:05:40 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: I checked my 2.6 and 2.7 systems; neither has "os" as a builtin module, and os has a lot more members than 'nt' on both, and seem to be the same lists on both versions of IP. In one of my previous notes in this thread, I listed a 'diff' of pyc.py from the 2.6 to 2.7 versions; does anyone know why the changes were made, and, more importantly, WHAT the effect of the changes is? I tried compiling on my 2.7 system, using the pyc.py from 2.6 and the .exe coughed up an exception, but wasn't able to make much of it, but wondering if the changes in pyc.py may be affecting what the compile module outputs? Ken On Wed, Oct 6, 2010 at 5:59 PM, Dino Viehland wrote: > Hurm, will __file__ will still be set even for pre-compiled DLLs so it?s > not picking up a pre-compiled version. What about sys.builtin_module_names > on 2.6? Does it have os listed in there somehow? And does os have the same > members at the nt module or does it have significantly more? For example if > I do ?set(dir(os)) - set(dir(nt))? there?s 40 more members in os (os just > builds on top of the underlying nt/unix/other platform built-in module so if > these are getting aliased somehow then os would have the same members). > > > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 2:28 PM > > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Funny, we did exactly that yesterday. When we run the .exe on the new > version (2.7), we get a sys.path of [".", "...\current_dir\DLLs", > "...\current_dir\Lib"], and the os.__file__ was "...\current_dir\os.py" - > where I happened to have put os.py. We rebuilt our 2.6 app, and sys.path was > the same, but os.__file__ was just plain "os" - no file path listed. Almost > as if it was built into the application DLL, which is where we think it used > to be.... > Ken > > On Wed, Oct 6, 2010 at 5:15 PM, Dino Viehland wrote: > > AFAIK nothing has changed between 2.7 and 2.6 w/ how we?d go about finding > the standard library ? and we certainly never had anything which would > automatically include it. Do you have something which works on 2.6 where > you could print out os.__file__ (or some other module you?re importing from > the std lib) and sys.path? That?d at least tell us where/how we?re picking > these up from in the previous version of the app and maybe we can work out > what?s going on from there. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 1:05 PM > *To:* Discussion of IronPython > > > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Hi Dino, > > On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland wrote: > > How are you distributing your app? I?m assuming you?re going to have > something like: > > > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > > > This is exactly how we have been distributing our app up to IP 2.6 with > .NET 3.5. We also have another DLL with our resources in it; XAML, icons, > image files, but no code per se. We have been distributing it this way to > customer systems that do NOT have IronPython installed, or any sign of the > std. library, or "os.py" specifically, and it's been working really, really > well. > > We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that > we should have to care about distributing the std. library now. The way it > was before was quite simple and robust; deliver a small handful of files and > it just worked, very easy to keep track of. Now instead of distributing 5 > files, we suddenly need to distribute 500??? We've figured out that it > certainly COULD work as described below, but it seems like a giant step > backwards on several fronts, including the potential for folks to > maliciously or accidentally tamper with the std lib. sources and affect the > functioning of our app. So, how do we get back to the old/better > functionality? > > On a *slightly *related note, our app imports some package directories in > addition, e.g. "import ctypes". When python encounters a directory import, > it looks for __init__.py in the directory, and derives the package import > directions from there, as I understand it. However, I can't specify the > "ctypes" directory as an argument to the pyc.py compile app; just causes it > to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and > the other files in the ctypes subdirectory, it seems like "import ctypes" > would have no clue that the __init__.py that was compiled in had anything to > do with the "ctypes" package, as the path names are presumably irrelevant to > the compiler as long as they specify a python file. I'm considering mod'ing > pyc.py to be able to incorporate a list of std lib modules to compile in: > simple enough for the standalone files like "os.py", but the compile modules > don't seem to be able to grok what to do with a package subdirectory. > Ken > > > > > > > You should be able to also distribute the standard library and just drop it > into a Lib directory next to IronPython.dll: > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > Lib\ > > os.py > > > > That lib dir should be on sys.path at startup and so it should be available > for importing. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 11:42 AM > *To:* Michael Foord > > > *Cc:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Hi Michael, > > > I started out on implementing this, but I am importing maybe a dozen of the > std. library modules, which then import others, and so on. It appears that > eventually, most of the std modules would have to be imported explicitly > (perhaps 400 or so files) which might make for a somewhat cumbersome command > line, incidentally also about 20K characters too long :-). I'm hoping to > find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. > > Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one > of us can suggest something if we have an understanding of what you're > trying to do. > Ken > > On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: > > On 05/10/2010 22:27, Ken MacDonald wrote: > > I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net > 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like > "os") are not being built into the .exe/.dll, but instead are required to be > imported in source form, e.g. "os.py" must be somewhere on sys.path. In the > IP 2.5 .exe's we had been building, they would run fine on machines without > the IP standard library installed at all, in other words, with "os.py" not > present on the machine at all. We did notice that the .exe in question went > from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 > version, and the newer version requires that the source code for the IP > standard library be on the path. Is this a deliberate change in behavior? We > never had to package the standard library source when we sent out .exe's to > customers before > > > > Hmmm... I'm pretty sure I always had to explicitly compile and bundle the > standard library with previous versions of Python. Odd. Anyway, the simple > solution is to ensure that you add any standard library modules you use to > the set you compile and ship. > > > > All the best, > > Michael Foord > > > >"os" is not an assembly but a Python module from the standard library. You > need to ensure >that the Python standard library (or the parts that you use > and their dependencies) is on the >path. > > > All the best, > > Michael Foord > > and how do I ensure it gets found from my .exe - is there a specific env. > variable, or the Windows %PATH% e.v., or something I haven't AddReference'd > to???? > Thanks, > Ken > > > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Fri Oct 8 02:06:28 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 8 Oct 2010 00:06:28 +0000 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: I think these changes came about due to this thread: http://www.mail-archive.com/users at lists.ironpython.com/msg08794.html where there was an issue w/ relative paths and starting an app. And you may have found the root of the problem but you didn't quote the code change. There's these additional lines: gen.EmitCall(OpCodes.Call, clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ()) gen.EmitCall(OpCodes.Call, clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ()) which might be causing the problem as we change the CWD before we really kick things off. Does replacing the set_CurrentDirectory line in pyc.py with: gen.EmitCall(OpCodes.Pop) possibly fix things for you (that'll make that a NOP but should leave all the other changes in place)? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 1:26 PM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? As an FYI, the following change was made from the IP 2.6 version of pyc.py to the IP 2.7 version (newer version shown first): < # get the ScriptCode assembly... < gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) < gen.EmitCall(OpCodes.Callvirt, clr.GetClrType(Assembly).GetMethod("get_Location"), ()) --- > # get the ScriptCode assembly... > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( (str, ) )); > gen.EmitCall(OpCodes.Call, clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); Don't know what these things do at this point, but wondering if the changes have to do with compiling the entire code tree into the application .DLL??? Ken On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald > wrote: Hi Dino, On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland > wrote: How are you distributing your app? I'm assuming you're going to have something like: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... This is exactly how we have been distributing our app up to IP 2.6 with .NET 3.5. We also have another DLL with our resources in it; XAML, icons, image files, but no code per se. We have been distributing it this way to customer systems that do NOT have IronPython installed, or any sign of the std. library, or "os.py" specifically, and it's been working really, really well. We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that we should have to care about distributing the std. library now. The way it was before was quite simple and robust; deliver a small handful of files and it just worked, very easy to keep track of. Now instead of distributing 5 files, we suddenly need to distribute 500??? We've figured out that it certainly COULD work as described below, but it seems like a giant step backwards on several fronts, including the potential for folks to maliciously or accidentally tamper with the std lib. sources and affect the functioning of our app. So, how do we get back to the old/better functionality? On a slightly related note, our app imports some package directories in addition, e.g. "import ctypes". When python encounters a directory import, it looks for __init__.py in the directory, and derives the package import directions from there, as I understand it. However, I can't specify the "ctypes" directory as an argument to the pyc.py compile app; just causes it to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and the other files in the ctypes subdirectory, it seems like "import ctypes" would have no clue that the __init__.py that was compiled in had anything to do with the "ctypes" package, as the path names are presumably irrelevant to the compiler as long as they specify a python file. I'm considering mod'ing pyc.py to be able to incorporate a list of std lib modules to compile in: simple enough for the standalone files like "os.py", but the compile modules don't seem to be able to grok what to do with a package subdirectory. Ken You should be able to also distribute the standard library and just drop it into a Lib directory next to IronPython.dll: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... Lib\ os.py That lib dir should be on sys.path at startup and so it should be available for importing. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 11:42 AM To: Michael Foord Cc: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Michael, I started out on implementing this, but I am importing maybe a dozen of the std. library modules, which then import others, and so on. It appears that eventually, most of the std modules would have to be imported explicitly (perhaps 400 or so files) which might make for a somewhat cumbersome command line, incidentally also about 20K characters too long :-). I'm hoping to find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one of us can suggest something if we have an understanding of what you're trying to do. Ken On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: On 05/10/2010 22:27, Ken MacDonald wrote: I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like "os") are not being built into the .exe/.dll, but instead are required to be imported in source form, e.g. "os.py" must be somewhere on sys.path. In the IP 2.5 .exe's we had been building, they would run fine on machines without the IP standard library installed at all, in other words, with "os.py" not present on the machine at all. We did notice that the .exe in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 version, and the newer version requires that the source code for the IP standard library be on the path. Is this a deliberate change in behavior? We never had to package the standard library source when we sent out .exe's to customers before Hmmm... I'm pretty sure I always had to explicitly compile and bundle the standard library with previous versions of Python. Odd. Anyway, the simple solution is to ensure that you add any standard library modules you use to the set you compile and ship. All the best, Michael Foord >"os" is not an assembly but a Python module from the standard library. You need to ensure >that the Python standard library (or the parts that you use and their dependencies) is on the >path. All the best, Michael Foord and how do I ensure it gets found from my .exe - is there a specific env. variable, or the Windows %PATH% e.v., or something I haven't AddReference'd to???? Thanks, Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Fri Oct 8 16:08:37 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Fri, 8 Oct 2010 10:08:37 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: HI Dino, Tried your change, it came up with an error on the statement below: TypeError: EmitCall() takes exactly 3 arguments (1 given) Not sure what the other args should be... it did seem to create a new myapp.dll despite the error, but it was the same size (1.2 MB) as the new version 2.7 dll, and was still missing std lib components. I'm not sure that this is valid, considering the error in the EmitCall() noted above. I did do a primitive analysis of the app to find out which std lib components it references; found roughly 25 of them, and as a test, incorporated the largest 10 or so explicitly in the compile; the size of the resulting dll was about 2.1 MB (much closer to the 2.6 dll version size, 2.9 MB), and was able to run it successfullyagainst a stripped-down std lib where I had removed those 10 files. It seems fairly clear to me that the 2.6 version dll did include the components it required from the std lib. Starting to look really promising; not quite there yet! Ken On Thu, Oct 7, 2010 at 8:06 PM, Dino Viehland wrote: > I think these changes came about due to this thread: > http://www.mail-archive.com/users at lists.ironpython.com/msg08794.html where > there was an issue w/ relative paths and starting an app. > > > > And you may have found the root of the problem but you didn?t quote the > code change. There?s these additional lines: > > > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ()) > gen.EmitCall(OpCodes.Call, > clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ()) > > > > which might be causing the problem as we change the CWD before we really > kick things off. Does replacing the set_CurrentDirectory line in pyc.py > with: > > > > gen.EmitCall(OpCodes.Pop) > > > > possibly fix things for you (that?ll make that a NOP but should leave all > the other changes in place)? > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 1:26 PM > *To:* Discussion of IronPython > > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > As an FYI, the following change was made from the IP 2.6 version of pyc.py > to the IP 2.7 version (newer version shown first): > > < # get the ScriptCode assembly... > < gen.EmitCall(OpCodes.Call, > clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) > < gen.EmitCall(OpCodes.Callvirt, > clr.GetClrType(Assembly).GetMethod("get_Location"), ()) > --- > > # get the ScriptCode assembly... > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); > > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( > (str, ) )); > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); > > Don't know what these things do at this point, but wondering if the changes > have to do with compiling the entire code tree into the application .DLL??? > Ken > > On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald wrote: > > Hi Dino, > > On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland wrote: > > How are you distributing your app? I?m assuming you?re going to have > something like: > > > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > > > This is exactly how we have been distributing our app up to IP 2.6 with > .NET 3.5. We also have another DLL with our resources in it; XAML, icons, > image files, but no code per se. We have been distributing it this way to > customer systems that do NOT have IronPython installed, or any sign of the > std. library, or "os.py" specifically, and it's been working really, really > well. > > We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that > we should have to care about distributing the std. library now. The way it > was before was quite simple and robust; deliver a small handful of files and > it just worked, very easy to keep track of. Now instead of distributing 5 > files, we suddenly need to distribute 500??? We've figured out that it > certainly COULD work as described below, but it seems like a giant step > backwards on several fronts, including the potential for folks to > maliciously or accidentally tamper with the std lib. sources and affect the > functioning of our app. So, how do we get back to the old/better > functionality? > > On a *slightly *related note, our app imports some package directories in > addition, e.g. "import ctypes". When python encounters a directory import, > it looks for __init__.py in the directory, and derives the package import > directions from there, as I understand it. However, I can't specify the > "ctypes" directory as an argument to the pyc.py compile app; just causes it > to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and > the other files in the ctypes subdirectory, it seems like "import ctypes" > would have no clue that the __init__.py that was compiled in had anything to > do with the "ctypes" package, as the path names are presumably irrelevant to > the compiler as long as they specify a python file. I'm considering mod'ing > pyc.py to be able to incorporate a list of std lib modules to compile in: > simple enough for the standalone files like "os.py", but the compile modules > don't seem to be able to grok what to do with a package subdirectory. > Ken > > > > > > > You should be able to also distribute the standard library and just drop it > into a Lib directory next to IronPython.dll: > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > Lib\ > > os.py > > > > That lib dir should be on sys.path at startup and so it should be available > for importing. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 11:42 AM > *To:* Michael Foord > > > *Cc:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Hi Michael, > > > I started out on implementing this, but I am importing maybe a dozen of the > std. library modules, which then import others, and so on. It appears that > eventually, most of the std modules would have to be imported explicitly > (perhaps 400 or so files) which might make for a somewhat cumbersome command > line, incidentally also about 20K characters too long :-). I'm hoping to > find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. > > Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one > of us can suggest something if we have an understanding of what you're > trying to do. > Ken > > On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: > > On 05/10/2010 22:27, Ken MacDonald wrote: > > I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net > 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like > "os") are not being built into the .exe/.dll, but instead are required to be > imported in source form, e.g. "os.py" must be somewhere on sys.path. In the > IP 2.5 .exe's we had been building, they would run fine on machines without > the IP standard library installed at all, in other words, with "os.py" not > present on the machine at all. We did notice that the .exe in question went > from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 > version, and the newer version requires that the source code for the IP > standard library be on the path. Is this a deliberate change in behavior? We > never had to package the standard library source when we sent out .exe's to > customers before > > > > Hmmm... I'm pretty sure I always had to explicitly compile and bundle the > standard library with previous versions of Python. Odd. Anyway, the simple > solution is to ensure that you add any standard library modules you use to > the set you compile and ship. > > > > All the best, > > Michael Foord > > > > >"os" is not an assembly but a Python module from the standard library. You > need to ensure >that the Python standard library (or the parts that you use > and their dependencies) is on the >path. > > > All the best, > > Michael Foord > > and how do I ensure it gets found from my .exe - is there a specific env. > variable, or the Windows %PATH% e.v., or something I haven't AddReference'd > to???? > Thanks, > Ken > > > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Fri Oct 8 16:28:23 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Fri, 8 Oct 2010 10:28:23 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Well, just was looking at the pyc.py again; the change you suggest is actually in the GenerateExe method, so should have no effect on what is contained in the .dll. The .dll is actually generated by this code: print "Output:\n\t%s" % output print "Target:\n\t%s" % target print 'Platform:\n\t%s' % platform print 'Machine:\n\t%s' % machine print 'Compiling...' clr.CompileModules(output + '.dll', mainModule = main_name, *files) so I'm getting the feeling that something changed in the CompileModules method between 2.6 and 2.7 such that it's no longer tracking down and including the std lib references. Ken On Fri, Oct 8, 2010 at 10:08 AM, Ken MacDonald wrote: > HI Dino, > Tried your change, it came up with an error on the statement below: > TypeError: EmitCall() takes exactly 3 arguments (1 given) > > Not sure what the other args should be... it did seem to create a new > myapp.dll despite the error, but it was the same size (1.2 MB) as the new > version 2.7 dll, and was still missing std lib components. I'm not sure that > this is valid, considering the error in the EmitCall() noted above. > > I did do a primitive analysis of the app to find out which std lib > components it references; found roughly 25 of them, and as a test, > incorporated the largest 10 or so explicitly in the compile; the size of the > resulting dll was about 2.1 MB (much closer to the 2.6 dll version size, 2.9 > MB), and was able to run it successfullyagainst a stripped-down std lib > where I had removed those 10 files. It seems fairly clear to me that the 2.6 > version dll did include the components it required from the std lib. > > Starting to look really promising; not quite there yet! > Ken > > > On Thu, Oct 7, 2010 at 8:06 PM, Dino Viehland wrote: > >> I think these changes came about due to this thread: >> http://www.mail-archive.com/users at lists.ironpython.com/msg08794.htmlwhere there was an issue w/ relative paths and starting an app. >> >> >> >> And you may have found the root of the problem but you didn?t quote the >> code change. There?s these additional lines: >> >> >> >> gen.EmitCall(OpCodes.Call, >> clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ()) >> gen.EmitCall(OpCodes.Call, >> clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ()) >> >> >> >> which might be causing the problem as we change the CWD before we really >> kick things off. Does replacing the set_CurrentDirectory line in pyc.py >> with: >> >> >> >> gen.EmitCall(OpCodes.Pop) >> >> >> >> possibly fix things for you (that?ll make that a NOP but should leave all >> the other changes in place)? >> >> >> >> *From:* users-bounces at lists.ironpython.com [mailto: >> users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald >> *Sent:* Wednesday, October 06, 2010 1:26 PM >> *To:* Discussion of IronPython >> >> *Subject:* Re: [IronPython] change in standard library behavior for >> compiled .exe/.dll??? >> >> >> >> As an FYI, the following change was made from the IP 2.6 version of pyc.py >> to the IP 2.7 version (newer version shown first): >> >> < # get the ScriptCode assembly... >> < gen.EmitCall(OpCodes.Call, >> clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) >> < gen.EmitCall(OpCodes.Callvirt, >> clr.GetClrType(Assembly).GetMethod("get_Location"), ()) >> --- >> > # get the ScriptCode assembly... >> > gen.EmitCall(OpCodes.Call, >> clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); >> > gen.EmitCall(OpCodes.Call, >> clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); >> > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( >> (str, ) )); >> > gen.EmitCall(OpCodes.Call, >> clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); >> >> Don't know what these things do at this point, but wondering if the >> changes have to do with compiling the entire code tree into the application >> .DLL??? >> Ken >> >> On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald wrote: >> >> Hi Dino, >> >> On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland >> wrote: >> >> How are you distributing your app? I?m assuming you?re going to have >> something like: >> >> >> >> MyApp\ >> >> MyApp.exe >> >> MyApp.dll >> >> IronPython.dll >> >> IronPython.Modules.DLL >> >> ? >> >> >> >> This is exactly how we have been distributing our app up to IP 2.6 with >> .NET 3.5. We also have another DLL with our resources in it; XAML, icons, >> image files, but no code per se. We have been distributing it this way to >> customer systems that do NOT have IronPython installed, or any sign of the >> std. library, or "os.py" specifically, and it's been working really, really >> well. >> >> We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that >> we should have to care about distributing the std. library now. The way it >> was before was quite simple and robust; deliver a small handful of files and >> it just worked, very easy to keep track of. Now instead of distributing 5 >> files, we suddenly need to distribute 500??? We've figured out that it >> certainly COULD work as described below, but it seems like a giant step >> backwards on several fronts, including the potential for folks to >> maliciously or accidentally tamper with the std lib. sources and affect the >> functioning of our app. So, how do we get back to the old/better >> functionality? >> >> On a *slightly *related note, our app imports some package directories in >> addition, e.g. "import ctypes". When python encounters a directory import, >> it looks for __init__.py in the directory, and derives the package import >> directions from there, as I understand it. However, I can't specify the >> "ctypes" directory as an argument to the pyc.py compile app; just causes it >> to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and >> the other files in the ctypes subdirectory, it seems like "import ctypes" >> would have no clue that the __init__.py that was compiled in had anything to >> do with the "ctypes" package, as the path names are presumably irrelevant to >> the compiler as long as they specify a python file. I'm considering mod'ing >> pyc.py to be able to incorporate a list of std lib modules to compile in: >> simple enough for the standalone files like "os.py", but the compile modules >> don't seem to be able to grok what to do with a package subdirectory. >> Ken >> >> >> >> >> >> >> You should be able to also distribute the standard library and just drop >> it into a Lib directory next to IronPython.dll: >> >> MyApp\ >> >> MyApp.exe >> >> MyApp.dll >> >> IronPython.dll >> >> IronPython.Modules.DLL >> >> ? >> >> Lib\ >> >> os.py >> >> >> >> That lib dir should be on sys.path at startup and so it should be >> available for importing. >> >> >> >> *From:* users-bounces at lists.ironpython.com [mailto: >> users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald >> *Sent:* Wednesday, October 06, 2010 11:42 AM >> *To:* Michael Foord >> >> >> *Cc:* Discussion of IronPython >> *Subject:* Re: [IronPython] change in standard library behavior for >> compiled .exe/.dll??? >> >> >> >> Hi Michael, >> >> >> I started out on implementing this, but I am importing maybe a dozen of >> the std. library modules, which then import others, and so on. It appears >> that eventually, most of the std modules would have to be imported >> explicitly (perhaps 400 or so files) which might make for a somewhat >> cumbersome command line, incidentally also about 20K characters too long >> :-). I'm hoping to find a way to get this to work as well as it did under IP >> 2.5 / .NET 3.5. >> >> Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe >> one of us can suggest something if we have an understanding of what you're >> trying to do. >> Ken >> >> On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord >> wrote: >> >> On 05/10/2010 22:27, Ken MacDonald wrote: >> >> I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net >> 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like >> "os") are not being built into the .exe/.dll, but instead are required to be >> imported in source form, e.g. "os.py" must be somewhere on sys.path. In the >> IP 2.5 .exe's we had been building, they would run fine on machines without >> the IP standard library installed at all, in other words, with "os.py" not >> present on the machine at all. We did notice that the .exe in question went >> from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 >> version, and the newer version requires that the source code for the IP >> standard library be on the path. Is this a deliberate change in behavior? We >> never had to package the standard library source when we sent out .exe's to >> customers before >> >> >> >> Hmmm... I'm pretty sure I always had to explicitly compile and bundle the >> standard library with previous versions of Python. Odd. Anyway, the simple >> solution is to ensure that you add any standard library modules you use to >> the set you compile and ship. >> >> >> >> All the best, >> >> Michael Foord >> >> >> >> >"os" is not an assembly but a Python module from the standard library. >> You need to ensure >that the Python standard library (or the parts that you >> use and their dependencies) is on the >path. >> >> >> All the best, >> >> Michael Foord >> >> and how do I ensure it gets found from my .exe - is there a specific >> env. variable, or the Windows %PATH% e.v., or something I haven't >> AddReference'd to???? >> Thanks, >> Ken >> >> >> >> _______________________________________________ >> >> Users mailing list >> >> Users at lists.ironpython.com >> >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> >> >> -- >> >> http://www.voidspace.org.uk/blog >> >> >> >> READ CAREFULLY. By accepting and reading this email you agree, >> >> on behalf of your employer, to release me from all obligations >> >> and waivers arising from any and all NON-NEGOTIATED agreements, >> >> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, >> >> confidentiality, non-disclosure, non-compete and acceptable use >> >> policies (?BOGUS AGREEMENTS?) that I have entered into with your >> >> employer, its partners, licensors, agents and assigns, in >> >> perpetuity, without prejudice to my ongoing rights and privileges. >> >> You further represent that you have the authority to release me >> >> from any BOGUS AGREEMENTS on behalf of your employer. >> >> >> >> >> >> -- >> >> http://www.voidspace.org.uk/blog >> >> >> >> READ CAREFULLY. By accepting and reading this email you agree, >> >> on behalf of your employer, to release me from all obligations >> >> and waivers arising from any and all NON-NEGOTIATED agreements, >> >> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, >> >> confidentiality, non-disclosure, non-compete and acceptable use >> >> policies (?BOGUS AGREEMENTS?) that I have entered into with your >> >> employer, its partners, licensors, agents and assigns, in >> >> perpetuity, without prejudice to my ongoing rights and privileges. >> >> You further represent that you have the authority to release me >> >> from any BOGUS AGREEMENTS on behalf of your employer. >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Fri Oct 8 20:15:08 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 8 Oct 2010 18:15:08 +0000 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Sorry, I think that should have been: gen.Emit(OpCodes.Pop) One way to know for sure if the DLL includes the pre-compiled std lib would be to run ildasm or reflector on it. There'll be a class called DLRCachedCode and in it there will (or won't) be a method called "os$#" where # is some number. Opening that method you'll see an attribute such as: .custom instance void [Microsoft.Dynamic]Microsoft.Scripting.Runtime.CachedOptimizedCodeAttribute::.ctor(string[]) = ( 01 00 07 00 00 00 08 5F 5F 6E 61 6D 65 5F 5F 08 // .......__name__. 5F 5F 66 69 6C 65 5F 5F 07 5F 5F 64 6F 63 5F 5F // __file__.__doc__ 08 5F 5F 70 61 74 68 5F 5F 0C 5F 5F 62 75 69 6C // .__path__.__buil 74 69 6E 73 5F 5F 0B 5F 5F 70 61 63 6B 61 67 65 // tins__.__package 5F 5F 02 6F 73 00 00 ) // __.os.. // Code size 223 (0xdf) The important part here is that the method name is os$ and the attribute is persent - the contents of the attribute here are just things which are referred to from the code so that part doesn't really matter. I'm absolutely certain though that there was never a feature to automatically figure out which parts of the std lib should be included though. And if I take a simple file lke: import os print os.__path__ Compile it like so: "C:\Program Files\IronPython 2.6\ipy.exe" "C:\Program Files\IronPython 2.6\Tools\Scripts\pyc.py" test.py /target:exe /main:test.py And run it we can't find os.py: C:\Users\Dino>.\test Unhandled Exception: IronPython.Runtime.Exceptions.ImportException: No module named os at Microsoft.Scripting.Actions.Calls.MethodCandidate.Caller.Call(Object[] args, Boolean& shouldOptimize) at IronPython.Runtime.Types.BuiltinFunction.BuiltinFunctionCaller`6.Call5(CallSite site, CodeContext context, TFuncType func, T0 arg0, T1 arg1, T2 arg2, T3 arg3, T4 arg4) at System.Dynamic.UpdateDelegates.UpdateAndExecute7[T0,T1,T2,T3,T4,T5,T6,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2, T3 arg3, T4 arg4, T5 arg5, T6 arg6) at IronPython.Runtime.Importer.Import(CodeContext context, String fullName, PythonTuple from, Int32 level) at IronPython.Runtime.Operations.PythonOps.InitializeModule(Assembly precompiled, String main, String[] references) at PythonMain.Main() From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Friday, October 08, 2010 7:09 AM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? HI Dino, Tried your change, it came up with an error on the statement below: TypeError: EmitCall() takes exactly 3 arguments (1 given) Not sure what the other args should be... it did seem to create a new myapp.dll despite the error, but it was the same size (1.2 MB) as the new version 2.7 dll, and was still missing std lib components. I'm not sure that this is valid, considering the error in the EmitCall() noted above. I did do a primitive analysis of the app to find out which std lib components it references; found roughly 25 of them, and as a test, incorporated the largest 10 or so explicitly in the compile; the size of the resulting dll was about 2.1 MB (much closer to the 2.6 dll version size, 2.9 MB), and was able to run it successfullyagainst a stripped-down std lib where I had removed those 10 files. It seems fairly clear to me that the 2.6 version dll did include the components it required from the std lib. Starting to look really promising; not quite there yet! Ken On Thu, Oct 7, 2010 at 8:06 PM, Dino Viehland > wrote: I think these changes came about due to this thread: http://www.mail-archive.com/users at lists.ironpython.com/msg08794.html where there was an issue w/ relative paths and starting an app. And you may have found the root of the problem but you didn't quote the code change. There's these additional lines: gen.EmitCall(OpCodes.Call, clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ()) gen.EmitCall(OpCodes.Call, clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ()) which might be causing the problem as we change the CWD before we really kick things off. Does replacing the set_CurrentDirectory line in pyc.py with: gen.EmitCall(OpCodes.Pop) possibly fix things for you (that'll make that a NOP but should leave all the other changes in place)? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 1:26 PM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? As an FYI, the following change was made from the IP 2.6 version of pyc.py to the IP 2.7 version (newer version shown first): < # get the ScriptCode assembly... < gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) < gen.EmitCall(OpCodes.Callvirt, clr.GetClrType(Assembly).GetMethod("get_Location"), ()) --- > # get the ScriptCode assembly... > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( (str, ) )); > gen.EmitCall(OpCodes.Call, clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); Don't know what these things do at this point, but wondering if the changes have to do with compiling the entire code tree into the application .DLL??? Ken On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald > wrote: Hi Dino, On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland > wrote: How are you distributing your app? I'm assuming you're going to have something like: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... This is exactly how we have been distributing our app up to IP 2.6 with .NET 3.5. We also have another DLL with our resources in it; XAML, icons, image files, but no code per se. We have been distributing it this way to customer systems that do NOT have IronPython installed, or any sign of the std. library, or "os.py" specifically, and it's been working really, really well. We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that we should have to care about distributing the std. library now. The way it was before was quite simple and robust; deliver a small handful of files and it just worked, very easy to keep track of. Now instead of distributing 5 files, we suddenly need to distribute 500??? We've figured out that it certainly COULD work as described below, but it seems like a giant step backwards on several fronts, including the potential for folks to maliciously or accidentally tamper with the std lib. sources and affect the functioning of our app. So, how do we get back to the old/better functionality? On a slightly related note, our app imports some package directories in addition, e.g. "import ctypes". When python encounters a directory import, it looks for __init__.py in the directory, and derives the package import directions from there, as I understand it. However, I can't specify the "ctypes" directory as an argument to the pyc.py compile app; just causes it to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and the other files in the ctypes subdirectory, it seems like "import ctypes" would have no clue that the __init__.py that was compiled in had anything to do with the "ctypes" package, as the path names are presumably irrelevant to the compiler as long as they specify a python file. I'm considering mod'ing pyc.py to be able to incorporate a list of std lib modules to compile in: simple enough for the standalone files like "os.py", but the compile modules don't seem to be able to grok what to do with a package subdirectory. Ken You should be able to also distribute the standard library and just drop it into a Lib directory next to IronPython.dll: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... Lib\ os.py That lib dir should be on sys.path at startup and so it should be available for importing. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 11:42 AM To: Michael Foord Cc: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Michael, I started out on implementing this, but I am importing maybe a dozen of the std. library modules, which then import others, and so on. It appears that eventually, most of the std modules would have to be imported explicitly (perhaps 400 or so files) which might make for a somewhat cumbersome command line, incidentally also about 20K characters too long :-). I'm hoping to find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one of us can suggest something if we have an understanding of what you're trying to do. Ken On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: On 05/10/2010 22:27, Ken MacDonald wrote: I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like "os") are not being built into the .exe/.dll, but instead are required to be imported in source form, e.g. "os.py" must be somewhere on sys.path. In the IP 2.5 .exe's we had been building, they would run fine on machines without the IP standard library installed at all, in other words, with "os.py" not present on the machine at all. We did notice that the .exe in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 version, and the newer version requires that the source code for the IP standard library be on the path. Is this a deliberate change in behavior? We never had to package the standard library source when we sent out .exe's to customers before Hmmm... I'm pretty sure I always had to explicitly compile and bundle the standard library with previous versions of Python. Odd. Anyway, the simple solution is to ensure that you add any standard library modules you use to the set you compile and ship. All the best, Michael Foord >"os" is not an assembly but a Python module from the standard library. You need to ensure >that the Python standard library (or the parts that you use and their dependencies) is on the >path. All the best, Michael Foord and how do I ensure it gets found from my .exe - is there a specific env. variable, or the Windows %PATH% e.v., or something I haven't AddReference'd to???? Thanks, Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Fri Oct 8 20:16:50 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 8 Oct 2010 18:16:50 +0000 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Are you not creating an EXE? My theory here is that changing the CWD is causing us to not pick up the std lib because I don't think the DLL has ever included the std lib. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Friday, October 08, 2010 7:28 AM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Well, just was looking at the pyc.py again; the change you suggest is actually in the GenerateExe method, so should have no effect on what is contained in the .dll. The .dll is actually generated by this code: print "Output:\n\t%s" % output print "Target:\n\t%s" % target print 'Platform:\n\t%s' % platform print 'Machine:\n\t%s' % machine print 'Compiling...' clr.CompileModules(output + '.dll', mainModule = main_name, *files) so I'm getting the feeling that something changed in the CompileModules method between 2.6 and 2.7 such that it's no longer tracking down and including the std lib references. Ken On Fri, Oct 8, 2010 at 10:08 AM, Ken MacDonald > wrote: HI Dino, Tried your change, it came up with an error on the statement below: TypeError: EmitCall() takes exactly 3 arguments (1 given) Not sure what the other args should be... it did seem to create a new myapp.dll despite the error, but it was the same size (1.2 MB) as the new version 2.7 dll, and was still missing std lib components. I'm not sure that this is valid, considering the error in the EmitCall() noted above. I did do a primitive analysis of the app to find out which std lib components it references; found roughly 25 of them, and as a test, incorporated the largest 10 or so explicitly in the compile; the size of the resulting dll was about 2.1 MB (much closer to the 2.6 dll version size, 2.9 MB), and was able to run it successfullyagainst a stripped-down std lib where I had removed those 10 files. It seems fairly clear to me that the 2.6 version dll did include the components it required from the std lib. Starting to look really promising; not quite there yet! Ken On Thu, Oct 7, 2010 at 8:06 PM, Dino Viehland > wrote: I think these changes came about due to this thread: http://www.mail-archive.com/users at lists.ironpython.com/msg08794.html where there was an issue w/ relative paths and starting an app. And you may have found the root of the problem but you didn't quote the code change. There's these additional lines: gen.EmitCall(OpCodes.Call, clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ()) gen.EmitCall(OpCodes.Call, clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ()) which might be causing the problem as we change the CWD before we really kick things off. Does replacing the set_CurrentDirectory line in pyc.py with: gen.EmitCall(OpCodes.Pop) possibly fix things for you (that'll make that a NOP but should leave all the other changes in place)? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 1:26 PM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? As an FYI, the following change was made from the IP 2.6 version of pyc.py to the IP 2.7 version (newer version shown first): < # get the ScriptCode assembly... < gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) < gen.EmitCall(OpCodes.Callvirt, clr.GetClrType(Assembly).GetMethod("get_Location"), ()) --- > # get the ScriptCode assembly... > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( (str, ) )); > gen.EmitCall(OpCodes.Call, clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); Don't know what these things do at this point, but wondering if the changes have to do with compiling the entire code tree into the application .DLL??? Ken On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald > wrote: Hi Dino, On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland > wrote: How are you distributing your app? I'm assuming you're going to have something like: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... This is exactly how we have been distributing our app up to IP 2.6 with .NET 3.5. We also have another DLL with our resources in it; XAML, icons, image files, but no code per se. We have been distributing it this way to customer systems that do NOT have IronPython installed, or any sign of the std. library, or "os.py" specifically, and it's been working really, really well. We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that we should have to care about distributing the std. library now. The way it was before was quite simple and robust; deliver a small handful of files and it just worked, very easy to keep track of. Now instead of distributing 5 files, we suddenly need to distribute 500??? We've figured out that it certainly COULD work as described below, but it seems like a giant step backwards on several fronts, including the potential for folks to maliciously or accidentally tamper with the std lib. sources and affect the functioning of our app. So, how do we get back to the old/better functionality? On a slightly related note, our app imports some package directories in addition, e.g. "import ctypes". When python encounters a directory import, it looks for __init__.py in the directory, and derives the package import directions from there, as I understand it. However, I can't specify the "ctypes" directory as an argument to the pyc.py compile app; just causes it to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and the other files in the ctypes subdirectory, it seems like "import ctypes" would have no clue that the __init__.py that was compiled in had anything to do with the "ctypes" package, as the path names are presumably irrelevant to the compiler as long as they specify a python file. I'm considering mod'ing pyc.py to be able to incorporate a list of std lib modules to compile in: simple enough for the standalone files like "os.py", but the compile modules don't seem to be able to grok what to do with a package subdirectory. Ken You should be able to also distribute the standard library and just drop it into a Lib directory next to IronPython.dll: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... Lib\ os.py That lib dir should be on sys.path at startup and so it should be available for importing. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 11:42 AM To: Michael Foord Cc: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Michael, I started out on implementing this, but I am importing maybe a dozen of the std. library modules, which then import others, and so on. It appears that eventually, most of the std modules would have to be imported explicitly (perhaps 400 or so files) which might make for a somewhat cumbersome command line, incidentally also about 20K characters too long :-). I'm hoping to find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one of us can suggest something if we have an understanding of what you're trying to do. Ken On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: On 05/10/2010 22:27, Ken MacDonald wrote: I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like "os") are not being built into the .exe/.dll, but instead are required to be imported in source form, e.g. "os.py" must be somewhere on sys.path. In the IP 2.5 .exe's we had been building, they would run fine on machines without the IP standard library installed at all, in other words, with "os.py" not present on the machine at all. We did notice that the .exe in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 version, and the newer version requires that the source code for the IP standard library be on the path. Is this a deliberate change in behavior? We never had to package the standard library source when we sent out .exe's to customers before Hmmm... I'm pretty sure I always had to explicitly compile and bundle the standard library with previous versions of Python. Odd. Anyway, the simple solution is to ensure that you add any standard library modules you use to the set you compile and ship. All the best, Michael Foord >"os" is not an assembly but a Python module from the standard library. You need to ensure >that the Python standard library (or the parts that you use and their dependencies) is on the >path. All the best, Michael Foord and how do I ensure it gets found from my .exe - is there a specific env. variable, or the Windows %PATH% e.v., or something I haven't AddReference'd to???? Thanks, Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Fri Oct 8 20:51:35 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Fri, 8 Oct 2010 14:51:35 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Hi Dino, Here's a portion of the the os$31 output from the .dll created with IP 2.6, as dumped by ildasm (that's a cool tool!). The .dll we get with IP 2.7 (without explicitly adding in the std lib references on the compile command line) does NOT contain os$#.anything.... Ken .method public specialname static object os$31(class [IronPython]IronPython.Runtime.CodeContext $globalContext, class [IronPython]IronPython.Runtime.FunctionCode functionCode) cil managed { .custom instance void [Microsoft.Dynamic]Microsoft.Scripting.Runtime.CachedOptimizedCodeAttribute::.ctor(string[]) = ( 01 00 69 00 00 00 08 5F 5F 6E 61 6D 65 5F 5F 08 // ..i....__name__. 5F 5F 66 69 6C 65 5F 5F 07 5F 5F 64 6F 63 5F 5F // __file__.__doc__ 08 5F 5F 70 61 74 68 5F 5F 0C 5F 5F 62 75 69 6C // .__path__.__buil 74 69 6E 73 5F 5F 0B 5F 5F 70 61 63 6B 61 67 65 // tins__.__package 5F 5F 03 73 79 73 05 65 72 72 6E 6F 06 5F 6E 61 // __.sys.errno._na 6D 65 73 07 5F 5F 61 6C 6C 5F 5F 11 5F 67 65 74 // mes.__all__._get 5F 65 78 70 6F 72 74 73 5F 6C 69 73 74 04 6E 61 // _exports_list.na 6D 65 07 6C 69 6E 65 73 65 70 05 5F 65 78 69 74 // me.linesep._exit 04 70 61 74 68 05 70 6F 73 69 78 02 6E 74 04 6C // .path.posix.nt.l 69 6E 6B 03 6F 73 32 02 63 65 06 72 69 73 63 6F // ink.os2.ce.risco 73 06 63 75 72 64 69 72 06 70 61 72 64 69 72 03 // s.curdir.pardir. 73 65 70 07 70 61 74 68 73 65 70 07 64 65 66 70 // sep.pathsep.defp 61 74 68 06 65 78 74 73 65 70 06 61 6C 74 73 65 // ath.extsep.altse 70 07 64 65 76 6E 75 6C 6C 08 53 45 45 4B 5F 53 // p.devnull.SEEK_S 45 54 08 53 45 45 4B 5F 43 55 52 08 53 45 45 4B // ET.SEEK_CUR.SEEK 5F 45 4E 44 08 6D 61 6B 65 64 69 72 73 0A 72 65 // _END.makedirs.re 6D 6F 76 65 64 69 72 73 07 72 65 6E 61 6D 65 73 // movedirs.renames 04 77 61 6C 6B 07 65 6E 76 69 72 6F 6E 05 65 78 // .walk.environ.ex 65 63 6C 06 65 78 65 63 6C 65 06 65 78 65 63 6C // ecl.execle.execl 70 07 65 78 65 63 6C 70 65 06 65 78 65 63 76 70 // p.execlpe.execvp 07 65 78 65 63 76 70 65 08 5F 65 78 65 63 76 70 // .execvpe._execvp 65 08 55 73 65 72 44 69 63 74 08 75 6E 73 65 74 // e.UserDict.unset 65 6E 76 08 5F 45 6E 76 69 72 6F 6E 06 67 65 74 // env._Environ.get 65 6E 76 07 5F 65 78 69 73 74 73 06 50 5F 57 41 // env._exists.P_WA 49 54 08 50 5F 4E 4F 57 41 49 54 09 50 5F 4E 4F // IT.P_NOWAIT.P_NO 57 41 49 54 4F 09 5F 73 70 61 77 6E 76 65 66 06 // WAITO._spawnvef. 73 70 61 77 6E 76 07 73 70 61 77 6E 76 65 07 73 // spawnv.spawnve.s 70 61 77 6E 76 70 08 73 70 61 77 6E 76 70 65 06 // pawnvp.spawnvpe. 73 70 61 77 6E 6C 07 73 70 61 77 6E 6C 65 07 73 // spawnl.spawnle.s 70 61 77 6E 6C 70 08 73 70 61 77 6E 6C 70 65 06 // pawnlp.spawnlpe. 70 6F 70 65 6E 32 06 70 6F 70 65 6E 33 06 70 6F // popen2.popen3.po 70 65 6E 34 09 5F 63 6F 70 79 5F 72 65 67 11 5F // pen4._copy_reg._ 6D 61 6B 65 5F 73 74 61 74 5F 72 65 73 75 6C 74 // make_stat_result 13 5F 70 69 63 6B 6C 65 5F 73 74 61 74 5F 72 65 // ._pickle_stat_re 73 75 6C 74 14 5F 6D 61 6B 65 5F 73 74 61 74 76 // sult._make_statv 66 73 5F 72 65 73 75 6C 74 16 5F 70 69 63 6B 6C // fs_result._pickl 65 5F 73 74 61 74 76 66 73 5F 72 65 73 75 6C 74 // e_statvfs_result 07 75 72 61 6E 64 6F 6D 04 6C 69 73 74 0E 41 74 // .urandom.list.At 74 72 69 62 75 74 65 45 72 72 6F 72 03 64 69 72 // tributeError.dir 07 4F 53 45 72 72 6F 72 05 6D 6B 64 69 72 05 72 // .OSError.mkdir.r 6D 64 69 72 05 65 72 72 6F 72 06 72 65 6E 61 6D // mdir.error.renam 65 07 6C 69 73 74 64 69 72 05 65 78 65 63 76 06 // e.listdir.execv. 65 78 65 63 76 65 06 70 75 74 65 6E 76 04 64 69 // execve.putenv.di 63 74 04 65 76 61 6C 04 54 72 75 65 09 4E 61 6D // ct.eval.True.Nam 65 45 72 72 6F 72 05 46 61 6C 73 65 04 66 6F 72 // eError.False.for 6B 07 77 61 69 74 70 69 64 0A 57 49 46 53 54 4F // k.waitpid.WIFSTO 50 50 45 44 0B 57 49 46 53 49 47 4E 41 4C 45 44 // PPED.WIFSIGNALED 08 57 54 45 52 4D 53 49 47 09 57 49 46 45 58 49 // .WTERMSIG.WIFEXI 54 45 44 0B 57 45 58 49 54 53 54 41 54 55 53 12 // TED.WEXITSTATUS. 44 65 70 72 65 63 61 74 69 6F 6E 57 61 72 6E 69 // DeprecationWarni 6E 67 0B 73 74 61 74 5F 72 65 73 75 6C 74 0E 73 // ng.stat_result.s 74 61 74 76 66 73 5F 72 65 73 75 6C 74 04 6F 70 // tatvfs_result.op 65 6E 08 4F 5F 52 44 4F 4E 4C 59 07 49 4F 45 72 // en.O_RDONLY.IOEr 72 6F 72 13 4E 6F 74 49 6D 70 6C 65 6D 65 6E 74 // ror.NotImplement 65 64 45 72 72 6F 72 03 6C 65 6E 04 72 65 61 64 // edError.len.read 05 63 6C 6F 73 65 0B 49 6D 70 6F 72 74 45 72 72 // .close.ImportErr 6F 72 00 00 ) // or.. // Code size 27367 (0x6ae7) .maxstack 425 .locals init (object[] V_0, class [IronPython]IronPython.Compiler.PythonGlobal[] V_1, class [Microsoft.Scripting.Core]System.Runtime.CompilerServices.StrongBox`1 V_2, object[] V_3, On Fri, Oct 8, 2010 at 10:28 AM, Ken MacDonald wrote: > Well, just was looking at the pyc.py again; the change you suggest is > actually in the GenerateExe method, so should have no effect on what is > contained in the .dll. > > The .dll is actually generated by this code: > > print "Output:\n\t%s" % output > print "Target:\n\t%s" % target > print 'Platform:\n\t%s' % platform > print 'Machine:\n\t%s' % machine > > print 'Compiling...' > clr.CompileModules(output + '.dll', mainModule = main_name, *files) > > so I'm getting the feeling that something changed in the CompileModules > method between 2.6 and 2.7 such that it's no longer tracking down and > including the std lib references. > Ken > > > On Fri, Oct 8, 2010 at 10:08 AM, Ken MacDonald wrote: > >> HI Dino, >> Tried your change, it came up with an error on the statement below: >> TypeError: EmitCall() takes exactly 3 arguments (1 given) >> >> Not sure what the other args should be... it did seem to create a new >> myapp.dll despite the error, but it was the same size (1.2 MB) as the new >> version 2.7 dll, and was still missing std lib components. I'm not sure that >> this is valid, considering the error in the EmitCall() noted above. >> >> I did do a primitive analysis of the app to find out which std lib >> components it references; found roughly 25 of them, and as a test, >> incorporated the largest 10 or so explicitly in the compile; the size of the >> resulting dll was about 2.1 MB (much closer to the 2.6 dll version size, 2.9 >> MB), and was able to run it successfullyagainst a stripped-down std lib >> where I had removed those 10 files. It seems fairly clear to me that the 2.6 >> version dll did include the components it required from the std lib. >> >> Starting to look really promising; not quite there yet! >> Ken >> >> >> On Thu, Oct 7, 2010 at 8:06 PM, Dino Viehland wrote: >> >>> I think these changes came about due to this thread: >>> http://www.mail-archive.com/users at lists.ironpython.com/msg08794.htmlwhere there was an issue w/ relative paths and starting an app. >>> >>> >>> >>> And you may have found the root of the problem but you didn?t quote the >>> code change. There?s these additional lines: >>> >>> >>> >>> gen.EmitCall(OpCodes.Call, >>> clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ()) >>> gen.EmitCall(OpCodes.Call, >>> clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ()) >>> >>> >>> >>> which might be causing the problem as we change the CWD before we really >>> kick things off. Does replacing the set_CurrentDirectory line in pyc.py >>> with: >>> >>> >>> >>> gen.EmitCall(OpCodes.Pop) >>> >>> >>> >>> possibly fix things for you (that?ll make that a NOP but should leave all >>> the other changes in place)? >>> >>> >>> >>> *From:* users-bounces at lists.ironpython.com [mailto: >>> users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald >>> *Sent:* Wednesday, October 06, 2010 1:26 PM >>> *To:* Discussion of IronPython >>> >>> *Subject:* Re: [IronPython] change in standard library behavior for >>> compiled .exe/.dll??? >>> >>> >>> >>> As an FYI, the following change was made from the IP 2.6 version of >>> pyc.py to the IP 2.7 version (newer version shown first): >>> >>> < # get the ScriptCode assembly... >>> < gen.EmitCall(OpCodes.Call, >>> clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) >>> < gen.EmitCall(OpCodes.Callvirt, >>> clr.GetClrType(Assembly).GetMethod("get_Location"), ()) >>> --- >>> > # get the ScriptCode assembly... >>> > gen.EmitCall(OpCodes.Call, >>> clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); >>> > gen.EmitCall(OpCodes.Call, >>> clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); >>> > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( >>> (str, ) )); >>> > gen.EmitCall(OpCodes.Call, >>> clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); >>> >>> Don't know what these things do at this point, but wondering if the >>> changes have to do with compiling the entire code tree into the application >>> .DLL??? >>> Ken >>> >>> On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald >>> wrote: >>> >>> Hi Dino, >>> >>> On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland >>> wrote: >>> >>> How are you distributing your app? I?m assuming you?re going to have >>> something like: >>> >>> >>> >>> MyApp\ >>> >>> MyApp.exe >>> >>> MyApp.dll >>> >>> IronPython.dll >>> >>> IronPython.Modules.DLL >>> >>> ? >>> >>> >>> >>> This is exactly how we have been distributing our app up to IP 2.6 with >>> .NET 3.5. We also have another DLL with our resources in it; XAML, icons, >>> image files, but no code per se. We have been distributing it this way to >>> customer systems that do NOT have IronPython installed, or any sign of the >>> std. library, or "os.py" specifically, and it's been working really, really >>> well. >>> >>> We're trying to understand what changed moving to IP 2.7 and .NET 4.0 >>> that we should have to care about distributing the std. library now. The way >>> it was before was quite simple and robust; deliver a small handful of files >>> and it just worked, very easy to keep track of. Now instead of distributing >>> 5 files, we suddenly need to distribute 500??? We've figured out that it >>> certainly COULD work as described below, but it seems like a giant step >>> backwards on several fronts, including the potential for folks to >>> maliciously or accidentally tamper with the std lib. sources and affect the >>> functioning of our app. So, how do we get back to the old/better >>> functionality? >>> >>> On a *slightly *related note, our app imports some package directories >>> in addition, e.g. "import ctypes". When python encounters a directory >>> import, it looks for __init__.py in the directory, and derives the package >>> import directions from there, as I understand it. However, I can't specify >>> the "ctypes" directory as an argument to the pyc.py compile app; just causes >>> it to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" >>> and the other files in the ctypes subdirectory, it seems like "import >>> ctypes" would have no clue that the __init__.py that was compiled in had >>> anything to do with the "ctypes" package, as the path names are presumably >>> irrelevant to the compiler as long as they specify a python file. I'm >>> considering mod'ing pyc.py to be able to incorporate a list of std lib >>> modules to compile in: simple enough for the standalone files like "os.py", >>> but the compile modules don't seem to be able to grok what to do with a >>> package subdirectory. >>> Ken >>> >>> >>> >>> >>> >>> >>> You should be able to also distribute the standard library and just drop >>> it into a Lib directory next to IronPython.dll: >>> >>> MyApp\ >>> >>> MyApp.exe >>> >>> MyApp.dll >>> >>> IronPython.dll >>> >>> IronPython.Modules.DLL >>> >>> ? >>> >>> Lib\ >>> >>> os.py >>> >>> >>> >>> That lib dir should be on sys.path at startup and so it should be >>> available for importing. >>> >>> >>> >>> *From:* users-bounces at lists.ironpython.com [mailto: >>> users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald >>> *Sent:* Wednesday, October 06, 2010 11:42 AM >>> *To:* Michael Foord >>> >>> >>> *Cc:* Discussion of IronPython >>> *Subject:* Re: [IronPython] change in standard library behavior for >>> compiled .exe/.dll??? >>> >>> >>> >>> Hi Michael, >>> >>> >>> I started out on implementing this, but I am importing maybe a dozen of >>> the std. library modules, which then import others, and so on. It appears >>> that eventually, most of the std modules would have to be imported >>> explicitly (perhaps 400 or so files) which might make for a somewhat >>> cumbersome command line, incidentally also about 20K characters too long >>> :-). I'm hoping to find a way to get this to work as well as it did under IP >>> 2.5 / .NET 3.5. >>> >>> Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe >>> one of us can suggest something if we have an understanding of what you're >>> trying to do. >>> Ken >>> >>> On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord >>> wrote: >>> >>> On 05/10/2010 22:27, Ken MacDonald wrote: >>> >>> I've been looking at the .exe's we built - using pyc.py - with IP >>> 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the >>> imports (like "os") are not being built into the .exe/.dll, but instead are >>> required to be imported in source form, e.g. "os.py" must be somewhere on >>> sys.path. In the IP 2.5 .exe's we had been building, they would run fine on >>> machines without the IP standard library installed at all, in other words, >>> with "os.py" not present on the machine at all. We did notice that the .exe >>> in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 >>> MB in the IP 2.7 version, and the newer version requires that the source >>> code for the IP standard library be on the path. Is this a deliberate change >>> in behavior? We never had to package the standard library source when we >>> sent out .exe's to customers before >>> >>> >>> >>> Hmmm... I'm pretty sure I always had to explicitly compile and bundle the >>> standard library with previous versions of Python. Odd. Anyway, the simple >>> solution is to ensure that you add any standard library modules you use to >>> the set you compile and ship. >>> >>> >>> >>> All the best, >>> >>> Michael Foord >>> >>> >>> >>> >"os" is not an assembly but a Python module from the standard library. >>> You need to ensure >that the Python standard library (or the parts that you >>> use and their dependencies) is on the >path. >>> >>> >>> All the best, >>> >>> Michael Foord >>> >>> and how do I ensure it gets found from my .exe - is there a specific >>> env. variable, or the Windows %PATH% e.v., or something I haven't >>> AddReference'd to???? >>> Thanks, >>> Ken >>> >>> >>> >>> _______________________________________________ >>> >>> Users mailing list >>> >>> Users at lists.ironpython.com >>> >>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >>> >>> >>> >>> -- >>> >>> http://www.voidspace.org.uk/blog >>> >>> >>> >>> READ CAREFULLY. By accepting and reading this email you agree, >>> >>> on behalf of your employer, to release me from all obligations >>> >>> and waivers arising from any and all NON-NEGOTIATED agreements, >>> >>> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, >>> >>> confidentiality, non-disclosure, non-compete and acceptable use >>> >>> policies (?BOGUS AGREEMENTS?) that I have entered into with your >>> >>> employer, its partners, licensors, agents and assigns, in >>> >>> perpetuity, without prejudice to my ongoing rights and privileges. >>> >>> You further represent that you have the authority to release me >>> >>> from any BOGUS AGREEMENTS on behalf of your employer. >>> >>> >>> >>> >>> >>> -- >>> >>> http://www.voidspace.org.uk/blog >>> >>> >>> >>> READ CAREFULLY. By accepting and reading this email you agree, >>> >>> on behalf of your employer, to release me from all obligations >>> >>> and waivers arising from any and all NON-NEGOTIATED agreements, >>> >>> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, >>> >>> confidentiality, non-disclosure, non-compete and acceptable use >>> >>> policies (?BOGUS AGREEMENTS?) that I have entered into with your >>> >>> employer, its partners, licensors, agents and assigns, in >>> >>> perpetuity, without prejudice to my ongoing rights and privileges. >>> >>> You further represent that you have the authority to release me >>> >>> from any BOGUS AGREEMENTS on behalf of your employer. >>> >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users at lists.ironpython.com >>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users at lists.ironpython.com >>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Fri Oct 8 20:54:31 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Fri, 8 Oct 2010 14:54:31 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Hi Dino, We are definitely creating both the stub .exe and .dll, and I have now verified using the ildasm tool that the older version .dll created by the 2.6 pyc.py DOES contain the required std lib modules (os, ntpath, fractions, decimal,... were all in there), and that the 2.7-compiled .dll does NOT contain any of the modules. Ken On Fri, Oct 8, 2010 at 2:16 PM, Dino Viehland wrote: > Are you not creating an EXE? My theory here is that changing the CWD is > causing us to not pick up the std lib because I don?t think the DLL has ever > included the std lib. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Friday, October 08, 2010 7:28 AM > > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Well, just was looking at the pyc.py again; the change you suggest is > actually in the GenerateExe method, so should have no effect on what is > contained in the .dll. > > The .dll is actually generated by this code: > > print "Output:\n\t%s" % output > print "Target:\n\t%s" % target > print 'Platform:\n\t%s' % platform > print 'Machine:\n\t%s' % machine > > print 'Compiling...' > clr.CompileModules(output + '.dll', mainModule = main_name, *files) > > so I'm getting the feeling that something changed in the CompileModules > method between 2.6 and 2.7 such that it's no longer tracking down and > including the std lib references. > Ken > > On Fri, Oct 8, 2010 at 10:08 AM, Ken MacDonald wrote: > > HI Dino, > Tried your change, it came up with an error on the statement below: > TypeError: EmitCall() takes exactly 3 arguments (1 given) > > Not sure what the other args should be... it did seem to create a new > myapp.dll despite the error, but it was the same size (1.2 MB) as the new > version 2.7 dll, and was still missing std lib components. I'm not sure that > this is valid, considering the error in the EmitCall() noted above. > > I did do a primitive analysis of the app to find out which std lib > components it references; found roughly 25 of them, and as a test, > incorporated the largest 10 or so explicitly in the compile; the size of the > resulting dll was about 2.1 MB (much closer to the 2.6 dll version size, 2.9 > MB), and was able to run it successfullyagainst a stripped-down std lib > where I had removed those 10 files. It seems fairly clear to me that the 2.6 > version dll did include the components it required from the std lib. > > Starting to look really promising; not quite there yet! > Ken > > > > On Thu, Oct 7, 2010 at 8:06 PM, Dino Viehland wrote: > > I think these changes came about due to this thread: > http://www.mail-archive.com/users at lists.ironpython.com/msg08794.html where > there was an issue w/ relative paths and starting an app. > > > > And you may have found the root of the problem but you didn?t quote the > code change. There?s these additional lines: > > > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ()) > gen.EmitCall(OpCodes.Call, > clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ()) > > > > which might be causing the problem as we change the CWD before we really > kick things off. Does replacing the set_CurrentDirectory line in pyc.py > with: > > > > gen.EmitCall(OpCodes.Pop) > > > > possibly fix things for you (that?ll make that a NOP but should leave all > the other changes in place)? > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 1:26 PM > *To:* Discussion of IronPython > > > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > As an FYI, the following change was made from the IP 2.6 version of pyc.py > to the IP 2.7 version (newer version shown first): > > < # get the ScriptCode assembly... > < gen.EmitCall(OpCodes.Call, > clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) > < gen.EmitCall(OpCodes.Callvirt, > clr.GetClrType(Assembly).GetMethod("get_Location"), ()) > --- > > # get the ScriptCode assembly... > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); > > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( > (str, ) )); > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); > > Don't know what these things do at this point, but wondering if the changes > have to do with compiling the entire code tree into the application .DLL??? > Ken > > On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald wrote: > > Hi Dino, > > On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland wrote: > > How are you distributing your app? I?m assuming you?re going to have > something like: > > > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > > > This is exactly how we have been distributing our app up to IP 2.6 with > .NET 3.5. We also have another DLL with our resources in it; XAML, icons, > image files, but no code per se. We have been distributing it this way to > customer systems that do NOT have IronPython installed, or any sign of the > std. library, or "os.py" specifically, and it's been working really, really > well. > > We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that > we should have to care about distributing the std. library now. The way it > was before was quite simple and robust; deliver a small handful of files and > it just worked, very easy to keep track of. Now instead of distributing 5 > files, we suddenly need to distribute 500??? We've figured out that it > certainly COULD work as described below, but it seems like a giant step > backwards on several fronts, including the potential for folks to > maliciously or accidentally tamper with the std lib. sources and affect the > functioning of our app. So, how do we get back to the old/better > functionality? > > On a *slightly *related note, our app imports some package directories in > addition, e.g. "import ctypes". When python encounters a directory import, > it looks for __init__.py in the directory, and derives the package import > directions from there, as I understand it. However, I can't specify the > "ctypes" directory as an argument to the pyc.py compile app; just causes it > to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and > the other files in the ctypes subdirectory, it seems like "import ctypes" > would have no clue that the __init__.py that was compiled in had anything to > do with the "ctypes" package, as the path names are presumably irrelevant to > the compiler as long as they specify a python file. I'm considering mod'ing > pyc.py to be able to incorporate a list of std lib modules to compile in: > simple enough for the standalone files like "os.py", but the compile modules > don't seem to be able to grok what to do with a package subdirectory. > Ken > > > > > > > You should be able to also distribute the standard library and just drop it > into a Lib directory next to IronPython.dll: > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > Lib\ > > os.py > > > > That lib dir should be on sys.path at startup and so it should be available > for importing. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 11:42 AM > *To:* Michael Foord > > > *Cc:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Hi Michael, > > > I started out on implementing this, but I am importing maybe a dozen of the > std. library modules, which then import others, and so on. It appears that > eventually, most of the std modules would have to be imported explicitly > (perhaps 400 or so files) which might make for a somewhat cumbersome command > line, incidentally also about 20K characters too long :-). I'm hoping to > find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. > > Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one > of us can suggest something if we have an understanding of what you're > trying to do. > Ken > > On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: > > On 05/10/2010 22:27, Ken MacDonald wrote: > > I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net > 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like > "os") are not being built into the .exe/.dll, but instead are required to be > imported in source form, e.g. "os.py" must be somewhere on sys.path. In the > IP 2.5 .exe's we had been building, they would run fine on machines without > the IP standard library installed at all, in other words, with "os.py" not > present on the machine at all. We did notice that the .exe in question went > from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 > version, and the newer version requires that the source code for the IP > standard library be on the path. Is this a deliberate change in behavior? We > never had to package the standard library source when we sent out .exe's to > customers before > > > > Hmmm... I'm pretty sure I always had to explicitly compile and bundle the > standard library with previous versions of Python. Odd. Anyway, the simple > solution is to ensure that you add any standard library modules you use to > the set you compile and ship. > > > > All the best, > > Michael Foord > > > >"os" is not an assembly but a Python module from the standard library. You > need to ensure >that the Python standard library (or the parts that you use > and their dependencies) is on the >path. > > > All the best, > > Michael Foord > > and how do I ensure it gets found from my .exe - is there a specific env. > variable, or the Windows %PATH% e.v., or something I haven't AddReference'd > to???? > Thanks, > Ken > > > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Fri Oct 8 22:25:56 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 8 Oct 2010 20:25:56 +0000 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: Ok, very strange but at least we're getting some data :)... can you send the command line you're building each pyc.py with? And could you modify each pyc.py so the call before clr.CompileModules we print out the args like: print main_name, files clr.CompileModules(output + '.dll', mainModule = main_name, *files) I'm hoping that we'll see os.py in there somewhere. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Friday, October 08, 2010 11:55 AM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Dino, We are definitely creating both the stub .exe and .dll, and I have now verified using the ildasm tool that the older version .dll created by the 2.6 pyc.py DOES contain the required std lib modules (os, ntpath, fractions, decimal,... were all in there), and that the 2.7-compiled .dll does NOT contain any of the modules. Ken On Fri, Oct 8, 2010 at 2:16 PM, Dino Viehland > wrote: Are you not creating an EXE? My theory here is that changing the CWD is causing us to not pick up the std lib because I don't think the DLL has ever included the std lib. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Friday, October 08, 2010 7:28 AM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Well, just was looking at the pyc.py again; the change you suggest is actually in the GenerateExe method, so should have no effect on what is contained in the .dll. The .dll is actually generated by this code: print "Output:\n\t%s" % output print "Target:\n\t%s" % target print 'Platform:\n\t%s' % platform print 'Machine:\n\t%s' % machine print 'Compiling...' clr.CompileModules(output + '.dll', mainModule = main_name, *files) so I'm getting the feeling that something changed in the CompileModules method between 2.6 and 2.7 such that it's no longer tracking down and including the std lib references. Ken On Fri, Oct 8, 2010 at 10:08 AM, Ken MacDonald > wrote: HI Dino, Tried your change, it came up with an error on the statement below: TypeError: EmitCall() takes exactly 3 arguments (1 given) Not sure what the other args should be... it did seem to create a new myapp.dll despite the error, but it was the same size (1.2 MB) as the new version 2.7 dll, and was still missing std lib components. I'm not sure that this is valid, considering the error in the EmitCall() noted above. I did do a primitive analysis of the app to find out which std lib components it references; found roughly 25 of them, and as a test, incorporated the largest 10 or so explicitly in the compile; the size of the resulting dll was about 2.1 MB (much closer to the 2.6 dll version size, 2.9 MB), and was able to run it successfullyagainst a stripped-down std lib where I had removed those 10 files. It seems fairly clear to me that the 2.6 version dll did include the components it required from the std lib. Starting to look really promising; not quite there yet! Ken On Thu, Oct 7, 2010 at 8:06 PM, Dino Viehland > wrote: I think these changes came about due to this thread: http://www.mail-archive.com/users at lists.ironpython.com/msg08794.html where there was an issue w/ relative paths and starting an app. And you may have found the root of the problem but you didn't quote the code change. There's these additional lines: gen.EmitCall(OpCodes.Call, clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ()) gen.EmitCall(OpCodes.Call, clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ()) which might be causing the problem as we change the CWD before we really kick things off. Does replacing the set_CurrentDirectory line in pyc.py with: gen.EmitCall(OpCodes.Pop) possibly fix things for you (that'll make that a NOP but should leave all the other changes in place)? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 1:26 PM To: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? As an FYI, the following change was made from the IP 2.6 version of pyc.py to the IP 2.7 version (newer version shown first): < # get the ScriptCode assembly... < gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) < gen.EmitCall(OpCodes.Callvirt, clr.GetClrType(Assembly).GetMethod("get_Location"), ()) --- > # get the ScriptCode assembly... > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); > gen.EmitCall(OpCodes.Call, clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( (str, ) )); > gen.EmitCall(OpCodes.Call, clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); Don't know what these things do at this point, but wondering if the changes have to do with compiling the entire code tree into the application .DLL??? Ken On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald > wrote: Hi Dino, On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland > wrote: How are you distributing your app? I'm assuming you're going to have something like: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... This is exactly how we have been distributing our app up to IP 2.6 with .NET 3.5. We also have another DLL with our resources in it; XAML, icons, image files, but no code per se. We have been distributing it this way to customer systems that do NOT have IronPython installed, or any sign of the std. library, or "os.py" specifically, and it's been working really, really well. We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that we should have to care about distributing the std. library now. The way it was before was quite simple and robust; deliver a small handful of files and it just worked, very easy to keep track of. Now instead of distributing 5 files, we suddenly need to distribute 500??? We've figured out that it certainly COULD work as described below, but it seems like a giant step backwards on several fronts, including the potential for folks to maliciously or accidentally tamper with the std lib. sources and affect the functioning of our app. So, how do we get back to the old/better functionality? On a slightly related note, our app imports some package directories in addition, e.g. "import ctypes". When python encounters a directory import, it looks for __init__.py in the directory, and derives the package import directions from there, as I understand it. However, I can't specify the "ctypes" directory as an argument to the pyc.py compile app; just causes it to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and the other files in the ctypes subdirectory, it seems like "import ctypes" would have no clue that the __init__.py that was compiled in had anything to do with the "ctypes" package, as the path names are presumably irrelevant to the compiler as long as they specify a python file. I'm considering mod'ing pyc.py to be able to incorporate a list of std lib modules to compile in: simple enough for the standalone files like "os.py", but the compile modules don't seem to be able to grok what to do with a package subdirectory. Ken You should be able to also distribute the standard library and just drop it into a Lib directory next to IronPython.dll: MyApp\ MyApp.exe MyApp.dll IronPython.dll IronPython.Modules.DLL ... Lib\ os.py That lib dir should be on sys.path at startup and so it should be available for importing. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Wednesday, October 06, 2010 11:42 AM To: Michael Foord Cc: Discussion of IronPython Subject: Re: [IronPython] change in standard library behavior for compiled .exe/.dll??? Hi Michael, I started out on implementing this, but I am importing maybe a dozen of the std. library modules, which then import others, and so on. It appears that eventually, most of the std modules would have to be imported explicitly (perhaps 400 or so files) which might make for a somewhat cumbersome command line, incidentally also about 20K characters too long :-). I'm hoping to find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one of us can suggest something if we have an understanding of what you're trying to do. Ken On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: On 05/10/2010 22:27, Ken MacDonald wrote: I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like "os") are not being built into the .exe/.dll, but instead are required to be imported in source form, e.g. "os.py" must be somewhere on sys.path. In the IP 2.5 .exe's we had been building, they would run fine on machines without the IP standard library installed at all, in other words, with "os.py" not present on the machine at all. We did notice that the .exe in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 version, and the newer version requires that the source code for the IP standard library be on the path. Is this a deliberate change in behavior? We never had to package the standard library source when we sent out .exe's to customers before Hmmm... I'm pretty sure I always had to explicitly compile and bundle the standard library with previous versions of Python. Odd. Anyway, the simple solution is to ensure that you add any standard library modules you use to the set you compile and ship. All the best, Michael Foord >"os" is not an assembly but a Python module from the standard library. You need to ensure >that the Python standard library (or the parts that you use and their dependencies) is on the >path. All the best, Michael Foord and how do I ensure it gets found from my .exe - is there a specific env. variable, or the Windows %PATH% e.v., or something I haven't AddReference'd to???? Thanks, Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -- http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Fri Oct 8 23:18:49 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Fri, 8 Oct 2010 17:18:49 -0400 Subject: [IronPython] change in standard library behavior for compiled .exe/.dll??? In-Reply-To: References: <4CAC58BC.206@voidspace.org.uk> Message-ID: BIG DUH. That's it. Our 2.6 version of pyc.py is somehow adding the refs it needs to the files list. So in 2.6 we were generating a command line that not only included our myapp modules, but selected std lib modules as well. Duh. Sorry for all the noise; thanks to all (esp. Dino) for helping us sort this out. We should be able to fix the 2.7 version to generate the add'l modules as well. Ken On Fri, Oct 8, 2010 at 4:25 PM, Dino Viehland wrote: > Ok, very strange but at least we?re getting some data J? can you send > the command line you?re building each pyc.py with? And could you modify > each pyc.py so the call before clr.CompileModules we print out the args > like: > > > > print main_name, files > > clr.CompileModules(output + '.dll', mainModule = main_name, *files) > > > > I?m hoping that we?ll see os.py in there somewhere. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Friday, October 08, 2010 11:55 AM > > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Hi Dino, > We are definitely creating both the stub .exe and .dll, and I have now > verified using the ildasm tool that the older version .dll created by the > 2.6 pyc.py DOES contain the required std lib modules (os, ntpath, fractions, > decimal,... were all in there), and that the 2.7-compiled .dll does NOT > contain any of the modules. > Ken > > On Fri, Oct 8, 2010 at 2:16 PM, Dino Viehland wrote: > > Are you not creating an EXE? My theory here is that changing the CWD is > causing us to not pick up the std lib because I don?t think the DLL has ever > included the std lib. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Friday, October 08, 2010 7:28 AM > > > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Well, just was looking at the pyc.py again; the change you suggest is > actually in the GenerateExe method, so should have no effect on what is > contained in the .dll. > > The .dll is actually generated by this code: > > print "Output:\n\t%s" % output > print "Target:\n\t%s" % target > print 'Platform:\n\t%s' % platform > print 'Machine:\n\t%s' % machine > > print 'Compiling...' > clr.CompileModules(output + '.dll', mainModule = main_name, *files) > > so I'm getting the feeling that something changed in the CompileModules > method between 2.6 and 2.7 such that it's no longer tracking down and > including the std lib references. > Ken > > On Fri, Oct 8, 2010 at 10:08 AM, Ken MacDonald wrote: > > HI Dino, > Tried your change, it came up with an error on the statement below: > TypeError: EmitCall() takes exactly 3 arguments (1 given) > > Not sure what the other args should be... it did seem to create a new > myapp.dll despite the error, but it was the same size (1.2 MB) as the new > version 2.7 dll, and was still missing std lib components. I'm not sure that > this is valid, considering the error in the EmitCall() noted above. > > I did do a primitive analysis of the app to find out which std lib > components it references; found roughly 25 of them, and as a test, > incorporated the largest 10 or so explicitly in the compile; the size of the > resulting dll was about 2.1 MB (much closer to the 2.6 dll version size, 2.9 > MB), and was able to run it successfullyagainst a stripped-down std lib > where I had removed those 10 files. It seems fairly clear to me that the 2.6 > version dll did include the components it required from the std lib. > > Starting to look really promising; not quite there yet! > Ken > > > > On Thu, Oct 7, 2010 at 8:06 PM, Dino Viehland wrote: > > I think these changes came about due to this thread: > http://www.mail-archive.com/users at lists.ironpython.com/msg08794.html where > there was an issue w/ relative paths and starting an app. > > > > And you may have found the root of the problem but you didn?t quote the > code change. There?s these additional lines: > > > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ()) > gen.EmitCall(OpCodes.Call, > clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ()) > > > > which might be causing the problem as we change the CWD before we really > kick things off. Does replacing the set_CurrentDirectory line in pyc.py > with: > > > > gen.EmitCall(OpCodes.Pop) > > > > possibly fix things for you (that?ll make that a NOP but should leave all > the other changes in place)? > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 1:26 PM > *To:* Discussion of IronPython > > > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > As an FYI, the following change was made from the IP 2.6 version of pyc.py > to the IP 2.7 version (newer version shown first): > > < # get the ScriptCode assembly... > < gen.EmitCall(OpCodes.Call, > clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()) > < gen.EmitCall(OpCodes.Callvirt, > clr.GetClrType(Assembly).GetMethod("get_Location"), ()) > --- > > # get the ScriptCode assembly... > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ()); > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ()); > > gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor( > (str, ) )); > > gen.EmitCall(OpCodes.Call, > clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ()); > > Don't know what these things do at this point, but wondering if the changes > have to do with compiling the entire code tree into the application .DLL??? > Ken > > On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald wrote: > > Hi Dino, > > On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland wrote: > > How are you distributing your app? I?m assuming you?re going to have > something like: > > > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > > > This is exactly how we have been distributing our app up to IP 2.6 with > .NET 3.5. We also have another DLL with our resources in it; XAML, icons, > image files, but no code per se. We have been distributing it this way to > customer systems that do NOT have IronPython installed, or any sign of the > std. library, or "os.py" specifically, and it's been working really, really > well. > > We're trying to understand what changed moving to IP 2.7 and .NET 4.0 that > we should have to care about distributing the std. library now. The way it > was before was quite simple and robust; deliver a small handful of files and > it just worked, very easy to keep track of. Now instead of distributing 5 > files, we suddenly need to distribute 500??? We've figured out that it > certainly COULD work as described below, but it seems like a giant step > backwards on several fronts, including the potential for folks to > maliciously or accidentally tamper with the std lib. sources and affect the > functioning of our app. So, how do we get back to the old/better > functionality? > > On a *slightly *related note, our app imports some package directories in > addition, e.g. "import ctypes". When python encounters a directory import, > it looks for __init__.py in the directory, and derives the package import > directions from there, as I understand it. However, I can't specify the > "ctypes" directory as an argument to the pyc.py compile app; just causes it > to croak. If I explicitly specified paths like "lib\ctypes\__init__.py" and > the other files in the ctypes subdirectory, it seems like "import ctypes" > would have no clue that the __init__.py that was compiled in had anything to > do with the "ctypes" package, as the path names are presumably irrelevant to > the compiler as long as they specify a python file. I'm considering mod'ing > pyc.py to be able to incorporate a list of std lib modules to compile in: > simple enough for the standalone files like "os.py", but the compile modules > don't seem to be able to grok what to do with a package subdirectory. > Ken > > > > > > > You should be able to also distribute the standard library and just drop it > into a Lib directory next to IronPython.dll: > > MyApp\ > > MyApp.exe > > MyApp.dll > > IronPython.dll > > IronPython.Modules.DLL > > ? > > Lib\ > > os.py > > > > That lib dir should be on sys.path at startup and so it should be available > for importing. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Wednesday, October 06, 2010 11:42 AM > *To:* Michael Foord > > > *Cc:* Discussion of IronPython > *Subject:* Re: [IronPython] change in standard library behavior for > compiled .exe/.dll??? > > > > Hi Michael, > > > I started out on implementing this, but I am importing maybe a dozen of the > std. library modules, which then import others, and so on. It appears that > eventually, most of the std modules would have to be imported explicitly > (perhaps 400 or so files) which might make for a somewhat cumbersome command > line, incidentally also about 20K characters too long :-). I'm hoping to > find a way to get this to work as well as it did under IP 2.5 / .NET 3.5. > > Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe one > of us can suggest something if we have an understanding of what you're > trying to do. > Ken > > On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord > wrote: > > On 05/10/2010 22:27, Ken MacDonald wrote: > > I've been looking at the .exe's we built - using pyc.py - with IP 2.5/.Net > 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the imports (like > "os") are not being built into the .exe/.dll, but instead are required to be > imported in source form, e.g. "os.py" must be somewhere on sys.path. In the > IP 2.5 .exe's we had been building, they would run fine on machines without > the IP standard library installed at all, in other words, with "os.py" not > present on the machine at all. We did notice that the .exe in question went > from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2 MB in the IP 2.7 > version, and the newer version requires that the source code for the IP > standard library be on the path. Is this a deliberate change in behavior? We > never had to package the standard library source when we sent out .exe's to > customers before > > > > Hmmm... I'm pretty sure I always had to explicitly compile and bundle the > standard library with previous versions of Python. Odd. Anyway, the simple > solution is to ensure that you add any standard library modules you use to > the set you compile and ship. > > > > All the best, > > Michael Foord > > > >"os" is not an assembly but a Python module from the standard library. You > need to ensure >that the Python standard library (or the parts that you use > and their dependencies) is on the >path. > > > All the best, > > Michael Foord > > and how do I ensure it gets found from my .exe - is there a specific env. > variable, or the Windows %PATH% e.v., or something I haven't AddReference'd > to???? > Thanks, > Ken > > > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > > -- > > http://www.voidspace.org.uk/blog > > > > READ CAREFULLY. By accepting and reading this email you agree, > > on behalf of your employer, to release me from all obligations > > and waivers arising from any and all NON-NEGOTIATED agreements, > > licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, > > confidentiality, non-disclosure, non-compete and acceptable use > > policies (?BOGUS AGREEMENTS?) that I have entered into with your > > employer, its partners, licensors, agents and assigns, in > > perpetuity, without prejudice to my ongoing rights and privileges. > > You further represent that you have the authority to release me > > from any BOGUS AGREEMENTS on behalf of your employer. > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From empirebuilder at gmail.com Sun Oct 10 22:01:20 2010 From: empirebuilder at gmail.com (Dody Gunawinata) Date: Sun, 10 Oct 2010 18:01:20 -0200 Subject: [IronPython] Embedding IronPython in ASP.NET/SharePoint2010/Commerce Server In-Reply-To: References: <4EFFA72C340FF54A9A2FA84786378C6DC4F6211593@RCMTMAIL.rcmt.com> Message-ID: I am using similar approach in our business license software that runs the city of Alexandria, Egypt. We have to deal with 350 different type of business licenses with multiple factor for calculating the fees (minimum 5). So we just construct the fees calculation formula in Python script that a business admin can change at any time and import all the necessary variables for the formula to work. On Sun, Oct 3, 2010 at 1:45 PM, Jeff Hardy wrote: > Hi Charles, > The approach I've used for web applications in the past is to have the > runtime and engine be application-wide and create a new scope for each > script on every request. Scopes are very cheap to create and will keep the > scripts isolated so that they don't affect each other. The engine and > runtime are thread-safe, so you can safely get away with only having one per > application. I also use a cache of CompiledCode objects for each script to > avoid recompiling them each time. > > - Jeff > > On Sun, Oct 3, 2010 at 6:15 AM, Medcoff, Charles > wrote: > >> Hello, >> >> >> >> I have a client for which I am developing a SharePoint 2010/Microsoft >> Commerce Server 2009 applications. The client has a fairly complex set of >> business rules for their ordering process which may change frequently. >> Rules engines are pretty expensive so I?m considering the approach of >> embedding Iron Python into the app as an alternative to making it a bit >> easier to make rules updates via scripting. >> >> >> >> My approach is to launch an Iron Python script from within an Operation >> Sequence Component ( >> http://msdn.microsoft.com/en-us/library/dd464335(CS.90).aspx) which is a >> standard extensibility point within Commerce Server. In nutshell an >> Operation Sequence Component is a .NET assembly containing a class which >> implements a particular interface, and is loaded via reflection by the >> Commerce Server runtime. The class is then loaded/executed when the >> relevant Commerce Server API is used. >> >> >> >> I?ve successfully written a simple Operation Sequence Component which >> executes a script. The only catch to getting it to work is to register all >> of the related Iron Python DLL in the GAC. Here is an example of what I?m >> doing: >> >> >> >> namespace IPythonOperationalSequenceComponent >> { >> public class IronPythonOperationSequenceComponent >> : Microsoft.Commerce.Providers.Components.OperationSequenceComponent >> { >> public override void ExecuteQuery( >> Microsoft.Commerce.Contracts.Messages.CommerceQueryOperation >> queryOperation, >> Microsoft.Commerce.Broker.OperationCacheDictionary >> operationCache, >> Microsoft.Commerce.Contracts.Messages. >> CommerceQueryOperationResponse response) >> { >> try >> { >> ScriptEngine pyEngine = Python.CreateEngine(); >> ScriptScope pyScope = pyEngine.CreateScope(); >> pyScope.SetVariable("request", queryOperation); >> pyScope.SetVariable("cache", operationCache); >> pyScope.SetVariable("response", response); >> ScriptSource >> source = pyEngine.CreateScriptSourceFromFile(@"rules.py"); >> CompiledCode compiled = source.Compile(); >> compiled.Execute(pyScope); >> } >> catch (Exception ex) >> { >> // error handling elided >> } >> } >> } >> } >> >> >> >> >> >> Now this is a ?toy? approach as I shouldn?t need to create an engine, >> scope, compile the script, etc. for every post back. I?m thinking a better >> approach is to have the script engine, scope and compiled script live at the >> HttpWebApplication level and be shared across all threads, requests, users. >> My concern is that will this approach work. What are the threading, >> concurrency, and performance issues involved? Has anyone done anything like >> this in ASP, let alone SharePoint and can share their experiences with >> regards to what works, what does not work, best approach, etc. >> >> >> >> --chuck >> >> >> >> >> >> Best Regards, >> >> Chuck >> >> >> >> >> >> Charles Medcoff >> >> Principal Consultant* *|* *Enterprise Integration Solutions* * >> >> [image: Description: Description: mailsiglogo] >> >> Tel: (248) 687-5623 >> >> Cell: (248) 884-2854 >> >> www.rcmt.com/eis >> >> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -- nomadlife.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From msenko at completegenomics.com Mon Oct 11 19:40:09 2010 From: msenko at completegenomics.com (Mark Senko) Date: Mon, 11 Oct 2010 10:40:09 -0700 Subject: [IronPython] Return value from Execute() Message-ID: <2E3279D4F240C54C967B380B6B863145383492@ws-be-exchange.completegenomics.com> I'm trying to implement IronPython as a scripting enhancement to our application. Part of the design is to have an interactive screen available. When I call the Execute() method with a command typed in by the user, an object is returned. I can easily write this to my window using _pythonRes.ToString() if the result is a simple type like Int32. But I've had an impossible time trying to figure out how to iterate over an IronPython.Runtime.List as returned by a command such as dir() since it seems to not be enumerable. Typecasting this object doesn't to an array or a List<> doesn't seem to work. Any help would be appreciated. Being new to IronPython, I'm also a little worried about other types that I may not be aware of, yet! This is how I obtain my return value: ScriptEngine _pengine = Python.CreateEngine(); ScriptSource source = _pengine.CreateScriptSourceFromString(_pythonCmd, SourceCodeKind.AutoDetect); var _pythonRes = source.Execute(_pscope); ____ The contents of this e-mail and any attachments are confidential and only for use by the intended recipient. Any unauthorized use, distribution or copying of this message is strictly prohibited. If you are not the intended recipient please inform the sender immediately by reply e-mail and delete this message from your system. Thank you for your co-operation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Mon Oct 11 21:09:55 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Mon, 11 Oct 2010 13:09:55 -0600 Subject: [IronPython] Return value from Execute() In-Reply-To: <2E3279D4F240C54C967B380B6B863145383492@ws-be-exchange.completegenomics.com> References: <2E3279D4F240C54C967B380B6B863145383492@ws-be-exchange.completegenomics.com> Message-ID: On Mon, Oct 11, 2010 at 11:40 AM, Mark Senko wrote: > But I?ve had an impossible time trying to figure out how to iterate over an > IronPython.Runtime.List as returned by a command such as dir() since it > seems to not be enumerable. > > Typecasting this object doesn?t to an array or a List<> doesn?t seem to > work. Have you tried casting to IEnumerable? That has always* worked for me. var _pythonRes = source.Execute(_pscope); if(_pythonRes is IEnumerable) { ... } Or, in C# 4, you should be able to make _pythonRes dynamic: dynamic _pythonRes = source.Execute(_pscope); foreach(dynamic r in _pythonRes) { ... } Whichever works better for you. - Jeff * There used to be some bugs, but they should be long gone. From msenko at completegenomics.com Mon Oct 11 21:53:24 2010 From: msenko at completegenomics.com (Mark Senko) Date: Mon, 11 Oct 2010 12:53:24 -0700 Subject: [IronPython] Return value from Execute() In-Reply-To: References: <2E3279D4F240C54C967B380B6B863145383492@ws-be-exchange.completegenomics.com> Message-ID: <2E3279D4F240C54C967B380B6B8631453834DF@ws-be-exchange.completegenomics.com> So simple! Casting to IEnumerable worked great! Maybe I should have mentioned that I'm relatively new to C#, too. I didn't realize you could cast to an interface, though now that I think about it, it makes perfect sense. Thank you, Jeff. -----Original Message----- From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Jeff Hardy Sent: Monday, October 11, 2010 12:10 PM To: Discussion of IronPython Subject: Re: [IronPython] Return value from Execute() On Mon, Oct 11, 2010 at 11:40 AM, Mark Senko wrote: > But I've had an impossible time trying to figure out how to iterate over an > IronPython.Runtime.List as returned by a command such as dir() since it > seems to not be enumerable. > > Typecasting this object doesn't to an array or a List<> doesn't seem to > work. Have you tried casting to IEnumerable? That has always* worked for me. var _pythonRes = source.Execute(_pscope); if(_pythonRes is IEnumerable) { ... } Or, in C# 4, you should be able to make _pythonRes dynamic: dynamic _pythonRes = source.Execute(_pscope); foreach(dynamic r in _pythonRes) { ... } Whichever works better for you. - Jeff * There used to be some bugs, but they should be long gone. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ____ The contents of this e-mail and any attachments are confidential and only for use by the intended recipient. Any unauthorized use, distribution or copying of this message is strictly prohibited. If you are not the intended recipient please inform the sender immediately by reply e-mail and delete this message from your system. Thank you for your co-operation. From twyatt at ppi.ca Mon Oct 11 21:56:03 2010 From: twyatt at ppi.ca (Tim P. Wyatt) Date: Mon, 11 Oct 2010 12:56:03 -0700 Subject: [IronPython] Tim Wyatt is out of the office Message-ID: I will be out of the office starting 11/10/2010 and will not return until 14/10/2010. I will be reviewing my email less frequently than usual. For urgent matters please contact the Vancouver Help Desk at 1-800-661-7712 (helpdesk at ppi.ca) From foreachdev at gmail.com Tue Oct 12 01:00:59 2010 From: foreachdev at gmail.com (JR Kincaid) Date: Mon, 11 Oct 2010 19:00:59 -0400 Subject: [IronPython] Tim Wyatt is out of the office In-Reply-To: References: Message-ID: Can you unhook this list from your work email. Thanks, JR Kincaid On Mon, Oct 11, 2010 at 3:56 PM, Tim P. Wyatt wrote: > > I will be out of the office starting 11/10/2010 and will not return until > 14/10/2010. > > I will be reviewing my email less frequently than usual. > > For urgent matters please contact the Vancouver Help Desk at 1-800-661-7712 > (helpdesk at ppi.ca) > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablodalma93 at hotmail.com Tue Oct 12 16:29:47 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Tue, 12 Oct 2010 11:29:47 -0300 Subject: [IronPython] Bug in copy module (really need help with this!) :) Message-ID: Hello guys,we've found a bug in the copy module, but the real problem is we cant reproduce it. Sometimes when we try to deepcopy an object of a custom object relation mapping type, we get this error"name types is not defined"Line 55: for t in (type(None),int,long,float,bool,str,tuple), frozenset,type,xrange,types.ClassType,types.BuiltinFunctionType, type(Ellipsis),the other problem is in production server, where this error uses to happen sporadically, most times we dont get this message but an error message of the most outer module which calls the copy module, which is a generic message which tells us nothing I know this is very little information to find a bug but, do you know of any bugs related to the module copy (aside it didnt copy System.DBnull which I believe it's not the one for this case)Greetings, Pablo Dalmazzo -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at voidspace.org.uk Tue Oct 12 16:43:45 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Tue, 12 Oct 2010 15:43:45 +0100 Subject: [IronPython] Bug in copy module (really need help with this!) :) In-Reply-To: References: Message-ID: <4CB47421.4070207@voidspace.org.uk> On 12/10/2010 15:29, Pablo Dalmazzo wrote: > Hello guys, > > we've found a bug in the copy module, but the real problem is we cant > reproduce it. > > Sometimes when we try to deepcopy an object of a custom object > relation mapping type, we get this error > > "name types is not defined" > > Line 55: > for t in (type(None),int,long,float,bool,str,tuple), > frozenset,type,xrange,types.ClassType,types.BuiltinFunctionType, > type(Ellipsis), > In my version of Python 2.6 this is line 102 of the copy module. Line 51 is "import types", so a "name types is not defined" error is most perplexing. Is it possible that you are using a custom version of the copy module? All the best, Michael > the other problem is in production server, where this error uses to > happen sporadically, most times we dont get this message but an error > message of the most outer module which calls the copy module, which is > a generic message which tells us nothing > > I know this is very little information to find a bug but, do you know > of any bugs related to the module copy (aside it didnt copy > System.DBnull which I believe it's not the one for this case) > > Greetings, Pablo Dalmazzo > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablodalma93 at hotmail.com Tue Oct 12 17:40:13 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Tue, 12 Oct 2010 12:40:13 -0300 Subject: [IronPython] Bug in copy module (really need help with this!) :) In-Reply-To: <4CB47421.4070207@voidspace.org.uk> References: , <4CB47421.4070207@voidspace.org.uk> Message-ID: Hi Michael, yes, sorry, it's line 102. I have added a couple of lines to "fix" the problem with the module copy not copying System.DBNull types before, it looked like they worked with that problem. (and I erased the commentaries of the module) But this seems to be another problem. I compared both files with Diffuse, the only lines I've added before are import clr clr.AddReference('System') import System (at the begining) and if args[0] is System.DBNull: y = System.DBNull else: y = callable(*args) in line 323 of the original file P.S: this is old but I've read your article on dynamic scope for Python to do something similar to Ruby code blocks, that was quite cool for me :) Date: Tue, 12 Oct 2010 15:43:45 +0100 From: michael at voidspace.org.uk To: users at lists.ironpython.com CC: pablodalma93 at hotmail.com Subject: Re: [IronPython] Bug in copy module (really need help with this!) :) On 12/10/2010 15:29, Pablo Dalmazzo wrote: Hello guys, we've found a bug in the copy module, but the real problem is we cant reproduce it. Sometimes when we try to deepcopy an object of a custom object relation mapping type, we get this error "name types is not defined" Line 55: for t in (type(None),int,long,float,bool,str,tuple), frozenset,type,xrange,types.ClassType,types.BuiltinFunctionType, type(Ellipsis), In my version of Python 2.6 this is line 102 of the copy module. Line 51 is "import types", so a "name types is not defined" error is most perplexing. Is it possible that you are using a custom version of the copy module? All the best, Michael the other problem is in production server, where this error uses to happen sporadically, most times we dont get this message but an error message of the most outer module which calls the copy module, which is a generic message which tells us nothing I know this is very little information to find a bug but, do you know of any bugs related to the module copy (aside it didnt copy System.DBnull which I believe it's not the one for this case) Greetings, Pablo Dalmazzo _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Tue Oct 12 17:43:40 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Tue, 12 Oct 2010 11:43:40 -0400 Subject: [IronPython] What happened to InitializeModule() in IP 2.7? Message-ID: Hi, I'm running the pyc.py IronPython compiler. In the GenerateExe method, there is a call to ....GetMethod("InitializeModule"). This method was available in IP 2.6 but seems to have disappeared in IP 2.7. I used "GetMethods()" to confirm that in IP 2.6 we have: Int32* InitializeModule*(System.Reflection.Assembly, System.String, System.String[]) but if I run in IP 2.7, the method is non-existent. This causes the GenerateExe method to fail, and the stub .exe file is not created. Fortunately, the old stub module .exe still works with the new .dll created. Anyone know if there is something else that will work, or if the method has been renamed, or...?????? Thanks, Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Tue Oct 12 19:30:32 2010 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 12 Oct 2010 17:30:32 +0000 Subject: [IronPython] What happened to InitializeModule() in IP 2.7? In-Reply-To: References: Message-ID: It should still be there at least I still see it in the sources. It's not defined for Silverlight builds (I'm not sure if that's a change or not). One thing to be careful of is that 2.6 and 2.7 do have different .NET assembly versions so they will be treated as different types. In other words you need to re-build a 2.6 pre-compiled app to work against 2.7. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Tuesday, October 12, 2010 8:44 AM To: Discussion of IronPython Subject: [IronPython] What happened to InitializeModule() in IP 2.7? Hi, I'm running the pyc.py IronPython compiler. In the GenerateExe method, there is a call to ....GetMethod("InitializeModule"). This method was available in IP 2.6 but seems to have disappeared in IP 2.7. I used "GetMethods()" to confirm that in IP 2.6 we have: Int32 InitializeModule(System.Reflection.Assembly, System.String, System.String[]) but if I run in IP 2.7, the method is non-existent. This causes the GenerateExe method to fail, and the stub .exe file is not created. Fortunately, the old stub module .exe still works with the new .dll created. Anyone know if there is something else that will work, or if the method has been renamed, or...?????? Thanks, Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Tue Oct 12 19:52:24 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Tue, 12 Oct 2010 13:52:24 -0400 Subject: [IronPython] What happened to InitializeModule() in IP 2.7? In-Reply-To: References: Message-ID: Thanks Dino, OK, after installing my IP 2.7, the only IP .dll's are in c:\program files\ironpython 2.7\Silverlight\bin\*.dll. Are these the "Silverlight builds" you mentioned? If so, how would I get the standard builds? I noticed that in my 2.6 installation, the dll's are in c:\program files\Ironpython 2.6\*.dll, they are NOT in a Silverlight directory. Maybe this is the problem? In any case, the stub application .exe (leftover from a 2.6 build, perhaps?) seems to startup the 2.7 application .dll just fine, but would like to have the whole build process working properly if possible. Ken On Tue, Oct 12, 2010 at 1:30 PM, Dino Viehland wrote: > It should still be there at least I still see it in the sources. It?s > not defined for Silverlight builds (I?m not sure if that?s a change or > not). One thing to be careful of is that 2.6 and 2.7 do have different .NET > assembly versions so they will be treated as different types. In other > words you need to re-build a 2.6 pre-compiled app to work against 2.7. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Tuesday, October 12, 2010 8:44 AM > *To:* Discussion of IronPython > *Subject:* [IronPython] What happened to InitializeModule() in IP 2.7? > > > > Hi, > I'm running the pyc.py IronPython compiler. In the GenerateExe method, > there is a call to ....GetMethod("InitializeModule"). This method was > available in IP 2.6 but seems to have disappeared in IP 2.7. > > I used "GetMethods()" to confirm that in IP 2.6 we have: > > Int32* InitializeModule*(System.Reflection.Assembly, System.String, > System.String[]) > > but if I run in IP 2.7, the method is non-existent. This causes the > GenerateExe method to fail, and the stub .exe file is not created. > Fortunately, the old stub module .exe still works with the new .dll created. > > Anyone know if there is something else that will work, or if the method has > been renamed, or...?????? > Thanks, > Ken > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablodalma93 at hotmail.com Tue Oct 12 20:00:07 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Tue, 12 Oct 2010 15:00:07 -0300 Subject: [IronPython] Bug in copy module (really need help with this!) :) In-Reply-To: References: , , <4CB47421.4070207@voidspace.org.uk>, Message-ID: I just got another error, no idea what triggered it but just reporting it anyway, it happened when copying another object with data from a sql server table. line 191 return memo[d] object reference not set to object instance (sort of, Im translating it from spanish where it says "Referencia a objeto no establecida como instancia de objeto") Greetings From: pablodalma93 at hotmail.com To: michael at voidspace.org.uk; users at lists.ironpython.com Date: Tue, 12 Oct 2010 12:40:13 -0300 Subject: Re: [IronPython] Bug in copy module (really need help with this!) :) Hi Michael, yes, sorry, it's line 102. I have added a couple of lines to "fix" the problem with the module copy not copying System.DBNull types before, it looked like they worked with that problem. (and I erased the commentaries of the module) But this seems to be another problem. I compared both files with Diffuse, the only lines I've added before are import clr clr.AddReference('System') import System (at the begining) and if args[0] is System.DBNull: y = System.DBNull else: y = callable(*args) in line 323 of the original file P.S: this is old but I've read your article on dynamic scope for Python to do something similar to Ruby code blocks, that was quite cool for me :) Date: Tue, 12 Oct 2010 15:43:45 +0100 From: michael at voidspace.org.uk To: users at lists.ironpython.com CC: pablodalma93 at hotmail.com Subject: Re: [IronPython] Bug in copy module (really need help with this!) :) On 12/10/2010 15:29, Pablo Dalmazzo wrote: Hello guys, we've found a bug in the copy module, but the real problem is we cant reproduce it. Sometimes when we try to deepcopy an object of a custom object relation mapping type, we get this error "name types is not defined" Line 55: for t in (type(None),int,long,float,bool,str,tuple), frozenset,type,xrange,types.ClassType,types.BuiltinFunctionType, type(Ellipsis), In my version of Python 2.6 this is line 102 of the copy module. Line 51 is "import types", so a "name types is not defined" error is most perplexing. Is it possible that you are using a custom version of the copy module? All the best, Michael the other problem is in production server, where this error uses to happen sporadically, most times we dont get this message but an error message of the most outer module which calls the copy module, which is a generic message which tells us nothing I know this is very little information to find a bug but, do you know of any bugs related to the module copy (aside it didnt copy System.DBnull which I believe it's not the one for this case) Greetings, Pablo Dalmazzo _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Tue Oct 12 20:01:37 2010 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 12 Oct 2010 18:01:37 +0000 Subject: [IronPython] What happened to InitializeModule() in IP 2.7? In-Reply-To: References: Message-ID: Ahh, yes, in 2.7 we switched to installing the binaries into the GAC so they won't be in C:\Program Files anymore. Instead they should be in C:\Windows\Microsoft.NET\assembly\GAC_MSIL\IronPython\... Also, are you using the 2.7 PYC w/ 2.7 or are you using the 2.6 PYC? I'm not sure it'll make a difference but it's probably best to use the 2.7 PYC if you're not. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Tuesday, October 12, 2010 10:52 AM To: Discussion of IronPython Subject: Re: [IronPython] What happened to InitializeModule() in IP 2.7? Thanks Dino, OK, after installing my IP 2.7, the only IP .dll's are in c:\program files\ironpython 2.7\Silverlight\bin\*.dll. Are these the "Silverlight builds" you mentioned? If so, how would I get the standard builds? I noticed that in my 2.6 installation, the dll's are in c:\program files\Ironpython 2.6\*.dll, they are NOT in a Silverlight directory. Maybe this is the problem? In any case, the stub application .exe (leftover from a 2.6 build, perhaps?) seems to startup the 2.7 application .dll just fine, but would like to have the whole build process working properly if possible. Ken On Tue, Oct 12, 2010 at 1:30 PM, Dino Viehland > wrote: It should still be there at least I still see it in the sources. It's not defined for Silverlight builds (I'm not sure if that's a change or not). One thing to be careful of is that 2.6 and 2.7 do have different .NET assembly versions so they will be treated as different types. In other words you need to re-build a 2.6 pre-compiled app to work against 2.7. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ken MacDonald Sent: Tuesday, October 12, 2010 8:44 AM To: Discussion of IronPython Subject: [IronPython] What happened to InitializeModule() in IP 2.7? Hi, I'm running the pyc.py IronPython compiler. In the GenerateExe method, there is a call to ....GetMethod("InitializeModule"). This method was available in IP 2.6 but seems to have disappeared in IP 2.7. I used "GetMethods()" to confirm that in IP 2.6 we have: Int32 InitializeModule(System.Reflection.Assembly, System.String, System.String[]) but if I run in IP 2.7, the method is non-existent. This causes the GenerateExe method to fail, and the stub .exe file is not created. Fortunately, the old stub module .exe still works with the new .dll created. Anyone know if there is something else that will work, or if the method has been renamed, or...?????? Thanks, Ken _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Tue Oct 12 20:04:23 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Tue, 12 Oct 2010 12:04:23 -0600 Subject: [IronPython] What happened to InitializeModule() in IP 2.7? In-Reply-To: References: Message-ID: On Tue, Oct 12, 2010 at 12:01 PM, Dino Viehland wrote: > Ahh, yes, in 2.7 we switched to installing the binaries into the GAC so they > won?t be in C:\Program Files anymore.? Instead they should be in > C:\Windows\Microsoft.NET\assembly\GAC_MSIL\IronPython\... Or you can download IronPython-2.7A1-Bin.zip from http://ironpython.codeplex.com/releases/view/42434, which has them. Can this be changed, Dino? It's a couple MB on disk and no difference to the installer to have them in both places. - Jeff From dinov at microsoft.com Tue Oct 12 20:09:39 2010 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 12 Oct 2010 18:09:39 +0000 Subject: [IronPython] What happened to InitializeModule() in IP 2.7? In-Reply-To: References: Message-ID: Jeff wrote: > Can this be changed, Dino? It's a couple MB on disk and no difference to the > installer to have them in both places. Yep, it should be easy to change. I'll take a look later this week at doing that. From drken567 at gmail.com Tue Oct 12 20:30:53 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Tue, 12 Oct 2010 14:30:53 -0400 Subject: [IronPython] What happened to InitializeModule() in IP 2.7? In-Reply-To: References: Message-ID: Cool. Seems to work fine w/ the DLL's from GAC_MSIL\IronP\...... the method shows up and the .exe seems to build just fine. I'm using a version of pyc.py based on the 2.7 pyc.py. I've modified it so that can build all of the standard library source modules needed into the application dll, so that you do not need to have the IP standard library on your target system, just the stub .exe, application .dll and the IronPython DLL's. I'll post it at some point after I tune it up a bit; it's a very nice enhancement to the vanilla pyc.py. Ken On Tue, Oct 12, 2010 at 2:01 PM, Dino Viehland wrote: > Ahh, yes, in 2.7 we switched to installing the binaries into the GAC so > they won?t be in C:\Program Files anymore. Instead they should be in > C:\Windows\Microsoft.NET\assembly\GAC_MSIL\IronPython\... > > > > Also, are you using the 2.7 PYC w/ 2.7 or are you using the 2.6 PYC? I?m > not sure it?ll make a difference but it?s probably best to use the 2.7 PYC > if you?re not. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Tuesday, October 12, 2010 10:52 AM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] What happened to InitializeModule() in IP 2.7? > > > > Thanks Dino, > OK, after installing my IP 2.7, the only IP .dll's are in c:\program > files\ironpython 2.7\Silverlight\bin\*.dll. Are these the "Silverlight > builds" you mentioned? If so, how would I get the standard builds? > > I noticed that in my 2.6 installation, the dll's are in c:\program > files\Ironpython 2.6\*.dll, they are NOT in a Silverlight directory. Maybe > this is the problem? > > In any case, the stub application .exe (leftover from a 2.6 build, > perhaps?) seems to startup the 2.7 application .dll just fine, but would > like to have the whole build process working properly if possible. > Ken > > On Tue, Oct 12, 2010 at 1:30 PM, Dino Viehland > wrote: > > It should still be there at least I still see it in the sources. It?s not > defined for Silverlight builds (I?m not sure if that?s a change or not). > One thing to be careful of is that 2.6 and 2.7 do have different .NET > assembly versions so they will be treated as different types. In other > words you need to re-build a 2.6 pre-compiled app to work against 2.7. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald > *Sent:* Tuesday, October 12, 2010 8:44 AM > *To:* Discussion of IronPython > *Subject:* [IronPython] What happened to InitializeModule() in IP 2.7? > > > > Hi, > I'm running the pyc.py IronPython compiler. In the GenerateExe method, > there is a call to ....GetMethod("InitializeModule"). This method was > available in IP 2.6 but seems to have disappeared in IP 2.7. > > I used "GetMethods()" to confirm that in IP 2.6 we have: > > Int32* InitializeModule*(System.Reflection.Assembly, System.String, > System.String[]) > > but if I run in IP 2.7, the method is non-existent. This causes the > GenerateExe method to fail, and the stub .exe file is not created. > Fortunately, the old stub module .exe still works with the new .dll created. > > Anyone know if there is something else that will work, or if the method has > been renamed, or...?????? > Thanks, > Ken > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Tue Oct 12 20:38:08 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Tue, 12 Oct 2010 12:38:08 -0600 Subject: [IronPython] What happened to InitializeModule() in IP 2.7? In-Reply-To: References: Message-ID: On Tue, Oct 12, 2010 at 12:30 PM, Ken MacDonald wrote: > Cool. Seems to work fine w/ the DLL's from GAC_MSIL\IronP\...... the method > shows up and the .exe seems to build just fine. > > I'm using a version of pyc.py based on the 2.7 pyc.py. I've modified it so > that can build all of the standard library source modules needed into the > application dll, so that you do not need to have the IP standard library on > your target system, just the stub .exe, application .dll and the IronPython > DLL's. I'll post it at some point after I tune it up a bit; it's a very nice > enhancement to the vanilla pyc.py. Random crazy thought: can you use ilmerge to pull all of those into one .exe? Horribly bloated, I'm sure, but possibly useful for distribution. - Jeff From pablodalma93 at hotmail.com Tue Oct 12 20:47:00 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Tue, 12 Oct 2010 15:47:00 -0300 Subject: [IronPython] Bug in copy module (really need help with this!) :) In-Reply-To: References: , , , <4CB47421.4070207@voidspace.org.uk>, , , Message-ID: Sorry, it's line 238 :) From: pablodalma93 at hotmail.com To: users at lists.ironpython.com Date: Tue, 12 Oct 2010 15:00:07 -0300 Subject: Re: [IronPython] Bug in copy module (really need help with this!) :) I just got another error, no idea what triggered it but just reporting it anyway, it happened when copying another object with data from a sql server table. line 191 return memo[d] object reference not set to object instance (sort of, Im translating it from spanish where it says "Referencia a objeto no establecida como instancia de objeto") Greetings From: pablodalma93 at hotmail.com To: michael at voidspace.org.uk; users at lists.ironpython.com Date: Tue, 12 Oct 2010 12:40:13 -0300 Subject: Re: [IronPython] Bug in copy module (really need help with this!) :) Hi Michael, yes, sorry, it's line 102. I have added a couple of lines to "fix" the problem with the module copy not copying System.DBNull types before, it looked like they worked with that problem. (and I erased the commentaries of the module) But this seems to be another problem. I compared both files with Diffuse, the only lines I've added before are import clr clr.AddReference('System') import System (at the begining) and if args[0] is System.DBNull: y = System.DBNull else: y = callable(*args) in line 323 of the original file P.S: this is old but I've read your article on dynamic scope for Python to do something similar to Ruby code blocks, that was quite cool for me :) Date: Tue, 12 Oct 2010 15:43:45 +0100 From: michael at voidspace.org.uk To: users at lists.ironpython.com CC: pablodalma93 at hotmail.com Subject: Re: [IronPython] Bug in copy module (really need help with this!) :) On 12/10/2010 15:29, Pablo Dalmazzo wrote: Hello guys, we've found a bug in the copy module, but the real problem is we cant reproduce it. Sometimes when we try to deepcopy an object of a custom object relation mapping type, we get this error "name types is not defined" Line 55: for t in (type(None),int,long,float,bool,str,tuple), frozenset,type,xrange,types.ClassType,types.BuiltinFunctionType, type(Ellipsis), In my version of Python 2.6 this is line 102 of the copy module. Line 51 is "import types", so a "name types is not defined" error is most perplexing. Is it possible that you are using a custom version of the copy module? All the best, Michael the other problem is in production server, where this error uses to happen sporadically, most times we dont get this message but an error message of the most outer module which calls the copy module, which is a generic message which tells us nothing I know this is very little information to find a bug but, do you know of any bugs related to the module copy (aside it didnt copy System.DBnull which I believe it's not the one for this case) Greetings, Pablo Dalmazzo _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From drken567 at gmail.com Tue Oct 12 21:02:05 2010 From: drken567 at gmail.com (Ken MacDonald) Date: Tue, 12 Oct 2010 15:02:05 -0400 Subject: [IronPython] What happened to InitializeModule() in IP 2.7? In-Reply-To: References: Message-ID: > Random crazy thought: can you use ilmerge to pull all of those into > one .exe? Horribly bloated, I'm sure, but possibly useful for > distribution. > > - Jeff > > I was just happy to be able to get all of the source code into DLLs, which meant 1) not having to install IP on customer systems where we want to deploy our app; 2) reducing the possibility (or increasing the difficulty, anyway) of tampering with the app; 3) reducing the possibility that someone would install a different version of IP on their system and suddenly change the behavior of the standard library, and thus our app. This ilmerge idea seems like it would have some interesting application as well, but my goal was to at least get out of distributing any portion of our app as source code. Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Wed Oct 13 09:11:35 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Wed, 13 Oct 2010 09:11:35 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> Message-ID: <5256dbe8c1b389d472a4bee567e6da6c@mail.gmail.com> I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 6:59 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this configurable somehow and only turn it on if you?re actually debugging. It?s unfortunate that we can?t offer both debugging & collectability but right now that?s simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 9:09 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Wed Oct 13 15:48:10 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Wed, 13 Oct 2010 15:48:10 +0200 Subject: [IronPython] Possible WeakReference leak in Ipy 2.6.1 Message-ID: <00c635b41649eee3ff93ada75e4b8e41@mail.gmail.com> When definnning classes in a method that are subcallses, like: def d(): class A(BaseType): pass ? The new generated types is appended to the _subtypes weak reference list on BaseType. Looks like this list is only cleaned when RemoveSubType Is called, which might never happen if I understand correctly. Seems to me there should be a cleanup mechanism similar to WeakDictionary, so the list can't grow to infinity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From merllab at microsoft.com Wed Oct 13 18:04:49 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Wed, 13 Oct 2010 09:04:49 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: <62b94765-cd57-4dc8-88a7-563969939e93@tk5-exsmh-c101.redmond.corp.microsoft.com> This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/78371. MODIFIED SOURCES $/IronPython/IronPython_Main/Languages/Ruby/ClassInitGenerator/Libraries/LibraryDef.cs $/IronPython/IronPython_Main/Languages/Ruby/IronRuby.Tests/RubyTests.cs $/IronPython/IronPython_Main/Languages/Ruby/IronRuby.Tests/Runtime/BlockTests.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Initializers.Generated.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Builtins/ArrayOps.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Builtins/Comparable.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Builtins/Enumerable.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Builtins/MutableStringOps.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Builtins/RangeOps.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Extensions/IDictionaryOps.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Extensions/IListOps.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/socket/Socket.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Compiler/Generation/RubyTypeEmitter.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/BlockParam.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/CallSiteStorages.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/Protocols.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/RubyContext.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/Calls/BlockDispatcherN.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/Calls/BlockDispatcherProcN.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/Calls/BlockDispatchers.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/Calls/BlockDispatcherUnsplatN.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/Calls/BlockDispatcherUnsplatProcN.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/Calls/RubyMetaBinderFactory.cs $/IronPython/IronPython_Main/Runtime/Microsoft.Scripting/Runtime/Scope.cs From pablodalma93 at hotmail.com Wed Oct 13 20:51:28 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Wed, 13 Oct 2010 15:51:28 -0300 Subject: [IronPython] expected behavior of the module copy with .net objects Message-ID: Hi guys, Sorry for being annoying about this but I just want to know where I'm standing at :) It would seem to me the copy module has some problems copying .net objects in general. I was wondering if I'm pretending too much for that module and it will never fully copy objects instantiated from .net classes, or it's a goal for it to do it someday, or I just happened to find a couple of errors, or Im not using it rightly for what it was meant to be. I dont know 5% of what you know so it's an honest question :) Aside there are a couple of errors I cant reproduce, and that it didnt copy System.DBNull , now I see it loses property values when copyinga System.Data.SqlConnection object, properties such as "DataSource", "DataBase", etc. they get empty strings after being copied. Just in caseI tried with the original version of the module/file copy. I guess you have thousands of things to check but I just want to know, it's the module supposed to copy any object instantiated from .net classes or it's only supposed to fully work only for custom made classes? Greetings, Pablo Dalmazzo P.S: sorry about insisting again with this but I've also added another example where it seems to fail :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Wed Oct 13 21:47:04 2010 From: dinov at microsoft.com (Dino Viehland) Date: Wed, 13 Oct 2010 19:47:04 +0000 Subject: [IronPython] expected behavior of the module copy with .net objects In-Reply-To: References: Message-ID: I think as of now copy is certainly not suitable for all .NET objects. For starters not all .NET objects are serializable/copyable. There should probably be some effort to close the gaps where it makes sense. For example something which implements ICloneable could have a __copy__ method added to it. This is similar to how we add a __reduce_ex__ method for things which implement ISerializable. But if something doesn't implement either of these interfaces then there's no standard way for us to copy these objects. If you have specific object types which you know you need to copy I would suggest registering the type with copy_reg. That should at least give you a solution for your specific scenarios. You also mentioned the null reference exception ("object reference not set to object instance..."). That sounds like a bug - I would guess you're running in a multi-threaded environment? If you could get a repro of this that'd be great as it looks like it could be an issue w/ our dictionary implementation not being thread safe. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Pablo Dalmazzo Sent: Wednesday, October 13, 2010 11:51 AM To: IronPython Mailing list Subject: [IronPython] expected behavior of the module copy with .net objects Hi guys, Sorry for being annoying about this but I just want to know where I'm standing at :) It would seem to me the copy module has some problems copying .net objects in general. I was wondering if I'm pretending too much for that module and it will never fully copy objects instantiated from .net classes, or it's a goal for it to do it someday, or I just happened to find a couple of errors, or Im not using it rightly for what it was meant to be. I dont know 5% of what you know so it's an honest question :) Aside there are a couple of errors I cant reproduce, and that it didnt copy System.DBNull , now I see it loses property values when copying a System.Data.SqlConnection object, properties such as "DataSource", "DataBase", etc. they get empty strings after being copied. Just in case I tried with the original version of the module/file copy. I guess you have thousands of things to check but I just want to know, it's the module supposed to copy any object instantiated from .net classes or it's only supposed to fully work only for custom made classes? Greetings, Pablo Dalmazzo P.S: sorry about insisting again with this but I've also added another example where it seems to fail :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Wed Oct 13 21:48:41 2010 From: dinov at microsoft.com (Dino Viehland) Date: Wed, 13 Oct 2010 19:48:41 +0000 Subject: [IronPython] Possible WeakReference leak in Ipy 2.6.1 In-Reply-To: <00c635b41649eee3ff93ada75e4b8e41@mail.gmail.com> References: <00c635b41649eee3ff93ada75e4b8e41@mail.gmail.com> Message-ID: Yep, this definitely looks like an issue - could you open a bug? We should probably have a check when adding which causes us to scan the list occasionally and remove any dead types. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Wednesday, October 13, 2010 6:48 AM To: Discussion of IronPython Subject: [IronPython] Possible WeakReference leak in Ipy 2.6.1 When definnning classes in a method that are subcallses, like: def d(): class A(BaseType): pass ... The new generated types is appended to the _subtypes weak reference list on BaseType. Looks like this list is only cleaned when RemoveSubType Is called, which might never happen if I understand correctly. Seems to me there should be a cleanup mechanism similar to WeakDictionary, so the list can't grow to infinity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Wed Oct 13 22:01:05 2010 From: dinov at microsoft.com (Dino Viehland) Date: Wed, 13 Oct 2010 20:01:05 +0000 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: <5256dbe8c1b389d472a4bee567e6da6c@mail.gmail.com> References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> <5256dbe8c1b389d472a4bee567e6da6c@mail.gmail.com> Message-ID: If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs and DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we're continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspx leads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there's exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There's also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn't actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they're compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it's likely that it is one of the things I mentioned (or some other cache that's per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That's a bunch of different things to look at - hopefully it'll give some insight into what's going on and help track down the issue. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Wednesday, October 13, 2010 12:12 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 6:59 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as -X:Debug. In general I'd suggest making this configurable somehow and only turn it on if you're actually debugging. It's unfortunate that we can't offer both debugging & collectability but right now that's simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 9:09 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like -X:Debug? You reckon this could be the cause? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 5:53 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? My guess is that's code in the JIT heap that's building up but I'm not 100% certain. How is your code being executed? Do you have the debug option (-D or -X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 2:26 AM To: Discussion of IronPython Subject: [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From merllab at microsoft.com Wed Oct 13 23:48:32 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Wed, 13 Oct 2010 14:48:32 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/78395. ADDED SOURCES $/IronPython/IronPython_Main/Languages/IronPython/Internal $/IronPython/IronPython_Main/Languages/IronPython/Internal/cpy.bat $/IronPython/IronPython_Main/Languages/IronPython/Internal/ipy.bat MODIFIED SOURCES $/IronPython/IronPython_Main/Languages/IronPython/Internal/cpy.bat $/IronPython/IronPython_Main/Languages/IronPython/Internal/ipy.bat From grauenwolf at gmail.com Thu Oct 14 07:49:37 2010 From: grauenwolf at gmail.com (Jonathan Allen) Date: Wed, 13 Oct 2010 22:49:37 -0700 Subject: [IronPython] Will this leak memory? Message-ID: <000001cb6b63$9749c680$c5dd5380$@gmail.com> I've got a massive memory leak in my program. This is the first time I've used IronPython in a tight loop, so I'm wondering if this could be the cause. For Each column In m_Columns Dim rawValue As String rawValue = line.Substring(column.FileColumnIndex.Value - 1, column.FileColumnEndIndex.Value - column.FileColumnIndex.Value + 1) If column.IncludeColumnFunction <> "" Then Dim scope = m_ScriptEngine.CreateScope scope.SetVariable("Line", line) scope.SetVariable("Row", targetRow) If Not CBool(m_ScriptEngine.Execute(column.IncludeColumnFunction, scope)) Then Continue For 'skip this column End If targetRow(column.DatabaseColumnName) = column.ParseValue(rawValue, targetRow) Next The string named column.IncludeColumnFunction never changes for a given column. It is usually something simple like "Row['Foo'] == 'Bar'". Can I/should I be caching the compiled function? Should I be destroying the scope variable in some way when I'm done with it? Jonathan Allen 619-933-8527 -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Thu Oct 14 08:09:15 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Thu, 14 Oct 2010 08:09:15 +0200 Subject: [IronPython] Possible WeakReference leak in Ipy 2.6.1 In-Reply-To: References: <00c635b41649eee3ff93ada75e4b8e41@mail.gmail.com> Message-ID: <26cd82f0bff401899a127bb0af63dd84@mail.gmail.com> opened *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Wednesday, October 13, 2010 9:49 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] Possible WeakReference leak in Ipy 2.6.1 Yep, this definitely looks like an issue ? could you open a bug? We should probably have a check when adding which causes us to scan the list occasionally and remove any dead types. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Wednesday, October 13, 2010 6:48 AM *To:* Discussion of IronPython *Subject:* [IronPython] Possible WeakReference leak in Ipy 2.6.1 When definnning classes in a method that are subcallses, like: def d(): class A(BaseType): pass ? The new generated types is appended to the _subtypes weak reference list on BaseType. Looks like this list is only cleaned when RemoveSubType Is called, which might never happen if I understand correctly. Seems to me there should be a cleanup mechanism similar to WeakDictionary, so the list can't grow to infinity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Thu Oct 14 08:14:29 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Thu, 14 Oct 2010 08:14:29 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> <5256dbe8c1b389d472a4bee567e6da6c@mail.gmail.com> Message-ID: <0e6bba9fdfecd816a8ae0bbfc45d1e7d@mail.gmail.com> Thanks for the elaborate answer, I'll look into it. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Wednesday, October 13, 2010 10:01 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs *and*DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we?re continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there?s exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There?s also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn?t actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they?re compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it?s likely that it is one of the things I mentioned (or some other cache that?s per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That?s a bunch of different things to look at ? hopefully it?ll give some insight into what?s going on and help track down the issue. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Wednesday, October 13, 2010 12:12 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 6:59 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this configurable somehow and only turn it on if you?re actually debugging. It?s unfortunate that we can?t offer both debugging & collectability but right now that?s simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 9:09 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Thu Oct 14 08:26:35 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Thu, 14 Oct 2010 08:26:35 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> <5256dbe8c1b389d472a4bee567e6da6c@mail.gmail.com> Message-ID: <83f7bfb2c25ec57e2b194aaec3a8205d@mail.gmail.com> I should have mentioned that I can't run in debug mode. On the other hand I do have a windbg sessions on my application (one session when the heap is small and one when it is big). Can you tell me the object types that I should like for in the managed code, such that if their number is increasing then I can tell they are responsible for the leaks? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Wednesday, October 13, 2010 10:01 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs *and*DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we?re continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there?s exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There?s also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn?t actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they?re compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it?s likely that it is one of the things I mentioned (or some other cache that?s per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That?s a bunch of different things to look at ? hopefully it?ll give some insight into what?s going on and help track down the issue. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Wednesday, October 13, 2010 12:12 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 6:59 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this configurable somehow and only turn it on if you?re actually debugging. It?s unfortunate that we can?t offer both debugging & collectability but right now that?s simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 9:09 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablodalma93 at hotmail.com Thu Oct 14 13:48:33 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Thu, 14 Oct 2010 08:48:33 -0300 Subject: [IronPython] expected behavior of the module copy with .net objects In-Reply-To: References: , Message-ID: thanks for the reply Dino, If I get any additional info about it, I'll post it. In one application (which triggered the error you mentioned it) Im using an asp.net timer. That runs in another thread, right? Greetings, Pablo From: dinov at microsoft.com To: users at lists.ironpython.com Date: Wed, 13 Oct 2010 19:47:04 +0000 Subject: Re: [IronPython] expected behavior of the module copy with .net objects I think as of now copy is certainly not suitable for all .NET objects. For starters not all .NET objects are serializable/copyable. There should probably be some effort to close the gaps where it makes sense. For example something which implements ICloneable could have a __copy__ method added to it. This is similar to how we add a __reduce_ex__ method for things which implement ISerializable. But if something doesn?t implement either of these interfaces then there?s no standard way for us to copy these objects. If you have specific object types which you know you need to copy I would suggest registering the type with copy_reg. That should at least give you a solution for your specific scenarios. You also mentioned the null reference exception (?object reference not set to object instance??). That sounds like a bug ? I would guess you?re running in a multi-threaded environment? If you could get a repro of this that?d be great as it looks like it could be an issue w/ our dictionary implementation not being thread safe. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Pablo Dalmazzo Sent: Wednesday, October 13, 2010 11:51 AM To: IronPython Mailing list Subject: [IronPython] expected behavior of the module copy with .net objects Hi guys, Sorry for being annoying about this but I just want to know where I'm standing at :) It would seem to me the copy module has some problems copying .net objects in general. I was wondering if I'm pretending too much for that module and it will never fully copy objects instantiated from .net classes, or it's a goal for it to do it someday, or I just happened to find a couple of errors, or Im not using it rightly for what it was meant to be. I dont know 5% of what you know so it's an honest question :) Aside there are a couple of errors I cant reproduce, and that it didnt copy System.DBNull , now I see it loses property values when copying a System.Data.SqlConnection object, properties such as "DataSource", "DataBase", etc. they get empty strings after being copied. Just in case I tried with the original version of the module/file copy. I guess you have thousands of things to check but I just want to know, it's the module supposed to copy any object instantiated from .net classes or it's only supposed to fully work only for custom made classes? Greetings, Pablo Dalmazzo P.S: sorry about insisting again with this but I've also added another example where it seems to fail :) _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Thu Oct 14 15:44:18 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Thu, 14 Oct 2010 15:44:18 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> <5256dbe8c1b389d472a4bee567e6da6c@mail.gmail.com> Message-ID: <505150d4deb7d86b62a8da32003741de@mail.gmail.com> Hi, I've been looking on this for some time, and I'm still don?t understand some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET 2.0) a bit better. We have a long running (stateful) application that we cannot simulate or run with a debugger open (so no breakpoints). However, since the application is run on a VM, we can take snapshots of it and then open WinDbg instance and break in the middle of the application. We do this a few hours after the application restarted and again after two days so we can see the difference. This way we saw that Jit Code Heap is increasing by a few hundred MB per day, and the number of HostCodeHeap objects is increasing. We also compared the performance counters for the two snapshots, and saw the Jitted Code Bytes increased from 100MB to 862MB and the number of methods jitted increased from 700K to 6.3M. In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to 424MB. On the other hand, the object types you mentioned stay relatively the same: ? DynamicMethods count went from 16K to 18K ? FunctionCode count went from 4013 to 4025 ? The _*code*Count field in the PythonContext went from 4447 to 7800 Here is what we don't understand: 1. Is it normal for the application to keep jitting code and methods forever? Should is stabilize? 2. From the numbers I guess that some of the jitted code IS collected. Which types are collectable and which are not? How can I tell which ones I am using? 3. Are there any specific patterns I should avoid to decrease uncollectable code (or jitting in general). I am using a lot of closures, callbacks and generators. 4. What datatypes that are visible in WinDbg can I use to understand if and why IronPython is generating uncollectable code? 5. Is there a way to trace back the HostCodeHeap objects to my code (or IronPython specific features)? 6. Can I expect an improvement in these issues by moving to Ipy 2.7 and/or .Net 4.0? Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Wednesday, October 13, 2010 10:01 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs *and*DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we?re continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there?s exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There?s also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn?t actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they?re compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it?s likely that it is one of the things I mentioned (or some other cache that?s per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That?s a bunch of different things to look at ? hopefully it?ll give some insight into what?s going on and help track down the issue. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Wednesday, October 13, 2010 12:12 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 6:59 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this configurable somehow and only turn it on if you?re actually debugging. It?s unfortunate that we can?t offer both debugging & collectability but right now that?s simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 9:09 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From rawatsaurabh at yahoo.co.in Thu Oct 14 15:48:37 2010 From: rawatsaurabh at yahoo.co.in (saurabh rawat) Date: Thu, 14 Oct 2010 19:18:37 +0530 (IST) Subject: [IronPython] Getting an error while Passing Dictionary to a function Message-ID: <358830.38983.qm@web137403.mail.in.yahoo.com> I Am trying to pass a dictionary element to a function but even though the function takes 3 arguments , it is showing error as ERROR: Func() takes exactly 4 arguments (3 given). Why is it happening.is there a problem in Dictionary Porting from .NET. to IronPython ?public struct callCounters ??????? { ??????????? public double callReceivedTimer; ??????????? public double callDialledTimer; ??????????? public double callAllTimer; ??????????? public double callLifeTimer; ??????? } callValue = Dictionary[DateTime, callCounters]() prototype for Func() void func(string casename, DateTime time, Dictionary <,) Function call func(casename, time, callValue ) ERROR: Func() takes exactly 4 arguments (3 given). Plz assist. Rgds, Saurabh -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Thu Oct 14 16:49:31 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Thu, 14 Oct 2010 08:49:31 -0600 Subject: [IronPython] Getting an error while Passing Dictionary to a function In-Reply-To: <358830.38983.qm@web137403.mail.in.yahoo.com> References: <358830.38983.qm@web137403.mail.in.yahoo.com> Message-ID: Hi Saurabh, Is Func() an instance method of a class, or is it a static function? - Jeff On Thu, Oct 14, 2010 at 7:48 AM, saurabh rawat wrote: > > I Am trying to pass a dictionary element to a function but even though the function takes 3 arguments , it is showing error as ERROR: Func() takes exactly 4 arguments (3 given). > Why is it happening.is there a problem in Dictionary Porting from .NET. to IronPython > > ?public struct callCounters > ??????? { > ??????????? public double callReceivedTimer; > ??????????? public double callDialledTimer; > ??????????? public double callAllTimer; > ??????????? public double callLifeTimer; > ??????? } > > callValue = Dictionary[DateTime, callCounters]() > > prototype for Func() > > void func(string casename, DateTime time, Dictionary <,) > > Function call > > func(casename, time, callValue ) > > ERROR: Func() takes exactly 4 arguments (3 given). > > > Plz assist. > Rgds, > Saurabh > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > From dinov at microsoft.com Thu Oct 14 19:58:51 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 14 Oct 2010 17:58:51 +0000 Subject: [IronPython] expected behavior of the module copy with .net objects In-Reply-To: References: , Message-ID: Ahh, well if it's ASP.NET it's all multi-threaded because you're probably handling more than one request at a time. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Pablo Dalmazzo Sent: Thursday, October 14, 2010 4:49 AM To: IronPython Mailing list Subject: Re: [IronPython] expected behavior of the module copy with .net objects thanks for the reply Dino, If I get any additional info about it, I'll post it. In one application (which triggered the error you mentioned it) Im using an asp.net timer. That runs in another thread, right? Greetings, Pablo ________________________________ From: dinov at microsoft.com To: users at lists.ironpython.com Date: Wed, 13 Oct 2010 19:47:04 +0000 Subject: Re: [IronPython] expected behavior of the module copy with .net objects I think as of now copy is certainly not suitable for all .NET objects. For starters not all .NET objects are serializable/copyable. There should probably be some effort to close the gaps where it makes sense. For example something which implements ICloneable could have a __copy__ method added to it. This is similar to how we add a __reduce_ex__ method for things which implement ISerializable. But if something doesn't implement either of these interfaces then there's no standard way for us to copy these objects. If you have specific object types which you know you need to copy I would suggest registering the type with copy_reg. That should at least give you a solution for your specific scenarios. You also mentioned the null reference exception ("object reference not set to object instance..."). That sounds like a bug - I would guess you're running in a multi-threaded environment? If you could get a repro of this that'd be great as it looks like it could be an issue w/ our dictionary implementation not being thread safe. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Pablo Dalmazzo Sent: Wednesday, October 13, 2010 11:51 AM To: IronPython Mailing list Subject: [IronPython] expected behavior of the module copy with .net objects Hi guys, Sorry for being annoying about this but I just want to know where I'm standing at :) It would seem to me the copy module has some problems copying .net objects in general. I was wondering if I'm pretending too much for that module and it will never fully copy objects instantiated from .net classes, or it's a goal for it to do it someday, or I just happened to find a couple of errors, or Im not using it rightly for what it was meant to be. I dont know 5% of what you know so it's an honest question :) Aside there are a couple of errors I cant reproduce, and that it didnt copy System.DBNull , now I see it loses property values when copying a System.Data.SqlConnection object, properties such as "DataSource", "DataBase", etc. they get empty strings after being copied. Just in case I tried with the original version of the module/file copy. I guess you have thousands of things to check but I just want to know, it's the module supposed to copy any object instantiated from .net classes or it's only supposed to fully work only for custom made classes? Greetings, Pablo Dalmazzo P.S: sorry about insisting again with this but I've also added another example where it seems to fail :) _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From merllab at microsoft.com Thu Oct 14 20:20:23 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Thu, 14 Oct 2010 11:20:23 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: <401c575d-a8b7-406b-81e5-63ed498ff301@tk5-exsmh-c101.redmond.corp.microsoft.com> This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/78429. ADDED SOURCES $/IronPython/IronPython_Main/Util/Misc $/IronPython/IronPython_Main/Util/Misc/zip.exe MODIFIED SOURCES $/IronPython/IronPython_Main/Languages/IronPython/StdLib/StdLib.pyproj $/IronPython/IronPython_Main/Util/Misc/zip.exe From dinov at microsoft.com Thu Oct 14 20:31:07 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 14 Oct 2010 18:31:07 +0000 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: <505150d4deb7d86b62a8da32003741de@mail.gmail.com> References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> <5256dbe8c1b389d472a4bee567e6da6c@mail.gmail.com> <505150d4deb7d86b62a8da32003741de@mail.gmail.com> Message-ID: So from an IronPython/DLR perspective the process should stabilize over time. That could take a while - as various Python functions get used repeatedly we'll switch from interpreting them to compiling them. We'll also potentially produce new call site rules which are compiled. That could account for the increase of 16k->18k dynamic methods. That's a 12% increase and could be a reasonable amount of it remained steady from then on. Likewise the # of function codes seems stable but the _codeCount is rising. That might mean that you're defining new functions (via exec/execfile/compile) and they're getting collected - but we still have their (dead) weak references in the code list. Eventually that list should get cleaned out when we hit context._nextCodeCleanup (which should be greater than context._codeCount). I would expect if you were to walk the entire PythonContext._allCodes linked list that you'd see about half of the lists having a dead weak reference. If your windbg-foo is up for writing the script to do this that'd be great but mine is rusty enough I'd need to look it up in the documentation. As for jitted code - all dynamic methods are collectible and any normal RefEmit code is not collectible (there's an option to make assemblies collectible in .NET 4.0, but we don't use it as we generally don't generate that many types). Oh, we do also generate new types for subclasses but you should see that NewTypeMaker._newTypes has a stable count over time because we share these between types w/ common bases. Closures, callbacks, generators should all be fine. If you do .symfix, then .reload, in Windbg does "dt HostCodeHeap" show you the fields of the HostCodeHeap structure? Does !eeheap -loader give you the address of the HostCodeHeap? It might be useful to know what m_allocationCount is on the heap overtime as if that's relatively stable the heap could be getting fragmented. Looking at the CLR code I'm becoming more and more convinced HostCodeHeap is only used for dynamic methods so if that's growing then I'm thinking we would appear to be leaking dynamic methods. The only way I can think of figuring out where the allocations are coming from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow (hopefully this will show up in the public symbols). That may be difficult if you can't attach the debugger to the server but if the issue is blocking the server you could have a breakpoint here which does a stack trace and continues execution so you could inspect the stack trace later. I'd include both "kb" and "!ClrStack" from sos in the stack trace. 2.7 shouldn't really change this - whether .NET 4.0 would or not would depend on if the CLR changed anything here. But I'm not sure - I would assume it wouldn't. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Thursday, October 14, 2010 6:44 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Hi, I've been looking on this for some time, and I'm still don't understand some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET 2.0) a bit better. We have a long running (stateful) application that we cannot simulate or run with a debugger open (so no breakpoints). However, since the application is run on a VM, we can take snapshots of it and then open WinDbg instance and break in the middle of the application. We do this a few hours after the application restarted and again after two days so we can see the difference. This way we saw that Jit Code Heap is increasing by a few hundred MB per day, and the number of HostCodeHeap objects is increasing. We also compared the performance counters for the two snapshots, and saw the Jitted Code Bytes increased from 100MB to 862MB and the number of methods jitted increased from 700K to 6.3M. In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to 424MB. On the other hand, the object types you mentioned stay relatively the same: * DynamicMethods count went from 16K to 18K * FunctionCode count went from 4013 to 4025 * The _codeCount field in the PythonContext went from 4447 to 7800 Here is what we don't understand: 1. Is it normal for the application to keep jitting code and methods forever? Should is stabilize? 2. From the numbers I guess that some of the jitted code IS collected. Which types are collectable and which are not? How can I tell which ones I am using? 3. Are there any specific patterns I should avoid to decrease uncollectable code (or jitting in general). I am using a lot of closures, callbacks and generators. 4. What datatypes that are visible in WinDbg can I use to understand if and why IronPython is generating uncollectable code? 5. Is there a way to trace back the HostCodeHeap objects to my code (or IronPython specific features)? 6. Can I expect an improvement in these issues by moving to Ipy 2.7 and/or .Net 4.0? Thanks From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Wednesday, October 13, 2010 10:01 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs and DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we're continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspx leads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there's exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There's also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn't actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they're compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it's likely that it is one of the things I mentioned (or some other cache that's per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That's a bunch of different things to look at - hopefully it'll give some insight into what's going on and help track down the issue. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Wednesday, October 13, 2010 12:12 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 6:59 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as -X:Debug. In general I'd suggest making this configurable somehow and only turn it on if you're actually debugging. It's unfortunate that we can't offer both debugging & collectability but right now that's simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 9:09 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like -X:Debug? You reckon this could be the cause? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 5:53 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? My guess is that's code in the JIT heap that's building up but I'm not 100% certain. How is your code being executed? Do you have the debug option (-D or -X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 2:26 AM To: Discussion of IronPython Subject: [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From merllab at microsoft.com Thu Oct 14 21:25:38 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Thu, 14 Oct 2010 12:25:38 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/78431. ADDED SOURCES $/IronPython/IronPython_Main/Languages/IronPython/IronPython/Runtime/PythonHiddenBaseClassAttribute.cs MODIFIED SOURCES $/IronPython/IronPython_Main/Languages/IronPython/IronPython/IronPython.csproj $/IronPython/IronPython_Main/Languages/IronPython/IronPython/Runtime/PythonBuffer.cs $/IronPython/IronPython_Main/Languages/IronPython/IronPython/Runtime/PythonHiddenBaseClassAttribute.cs $/IronPython/IronPython_Main/Languages/IronPython/IronPython/Runtime/Binding/PythonBinder.cs $/IronPython/IronPython_Main/Languages/IronPython/IronPython/Runtime/Types/PythonType.cs $/IronPython/IronPython_Main/Languages/IronPython/IronPython/Runtime/Types/TypeInfo.cs $/IronPython/IronPython_Main/Languages/IronPython/IronPythonTest/EngineTest.cs $/IronPython/IronPython_Main/Languages/IronPython/Tests/test_ipye.py $/IronPython/IronPython_Main/Runtime/Microsoft.Dynamic/Actions/ActionBinder.cs From pablodalma93 at hotmail.com Thu Oct 14 21:34:01 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Thu, 14 Oct 2010 16:34:01 -0300 Subject: [IronPython] expected behavior of the module copy with .net objects In-Reply-To: References: , , , , Message-ID: You know, now I remember that twice I got errors that I thought they might have been generated by the copy module, and when one of those errors triggered in one page, several other different asp.net pages started to do the same at the same time. if they came from the copy module, could that be for the dictionary implementation not being thread safe as you mentioned? I'm not 100% if those 2 times the error came from the copy module but it was very likely. I was in production server, the copy module was called from an inner module and I got the error message from the outer module and err... I didnt know how to go down to the inner error in production server. In development I get the errors from the module where the error is actually placed Greetings, Pablo From: dinov at microsoft.com To: users at lists.ironpython.com Date: Thu, 14 Oct 2010 17:58:51 +0000 Subject: Re: [IronPython] expected behavior of the module copy with .net objects Ahh, well if it?s ASP.NET it?s all multi-threaded because you?re probably handling more than one request at a time. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Pablo Dalmazzo Sent: Thursday, October 14, 2010 4:49 AM To: IronPython Mailing list Subject: Re: [IronPython] expected behavior of the module copy with .net objects thanks for the reply Dino, If I get any additional info about it, I'll post it. In one application (which triggered the error you mentioned it) Im using an asp.net timer. That runs in another thread, right? Greetings, Pablo From: dinov at microsoft.com To: users at lists.ironpython.com Date: Wed, 13 Oct 2010 19:47:04 +0000 Subject: Re: [IronPython] expected behavior of the module copy with .net objects I think as of now copy is certainly not suitable for all .NET objects. For starters not all .NET objects are serializable/copyable. There should probably be some effort to close the gaps where it makes sense. For example something which implements ICloneable could have a __copy__ method added to it. This is similar to how we add a __reduce_ex__ method for things which implement ISerializable. But if something doesn?t implement either of these interfaces then there?s no standard way for us to copy these objects. If you have specific object types which you know you need to copy I would suggest registering the type with copy_reg. That should at least give you a solution for your specific scenarios. You also mentioned the null reference exception (?object reference not set to object instance??). That sounds like a bug ? I would guess you?re running in a multi-threaded environment? If you could get a repro of this that?d be great as it looks like it could be an issue w/ our dictionary implementation not being thread safe. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Pablo Dalmazzo Sent: Wednesday, October 13, 2010 11:51 AM To: IronPython Mailing list Subject: [IronPython] expected behavior of the module copy with .net objects Hi guys, Sorry for being annoying about this but I just want to know where I'm standing at :) It would seem to me the copy module has some problems copying .net objects in general. I was wondering if I'm pretending too much for that module and it will never fully copy objects instantiated from .net classes, or it's a goal for it to do it someday, or I just happened to find a couple of errors, or Im not using it rightly for what it was meant to be. I dont know 5% of what you know so it's an honest question :) Aside there are a couple of errors I cant reproduce, and that it didnt copy System.DBNull , now I see it loses property values when copying a System.Data.SqlConnection object, properties such as "DataSource", "DataBase", etc. they get empty strings after being copied. Just in case I tried with the original version of the module/file copy. I guess you have thousands of things to check but I just want to know, it's the module supposed to copy any object instantiated from .net classes or it's only supposed to fully work only for custom made classes? Greetings, Pablo Dalmazzo P.S: sorry about insisting again with this but I've also added another example where it seems to fail :) _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rawatsaurabh at yahoo.co.in Fri Oct 15 06:35:29 2010 From: rawatsaurabh at yahoo.co.in (saurabh rawat) Date: Fri, 15 Oct 2010 10:05:29 +0530 (IST) Subject: [IronPython] Getting an error while Passing Dictionary to a function In-Reply-To: Message-ID: <511344.44510.qm@web137412.mail.in.yahoo.com> ????????? HI Jeff, Its an instance method of a class..Dont think/Even tried passing self to it .Still R&D ing on it..!! Saurabh ?????????? ?????????? ?????????????? " The ultimate test of a relationship is to disagree but to hold hands..............."? ?????????????? ? --- On Thu, 14/10/10, Jeff Hardy wrote: From: Jeff Hardy Subject: Re: [IronPython] Getting an error while Passing Dictionary to a function To: "Discussion of IronPython" Date: Thursday, 14 October, 2010, 8:19 PM Hi Saurabh, Is Func() an instance method of a class, or is it a static function? - Jeff On Thu, Oct 14, 2010 at 7:48 AM, saurabh rawat wrote: > > I Am trying to pass a dictionary element to a function but even though the function takes 3 arguments , it is showing error as ERROR: Func() takes exactly 4 arguments (3 given). > Why is it happening.is there a problem in Dictionary Porting from .NET. to IronPython > > ?public struct callCounters > ??????? { > ??????????? public double callReceivedTimer; > ??????????? public double callDialledTimer; > ??????????? public double callAllTimer; > ??????????? public double callLifeTimer; > ??????? } > > callValue = Dictionary[DateTime, callCounters]() > > prototype for Func() > > void func(string casename, DateTime time, Dictionary <,) > > Function call > > func(casename, time, callValue ) > > ERROR: Func() takes exactly 4 arguments (3 given). > > > Plz assist. > Rgds, > Saurabh > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Fri Oct 15 18:00:19 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Fri, 15 Oct 2010 10:00:19 -0600 Subject: [IronPython] Getting an error while Passing Dictionary to a function In-Reply-To: <511344.44510.qm@web137412.mail.in.yahoo.com> References: <511344.44510.qm@web137412.mail.in.yahoo.com> Message-ID: Can you show the exact code used to call func? Is it being called inside the class (i.e. from another member of the same class) or from outside the class? - Jeff On Thu, Oct 14, 2010 at 10:35 PM, saurabh rawat wrote: > > > ????????? HI Jeff, > Its an instance method of a class..Dont think/Even tried passing self to it .Still R&D ing on it..!! > Saurabh > > > ?????????????? " The ultimate test of a relationship is to disagree but to hold hands..............." > > > > --- On Thu, 14/10/10, Jeff Hardy wrote: > > From: Jeff Hardy > Subject: Re: [IronPython] Getting an error while Passing Dictionary to a function > To: "Discussion of IronPython" > Date: Thursday, 14 October, 2010, 8:19 PM > > Hi Saurabh, > Is Func() an instance method of a class, or is it a static function? > - Jeff > > On Thu, Oct 14, 2010 at 7:48 AM, saurabh rawat wrote: > > > > I Am trying to pass a dictionary element to a function but even though the function takes 3 arguments , it is showing error as ERROR: Func() takes exactly 4 arguments (3 given). > > Why is it happening.is there a problem in Dictionary Porting from .NET. to IronPython > > > > ?public struct callCounters > > ??????? { > > ??????????? public double callReceivedTimer; > > ??????????? public double callDialledTimer; > > ??????????? public double callAllTimer; > > ??????????? public double callLifeTimer; > > ??????? } > > > > callValue = Dictionary[DateTime, callCounters]() > > > > prototype for Func() > > > > void func(string casename, DateTime time, Dictionary <,) > > > > Function call > > > > func(casename, time, callValue ) > > > > ERROR: Func() takes exactly 4 arguments (3 given). > > > > > > Plz assist. > > Rgds, > > Saurabh > > > > > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > From merllab at microsoft.com Fri Oct 15 18:04:42 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Fri, 15 Oct 2010 09:04:42 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: <45e00653-ab2e-46ad-b068-dd87712b2813@tk5-exsmh-c101.redmond.corp.microsoft.com> This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/78463. MODIFIED SOURCES $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Initializers.Generated.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Builtins/ArgFilesSingletonOps.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/Builtins/IoOps.cs $/IronPython/IronPython_Main/Languages/Ruby/Libraries.LCA_RESTRICTED/StringIO/StringIO.cs $/IronPython/IronPython_Main/Languages/Ruby/Tests/Scripts/irtests.rb $/IronPython/IronPython_Main/Languages/Ruby/Tests/Scripts/utr/gem_tests.rb From idan at cloudshare.com Sun Oct 17 16:28:48 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Sun, 17 Oct 2010 16:28:48 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> <5256dbe8c1b389d472a4bee567e6da6c@mail.gmail.com> <505150d4deb7d86b62a8da32003741de@mail.gmail.com> Message-ID: <959da5712511d9b5c63c66dbdfb1928f@mail.gmail.com> I tried most of what you suggested but it didn?t work for me (I am probably doing something wrong). The only thing I can say for sure is that the number jitted methods(in the perfmon) is raising constantly. This is something that is reproduced even when I run a fraction of the system on my local machine. Is there any way to find what these methods are? I thought about dumping the VTable size of each type and see who is getting bigger, should that work? Could you tell me on what module should the Jitted methods appear so I can narrow my search? Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Thursday, October 14, 2010 8:31 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? So from an IronPython/DLR perspective the process should stabilize over time. That could take a while ? as various Python functions get used repeatedly we?ll switch from interpreting them to compiling them. We?ll also potentially produce new call site rules which are compiled. That could account for the increase of 16k->18k dynamic methods. That?s a 12% increase and could be a reasonable amount of it remained steady from then on. Likewise the # of function codes seems stable but the _codeCount is rising. That might mean that you?re defining new functions (via exec/execfile/compile) and they?re getting collected - but we still have their (dead) weak references in the code list. Eventually that list should get cleaned out when we hit context._nextCodeCleanup (which should be greater than context._codeCount). I would expect if you were to walk the entire PythonContext._allCodes linked list that you?d see about half of the lists having a dead weak reference. If your windbg-foo is up for writing the script to do this that?d be great but mine is rusty enough I?d need to look it up in the documentation. As for jitted code ? all dynamic methods are collectible and any normal RefEmit code is not collectible (there?s an option to make assemblies collectible in .NET 4.0, but we don?t use it as we generally don?t generate that many types). Oh, we do also generate new types for subclasses but you should see that NewTypeMaker._newTypes has a stable count over time because we share these between types w/ common bases. Closures, callbacks, generators should all be fine. If you do .symfix, then .reload, in Windbg does ?dt HostCodeHeap? show you the fields of the HostCodeHeap structure? Does !eeheap ?loader give you the address of the HostCodeHeap? It might be useful to know what m_allocationCount is on the heap overtime as if that?s relatively stable the heap could be getting fragmented. Looking at the CLR code I?m becoming more and more convinced HostCodeHeap is only used for dynamic methods so if that?s growing then I?m thinking we would appear to be leaking dynamic methods. The only way I can think of figuring out where the allocations are coming from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow (hopefully this will show up in the public symbols). That may be difficult if you can?t attach the debugger to the server but if the issue is blocking the server you could have a breakpoint here which does a stack trace and continues execution so you could inspect the stack trace later. I?d include both ?kb? and ?!ClrStack? from sos in the stack trace. 2.7 shouldn?t really change this ? whether .NET 4.0 would or not would depend on if the CLR changed anything here. But I?m not sure ? I would assume it wouldn?t. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Thursday, October 14, 2010 6:44 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Hi, I've been looking on this for some time, and I'm still don?t understand some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET 2.0) a bit better. We have a long running (stateful) application that we cannot simulate or run with a debugger open (so no breakpoints). However, since the application is run on a VM, we can take snapshots of it and then open WinDbg instance and break in the middle of the application. We do this a few hours after the application restarted and again after two days so we can see the difference. This way we saw that Jit Code Heap is increasing by a few hundred MB per day, and the number of HostCodeHeap objects is increasing. We also compared the performance counters for the two snapshots, and saw the Jitted Code Bytes increased from 100MB to 862MB and the number of methods jitted increased from 700K to 6.3M. In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to 424MB. On the other hand, the object types you mentioned stay relatively the same: ? DynamicMethods count went from 16K to 18K ? FunctionCode count went from 4013 to 4025 ? The _*code*Count field in the PythonContext went from 4447 to 7800 Here is what we don't understand: 1. Is it normal for the application to keep jitting code and methods forever? Should is stabilize? 2. From the numbers I guess that some of the jitted code IS collected. Which types are collectable and which are not? How can I tell which ones I am using? 3. Are there any specific patterns I should avoid to decrease uncollectable code (or jitting in general). I am using a lot of closures, callbacks and generators. 4. What datatypes that are visible in WinDbg can I use to understand if and why IronPython is generating uncollectable code? 5. Is there a way to trace back the HostCodeHeap objects to my code (or IronPython specific features)? 6. Can I expect an improvement in these issues by moving to Ipy 2.7 and/or .Net 4.0? Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Wednesday, October 13, 2010 10:01 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs *and*DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we?re continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there?s exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There?s also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn?t actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they?re compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it?s likely that it is one of the things I mentioned (or some other cache that?s per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That?s a bunch of different things to look at ? hopefully it?ll give some insight into what?s going on and help track down the issue. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Wednesday, October 13, 2010 12:12 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 6:59 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this configurable somehow and only turn it on if you?re actually debugging. It?s unfortunate that we can?t offer both debugging & collectability but right now that?s simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 9:09 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From merllab at microsoft.com Sun Oct 17 23:47:59 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Sun, 17 Oct 2010 14:47:59 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: <01ea10a1-2b58-460d-aac6-cf9d56c83b0e@tk5-exsmh-c101.redmond.corp.microsoft.com> This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/78527. ADDED SOURCES $/IronPython/IronPython_Main/Languages/IronPython/CreateRelease.bat MODIFIED SOURCES $/IronPython/IronPython_Main/Languages/IronPython/CreateRelease.bat $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/Loader.cs $/IronPython/IronPython_Main/Languages/Ruby/Ruby/Runtime/RubyContext.cs $/IronPython/IronPython_Main/Msi/Python/Msi/Silverlight.wxi $/IronPython/IronPython_Main/Msi/Python/Msm/IpyRedist.wxs $/IronPython/IronPython_Main/Msi/Ruby/Version.wxi $/IronPython/IronPython_Main/Msi/Ruby/Msm/IrbRedist.wxs From merllab at microsoft.com Mon Oct 18 00:27:47 2010 From: merllab at microsoft.com (merllab at microsoft.com) Date: Sun, 17 Oct 2010 15:27:47 -0700 Subject: [IronPython] IronPython 2.6 CodePlex Source Update Message-ID: This is an automated email letting you know that sources have recently been pushed out. You can download these newer sources directly from http://ironpython.codeplex.com/SourceControl/changeset/view/78529. MODIFIED SOURCES $/IronPython/IronPython_Main/Languages/IronPython/CreateRelease.bat From rawatsaurabh at yahoo.co.in Mon Oct 18 06:41:17 2010 From: rawatsaurabh at yahoo.co.in (saurabh rawat) Date: Mon, 18 Oct 2010 10:11:17 +0530 (IST) Subject: [IronPython] Getting an error while Passing Dictionary to a function In-Reply-To: Message-ID: <939218.45338.qm@web137401.mail.in.yahoo.com> HI Jeff, I have replaced the code for function call with the function code itself at that place , as it was a small function it will not matter much whether i call it or not..I know thats bad to skip the error by following this usual practice(replacing function call by function codes line at that place)..anyways thanks for the conversation Jeff, I will get back to you with full details when i encounter it again.. Rgds, Saurabh --- On Fri, 15/10/10, Jeff Hardy wrote: From: Jeff Hardy Subject: Re: [IronPython] Getting an error while Passing Dictionary to a function To: "Discussion of IronPython" Date: Friday, 15 October, 2010, 9:30 PM Can you show the exact code used to call func? Is it being called inside the class (i.e. from another member of the same class) or from outside the class? - Jeff On Thu, Oct 14, 2010 at 10:35 PM, saurabh rawat wrote: > > > ????????? HI Jeff, > Its an instance method of a class..Dont think/Even tried passing self to it .Still R&D ing on it..!! > Saurabh > > > ?????????????? " The ultimate test of a relationship is to disagree but to hold hands..............." > > > > --- On Thu, 14/10/10, Jeff Hardy wrote: > > From: Jeff Hardy > Subject: Re: [IronPython] Getting an error while Passing Dictionary to a function > To: "Discussion of IronPython" > Date: Thursday, 14 October, 2010, 8:19 PM > > Hi Saurabh, > Is Func() an instance method of a class, or is it a static function? > - Jeff > > On Thu, Oct 14, 2010 at 7:48 AM, saurabh rawat wrote: > > > > I Am trying to pass a dictionary element to a function but even though the function takes 3 arguments , it is showing error as ERROR: Func() takes exactly 4 arguments (3 given). > > Why is it happening.is there a problem in Dictionary Porting from .NET. to IronPython > > > > ?public struct callCounters > > ??????? { > > ??????????? public double callReceivedTimer; > > ??????????? public double callDialledTimer; > > ??????????? public double callAllTimer; > > ??????????? public double callLifeTimer; > > ??????? } > > > > callValue = Dictionary[DateTime, callCounters]() > > > > prototype for Func() > > > > void func(string casename, DateTime time, Dictionary <,) > > > > Function call > > > > func(casename, time, callValue ) > > > > ERROR: Func() takes exactly 4 arguments (3 given). > > > > > > Plz assist. > > Rgds, > > Saurabh > > > > > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Mon Oct 18 11:09:49 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Mon, 18 Oct 2010 11:09:49 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <5bf30e74a24da2e038012204efb78507@mail.gmail.com> <5256dbe8c1b389d472a4bee567e6da6c@mail.gmail.com> <505150d4deb7d86b62a8da32003741de@mail.gmail.com> Message-ID: Ok, I finally succeeded in creating a simple reproduction for this problem. The following code generates a 1000 methods (according to the ".NET CLR JIT" performance counter), on Ipy 2.6.1 .Net 2.0 def test_method(): for i in xrange(1000): def func(*a,**kw): pass func(some_parm = None) test_method() This does not happen if you call f without keyword params (using the *a params is OK). If this is indeed a bug, we would like to know how to fix it in the code locally, if that is possible. Also, I am interested in what Ipy flow creates this methods, since I wasn?t able to find the function in the code that does this generation Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Thursday, October 14, 2010 8:31 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? So from an IronPython/DLR perspective the process should stabilize over time. That could take a while ? as various Python functions get used repeatedly we?ll switch from interpreting them to compiling them. We?ll also potentially produce new call site rules which are compiled. That could account for the increase of 16k->18k dynamic methods. That?s a 12% increase and could be a reasonable amount of it remained steady from then on. Likewise the # of function codes seems stable but the _codeCount is rising. That might mean that you?re defining new functions (via exec/execfile/compile) and they?re getting collected - but we still have their (dead) weak references in the code list. Eventually that list should get cleaned out when we hit context._nextCodeCleanup (which should be greater than context._codeCount). I would expect if you were to walk the entire PythonContext._allCodes linked list that you?d see about half of the lists having a dead weak reference. If your windbg-foo is up for writing the script to do this that?d be great but mine is rusty enough I?d need to look it up in the documentation. As for jitted code ? all dynamic methods are collectible and any normal RefEmit code is not collectible (there?s an option to make assemblies collectible in .NET 4.0, but we don?t use it as we generally don?t generate that many types). Oh, we do also generate new types for subclasses but you should see that NewTypeMaker._newTypes has a stable count over time because we share these between types w/ common bases. Closures, callbacks, generators should all be fine. If you do .symfix, then .reload, in Windbg does ?dt HostCodeHeap? show you the fields of the HostCodeHeap structure? Does !eeheap ?loader give you the address of the HostCodeHeap? It might be useful to know what m_allocationCount is on the heap overtime as if that?s relatively stable the heap could be getting fragmented. Looking at the CLR code I?m becoming more and more convinced HostCodeHeap is only used for dynamic methods so if that?s growing then I?m thinking we would appear to be leaking dynamic methods. The only way I can think of figuring out where the allocations are coming from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow (hopefully this will show up in the public symbols). That may be difficult if you can?t attach the debugger to the server but if the issue is blocking the server you could have a breakpoint here which does a stack trace and continues execution so you could inspect the stack trace later. I?d include both ?kb? and ?!ClrStack? from sos in the stack trace. 2.7 shouldn?t really change this ? whether .NET 4.0 would or not would depend on if the CLR changed anything here. But I?m not sure ? I would assume it wouldn?t. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Thursday, October 14, 2010 6:44 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Hi, I've been looking on this for some time, and I'm still don?t understand some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET 2.0) a bit better. We have a long running (stateful) application that we cannot simulate or run with a debugger open (so no breakpoints). However, since the application is run on a VM, we can take snapshots of it and then open WinDbg instance and break in the middle of the application. We do this a few hours after the application restarted and again after two days so we can see the difference. This way we saw that Jit Code Heap is increasing by a few hundred MB per day, and the number of HostCodeHeap objects is increasing. We also compared the performance counters for the two snapshots, and saw the Jitted Code Bytes increased from 100MB to 862MB and the number of methods jitted increased from 700K to 6.3M. In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to 424MB. On the other hand, the object types you mentioned stay relatively the same: ? DynamicMethods count went from 16K to 18K ? FunctionCode count went from 4013 to 4025 ? The _*code*Count field in the PythonContext went from 4447 to 7800 Here is what we don't understand: 1. Is it normal for the application to keep jitting code and methods forever? Should is stabilize? 2. From the numbers I guess that some of the jitted code IS collected. Which types are collectable and which are not? How can I tell which ones I am using? 3. Are there any specific patterns I should avoid to decrease uncollectable code (or jitting in general). I am using a lot of closures, callbacks and generators. 4. What datatypes that are visible in WinDbg can I use to understand if and why IronPython is generating uncollectable code? 5. Is there a way to trace back the HostCodeHeap objects to my code (or IronPython specific features)? 6. Can I expect an improvement in these issues by moving to Ipy 2.7 and/or .Net 4.0? Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Wednesday, October 13, 2010 10:01 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs *and*DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we?re continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there?s exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There?s also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn?t actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they?re compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it?s likely that it is one of the things I mentioned (or some other cache that?s per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That?s a bunch of different things to look at ? hopefully it?ll give some insight into what?s going on and help track down the issue. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Wednesday, October 13, 2010 12:12 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 6:59 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this configurable somehow and only turn it on if you?re actually debugging. It?s unfortunate that we can?t offer both debugging & collectability but right now that?s simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 9:09 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From idan at cloudshare.com Mon Oct 18 11:35:23 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Mon, 18 Oct 2010 11:35:23 +0200 Subject: [IronPython] HostCodeHeap leakage? Message-ID: <7c4757f1f0618419ef8a198e7ed1bfb1@mail.gmail.com> FYI, there is a bit simpler reproduction: def test_method(): for i in xrange(1000): def func(param = None): pass func(param = None) test_method() So, actually any use of keyword params in closure that are redefined causes the problem. *From:* Idan Zaltzberg [mailto:idan at cloudshare.com] *Sent:* Monday, October 18, 2010 11:10 AM *To:* 'Discussion of IronPython' *Subject:* RE: [IronPython] HostCodeHeap leakage? Ok, I finally succeeded in creating a simple reproduction for this problem. The following code generates a 1000 methods (according to the ".NET CLR JIT" performance counter), on Ipy 2.6.1 .Net 2.0 def test_method(): for i in xrange(1000): def func(*a,**kw): pass func(some_parm = None) test_method() This does not happen if you call f without keyword params (using the *a params is OK). If this is indeed a bug, we would like to know how to fix it in the code locally, if that is possible. Also, I am interested in what Ipy flow creates this methods, since I wasn?t able to find the function in the code that does this generation Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Thursday, October 14, 2010 8:31 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? So from an IronPython/DLR perspective the process should stabilize over time. That could take a while ? as various Python functions get used repeatedly we?ll switch from interpreting them to compiling them. We?ll also potentially produce new call site rules which are compiled. That could account for the increase of 16k->18k dynamic methods. That?s a 12% increase and could be a reasonable amount of it remained steady from then on. Likewise the # of function codes seems stable but the _codeCount is rising. That might mean that you?re defining new functions (via exec/execfile/compile) and they?re getting collected - but we still have their (dead) weak references in the code list. Eventually that list should get cleaned out when we hit context._nextCodeCleanup (which should be greater than context._codeCount). I would expect if you were to walk the entire PythonContext._allCodes linked list that you?d see about half of the lists having a dead weak reference. If your windbg-foo is up for writing the script to do this that?d be great but mine is rusty enough I?d need to look it up in the documentation. As for jitted code ? all dynamic methods are collectible and any normal RefEmit code is not collectible (there?s an option to make assemblies collectible in .NET 4.0, but we don?t use it as we generally don?t generate that many types). Oh, we do also generate new types for subclasses but you should see that NewTypeMaker._newTypes has a stable count over time because we share these between types w/ common bases. Closures, callbacks, generators should all be fine. If you do .symfix, then .reload, in Windbg does ?dt HostCodeHeap? show you the fields of the HostCodeHeap structure? Does !eeheap ?loader give you the address of the HostCodeHeap? It might be useful to know what m_allocationCount is on the heap overtime as if that?s relatively stable the heap could be getting fragmented. Looking at the CLR code I?m becoming more and more convinced HostCodeHeap is only used for dynamic methods so if that?s growing then I?m thinking we would appear to be leaking dynamic methods. The only way I can think of figuring out where the allocations are coming from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow (hopefully this will show up in the public symbols). That may be difficult if you can?t attach the debugger to the server but if the issue is blocking the server you could have a breakpoint here which does a stack trace and continues execution so you could inspect the stack trace later. I?d include both ?kb? and ?!ClrStack? from sos in the stack trace. 2.7 shouldn?t really change this ? whether .NET 4.0 would or not would depend on if the CLR changed anything here. But I?m not sure ? I would assume it wouldn?t. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Thursday, October 14, 2010 6:44 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Hi, I've been looking on this for some time, and I'm still don?t understand some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET 2.0) a bit better. We have a long running (stateful) application that we cannot simulate or run with a debugger open (so no breakpoints). However, since the application is run on a VM, we can take snapshots of it and then open WinDbg instance and break in the middle of the application. We do this a few hours after the application restarted and again after two days so we can see the difference. This way we saw that Jit Code Heap is increasing by a few hundred MB per day, and the number of HostCodeHeap objects is increasing. We also compared the performance counters for the two snapshots, and saw the Jitted Code Bytes increased from 100MB to 862MB and the number of methods jitted increased from 700K to 6.3M. In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to 424MB. On the other hand, the object types you mentioned stay relatively the same: ? DynamicMethods count went from 16K to 18K ? FunctionCode count went from 4013 to 4025 ? The _*code*Count field in the PythonContext went from 4447 to 7800 Here is what we don't understand: 1. Is it normal for the application to keep jitting code and methods forever? Should is stabilize? 2. From the numbers I guess that some of the jitted code IS collected. Which types are collectable and which are not? How can I tell which ones I am using? 3. Are there any specific patterns I should avoid to decrease uncollectable code (or jitting in general). I am using a lot of closures, callbacks and generators. 4. What datatypes that are visible in WinDbg can I use to understand if and why IronPython is generating uncollectable code? 5. Is there a way to trace back the HostCodeHeap objects to my code (or IronPython specific features)? 6. Can I expect an improvement in these issues by moving to Ipy 2.7 and/or .Net 4.0? Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Wednesday, October 13, 2010 10:01 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs *and*DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we?re continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there?s exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There?s also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn?t actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they?re compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it?s likely that it is one of the things I mentioned (or some other cache that?s per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That?s a bunch of different things to look at ? hopefully it?ll give some insight into what?s going on and help track down the issue. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Wednesday, October 13, 2010 12:12 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 6:59 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this configurable somehow and only turn it on if you?re actually debugging. It?s unfortunate that we can?t offer both debugging & collectability but right now that?s simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 9:09 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From cenovsky at bakalari.cz Mon Oct 18 19:01:59 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Mon, 18 Oct 2010 19:01:59 +0200 Subject: [IronPython] gcroot for ironpython object Message-ID: <4CBC7D87.3060602@bakalari.cz> Hi all, I have an instance of IronPython object and I'd like to find out why it is kept in memory (I use weakref to check it is still in the memory). What is the best way to find the object in WinDbg so I can call gcroot for it? Or is there a better way to find out why my object is kept in memory? Thanks. -- -- Luk?? From grauenwolf at gmail.com Mon Oct 18 20:04:46 2010 From: grauenwolf at gmail.com (Jonathan Allen) Date: Mon, 18 Oct 2010 11:04:46 -0700 Subject: [IronPython] gcroot for ironpython object Message-ID: <4cbc8c04.0a06df0a.4057.ffffc1be@mx.google.com> One trick is to give it a radioactive marker like they do in medical shows. Have that instance you care about reference an instance of EmailMessage or some other class you aren't actually using. Finding the EmailMessage object should be easy because there is only one, and from there you can find the IronPython object you are after. I use RedGate instead of WinDbg, so I'm not sure this will work for you. If you try it, please let me know the results. Jonathan -----Original Message----- From: Lukas Cenovsky Sent: Monday, October 18, 2010 10:01 AM To: Discussion of IronPython Subject: [IronPython] gcroot for ironpython object Hi all, I have an instance of IronPython object and I'd like to find out why it is kept in memory (I use weakref to check it is still in the memory). What is the best way to find the object in WinDbg so I can call gcroot for it? Or is there a better way to find out why my object is kept in memory? Thanks. -- -- Luk?? _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com From cenovsky at bakalari.cz Mon Oct 18 20:26:34 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Mon, 18 Oct 2010 20:26:34 +0200 Subject: [IronPython] gcroot for ironpython object In-Reply-To: <4cbc8c04.0a06df0a.4057.ffffc1be@mx.google.com> References: <4cbc8c04.0a06df0a.4057.ffffc1be@mx.google.com> Message-ID: <4CBC915A.50101@bakalari.cz> An HTML attachment was scrubbed... URL: From idan at cloudshare.com Mon Oct 18 20:48:12 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Mon, 18 Oct 2010 20:48:12 +0200 Subject: [IronPython] gcroot for ironpython object In-Reply-To: <4CBC915A.50101@bakalari.cz> References: <4cbc8c04.0a06df0a.4057.ffffc1be@mx.google.com> <4CBC915A.50101@bakalari.cz> Message-ID: I recently wrote a script that dumps an IronPython 2.6.1 (.NET 2.0) object. Probably won't work on anything else. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Lukas Cenovsky *Sent:* Monday, October 18, 2010 8:27 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] gcroot for ironpython object Thanks - that works. I've also found Kamil's blog post: http://blog.kamil.dworakowski.name/2008/02/debugging-memory-problems-in-ironpython.html Now I'm fighting with s - trying to convert them to something readable... Kamil's script does not seem to work - I don't see any dict holding the ironpython object in the stacktrace: ... 0330d0f4(System.Windows.Controls.StackPanel)-> 0330d22c(System.Windows.Controls.UIElementCollection)-> 0330d240(System.Windows.Media.VisualCollection)-> 0330d520(System.Object[])-> 0330d428(System.Windows.Controls.Button)-> 0824cb0c(System.Windows.EffectiveValueEntry[])-> 0824cb00(System.Windows.EventHandlersStore)-> 0824cc00(MS.Utility.SingleObjectMap)-> 0824cbe0(MS.Utility.FrugalObjectList`1[[System.Windows.RoutedEventHandlerInfo, PresentationCore]])-> 0824cbec(MS.Utility.SingleItemList`1[[System.Windows.RoutedEventHandlerInfo, PresentationCore]])-> 0824caa8(System.Windows.RoutedEventHandler)-> 0824ca44(System.Object[])-> * 0824ca18(IronPython.Runtime.Method)->** 08162f84()->* 08163d78(IronPython.Runtime.PythonDictionary)-> 08163d6c(IronPython.Runtime.StringDictionaryStorage)-> 08163d84(System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Object, mscorlib]])-> 081c5558(System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Object, mscorlib]][])-> 0816b3bc(System.GopherStyleUriParser) -- -- Luk?? On 18.10.2010 20:04, Jonathan Allen wrote: One trick is to give it a radioactive marker like they do in medical shows. Have that instance you care about reference an instance of EmailMessage or some other class you aren't actually using. Finding the EmailMessage object should be easy because there is only one, and from there you can find the IronPython object you are after. I use RedGate instead of WinDbg, so I'm not sure this will work for you. If you try it, please let me know the results. Jonathan -----Original Message----- From: Lukas Cenovsky Sent: Monday, October 18, 2010 10:01 AM To: Discussion of IronPython Subject: [IronPython] gcroot for ironpython object Hi all, I have an instance of IronPython object and I'd like to find out why it is kept in memory (I use weakref to check it is still in the memory). What is the best way to find the object in WinDbg so I can call gcroot for it? Or is there a better way to find out why my object is kept in memory? Thanks. -- -- Luk?? _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: py_obj.wds Type: application/octet-stream Size: 1061 bytes Desc: not available URL: From ronnie.maor at gmail.com Mon Oct 18 22:24:41 2010 From: ronnie.maor at gmail.com (Ronnie Maor) Date: Mon, 18 Oct 2010 22:24:41 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: <7c4757f1f0618419ef8a198e7ed1bfb1@mail.gmail.com> References: <7c4757f1f0618419ef8a198e7ed1bfb1@mail.gmail.com> Message-ID: Can someone from IPy team ack that you saw this? The issue is causing us a lot of trouble, so we'd really appreciate it if you could tell us how to fix - we've already built from source to fix a previous leak, so no problem building with another patch. BTW, the default value in the function definition is not needed, it's calling with named arguments that causes the issue. so this is a slightly simpler repro: def test_method(): for i in xrange(1000): def func(param): pass func(param = None) thanks! Ronnie On Mon, Oct 18, 2010 at 11:35 AM, Idan Zaltzberg wrote: > FYI, there is a bit simpler reproduction: > > def test_method(): > > for i in xrange(1000): > > def func(param = None): pass > > func(param = None) > > > > test_method() > > > > So, actually any use of keyword params in closure that are redefined causes > the problem. > > > > *From:* Idan Zaltzberg [mailto:idan at cloudshare.com] > *Sent:* Monday, October 18, 2010 11:10 AM > *To:* 'Discussion of IronPython' > *Subject:* RE: [IronPython] HostCodeHeap leakage? > > > > Ok, I finally succeeded in creating a simple reproduction for this problem. > > The following code generates a 1000 methods (according to the ".NET CLR > JIT" performance counter), on Ipy 2.6.1 .Net 2.0 > > > > def test_method(): > > for i in xrange(1000): > > def func(*a,**kw): pass > > func(some_parm = None) > > > > test_method() > > > > This does not happen if you call f without keyword params (using the *a > params is OK). > > If this is indeed a bug, we would like to know how to fix it in the code > locally, if that is possible. > > Also, I am interested in what Ipy flow creates this methods, since I wasn?t > able to find the function in the code that does this generation > > > > Thanks > > > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland > *Sent:* Thursday, October 14, 2010 8:31 PM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > So from an IronPython/DLR perspective the process should stabilize over > time. That could take a while ? as various Python functions get used > repeatedly we?ll switch from interpreting them to compiling them. We?ll > also potentially produce new call site rules which are compiled. That could > account for the increase of 16k->18k dynamic methods. That?s a 12% increase > and could be a reasonable amount of it remained steady from then on. > Likewise the # of function codes seems stable but the _codeCount is rising. > That might mean that you?re defining new functions (via > exec/execfile/compile) and they?re getting collected - but we still have > their (dead) weak references in the code list. Eventually that list should > get cleaned out when we hit context._nextCodeCleanup (which should be > greater than context._codeCount). > > > > I would expect if you were to walk the entire PythonContext._allCodes > linked list that you?d see about half of the lists having a dead weak > reference. If your windbg-foo is up for writing the script to do this > that?d be great but mine is rusty enough I?d need to look it up in the > documentation. > > > > As for jitted code ? all dynamic methods are collectible and any normal > RefEmit code is not collectible (there?s an option to make assemblies > collectible in .NET 4.0, but we don?t use it as we generally don?t generate > that many types). Oh, we do also generate new types for subclasses but you > should see that NewTypeMaker._newTypes has a stable count over time because > we share these between types w/ common bases. > > > > Closures, callbacks, generators should all be fine. > > > > If you do .symfix, then .reload, in Windbg does ?dt HostCodeHeap? show you > the fields of the HostCodeHeap structure? Does !eeheap ?loader give you the > address of the HostCodeHeap? It might be useful to know what > m_allocationCount is on the heap overtime as if that?s relatively stable the > heap could be getting fragmented. Looking at the CLR code I?m becoming more > and more convinced HostCodeHeap is only used for dynamic methods so if > that?s growing then I?m thinking we would appear to be leaking dynamic > methods. > > > > The only way I can think of figuring out where the allocations are coming > from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow > (hopefully this will show up in the public symbols). That may be difficult > if you can?t attach the debugger to the server but if the issue is blocking > the server you could have a breakpoint here which does a stack trace and > continues execution so you could inspect the stack trace later. I?d include > both ?kb? and ?!ClrStack? from sos in the stack trace. > > > > 2.7 shouldn?t really change this ? whether .NET 4.0 would or not would > depend on if the CLR changed anything here. But I?m not sure ? I would > assume it wouldn?t. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg > *Sent:* Thursday, October 14, 2010 6:44 AM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > Hi, > > > > I've been looking on this for some time, and I'm still don?t understand > some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET > 2.0) a bit better. > > > > We have a long running (stateful) application that we cannot simulate or > run with a debugger open (so no breakpoints). > > However, since the application is run on a VM, we can take snapshots of it > and then open WinDbg instance and break in the middle of the application. > > We do this a few hours after the application restarted and again after two > days so we can see the difference. > > > > This way we saw that Jit Code Heap is increasing by a few hundred MB per > day, and the number of HostCodeHeap objects is increasing. > > We also compared the performance counters for the two snapshots, and saw > the Jitted Code Bytes increased from 100MB to 862MB and the number of > methods jitted increased from 700K to 6.3M. > > In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to > 424MB. > > On the other hand, the object types you mentioned stay relatively the same: > > ? DynamicMethods count went from 16K to 18K > > ? FunctionCode count went from 4013 to 4025 > > ? The _*code*Count field in the PythonContext went from 4447 to > 7800 > > Here is what we don't understand: > > 1. Is it normal for the application to keep jitting code and methods > forever? Should is stabilize? > > 2. From the numbers I guess that some of the jitted code IS > collected. Which types are collectable and which are not? How can I tell > which ones I am using? > > 3. Are there any specific patterns I should avoid to decrease > uncollectable code (or jitting in general). I am using a lot of closures, > callbacks and generators. > > 4. What datatypes that are visible in WinDbg can I use to understand > if and why IronPython is generating uncollectable code? > > 5. Is there a way to trace back the HostCodeHeap objects to my code > (or IronPython specific features)? > > 6. Can I expect an improvement in these issues by moving to Ipy 2.7 > and/or .Net 4.0? > > > > Thanks > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland > *Sent:* Wednesday, October 13, 2010 10:01 PM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > If you build from source you could set some breakpoints in AssemblyGen.cs > in the DefineType method. You can also set one in DelegateUtils.cs *and*DelegateHelpers.cs in DefineDelegateType. I think those are all the places > where we are creating uncollectible types. If we?re continuously hitting > those breakpoints after you believe your app has reached steady state then > something is going wrong. > > > > This code > http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads me to believe the HostCodeHeap might also be used for DynamicMethods. > If that is the case then the other place to look would be if FunctionCode > objects are being re-created repeatedly. That will happen if there?s > exec/eval/compile calls which are happening and if those objects are being > kept alive then we could be growing the heap over time. > > > > There?s also some complicated code which deals with keeping a list of all > code that is alive. We do cleanup this list, and the list is a list of weak > references so it shouldn?t actually keep the code alive, but you could put > some breakpoints at FunctionCode.RegisterFunctionCode and > FunctionCode.CodeCleanup to see if that list is growing boundlessly (which > it would be if something was keeping code objects alive after an > exec/eval/compile). > > > > Another place where code generation could be occurring would be w/ > regexes. If you are dynamically generating reg-exes, or executing a huge > different variety of them over time, and they?re compiled, then the compiled > regexes could be staying in memory. There is a regex cache and you can > clear it by calling re.purge(). But it should only cache up to 100 regexes. > > > > A final possible thing to investigate might be what happens if you throw > away the entire ScriptEngine instance. Here you could try re-cycling the > ScriptEngine say every 6 hours and see if the problem goes away. If that > fixes the problem then it?s likely that it is one of the things I mentioned > (or some other cache that?s per-runtime). At least that would start to > narrow it down vs. some potentially global state (like the subtype list > which is shared across ScriptEngines). > > > > That?s a bunch of different things to look at ? hopefully it?ll give some > insight into what?s going on and help track down the issue. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg > *Sent:* Wednesday, October 13, 2010 12:12 AM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > I tried what you suggested (changed setup.DebugMode = false;) > > But still I get the same behavior: > > The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. > > Is there a way to verify from the IronPython code that DebugMode is off? > > Is there anything else I can do (other startup settings?) to > decrease/understand the increase in HostCodeHeap objects? > > Thanks. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland > *Sent:* Tuesday, October 05, 2010 6:59 PM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this > configurable somehow and only turn it on if you?re actually debugging. It?s > unfortunate that we can?t offer both debugging & collectability but right > now that?s simply a limitation of the CLR and/or our lack of a separate VS > debug engine which can debug Python code. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg > *Sent:* Tuesday, October 05, 2010 9:09 AM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > Im running using the engine from a hosting app. > > We have these lines in the startup: > > ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); > setup.DebugMode = true; > > ScriptRuntime runtime = Python.CreateRuntime(setup.Options); > > engine = runtime.GetEngine("py"); > > > > Is this is the same like ?X:Debug? > > You reckon this could be the cause? > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland > *Sent:* Tuesday, October 05, 2010 5:53 PM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > My guess is that?s code in the JIT heap that?s building up but I?m not 100% > certain. How is your code being executed? Do you have the debug option (-D > or ?X:Debug) enabled? To support debug mode we need to produce > uncollectible code which could be building up. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg > *Sent:* Tuesday, October 05, 2010 2:26 AM > *To:* Discussion of IronPython > *Subject:* [IronPython] HostCodeHeap leakage? > > > > I am trying to find a memory/"performance" leak in an Ipy application. > > Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is > getting bigger (150MB increase per day). From the !eeheap output it seems > that the increase is due to HostCodeHeap (objects?). > > As I understand these objects might be created by Ipy infra, is that right? > > Is there anyway I can get more info on their content, or prevent them from > growing? > > Thanks > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Mon Oct 18 22:28:22 2010 From: dinov at microsoft.com (Dino Viehland) Date: Mon, 18 Oct 2010 20:28:22 +0000 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <7c4757f1f0618419ef8a198e7ed1bfb1@mail.gmail.com> Message-ID: Yep, I've seen it... still thinking about what should be done here. I certainly understand the problem and what will cause it but the fix is probably non trivial (but also probably well contained to one or two classes). It may take me a day or two to respond w/ something substantial. We probably need to change our kw-calling to use a pre-compiled rule. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ronnie Maor Sent: Monday, October 18, 2010 1:25 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Can someone from IPy team ack that you saw this? The issue is causing us a lot of trouble, so we'd really appreciate it if you could tell us how to fix - we've already built from source to fix a previous leak, so no problem building with another patch. BTW, the default value in the function definition is not needed, it's calling with named arguments that causes the issue. so this is a slightly simpler repro: def test_method(): for i in xrange(1000): def func(param): pass func(param = None) thanks! Ronnie On Mon, Oct 18, 2010 at 11:35 AM, Idan Zaltzberg > wrote: FYI, there is a bit simpler reproduction: def test_method(): for i in xrange(1000): def func(param = None): pass func(param = None) test_method() So, actually any use of keyword params in closure that are redefined causes the problem. From: Idan Zaltzberg [mailto:idan at cloudshare.com] Sent: Monday, October 18, 2010 11:10 AM To: 'Discussion of IronPython' Subject: RE: [IronPython] HostCodeHeap leakage? Ok, I finally succeeded in creating a simple reproduction for this problem. The following code generates a 1000 methods (according to the ".NET CLR JIT" performance counter), on Ipy 2.6.1 .Net 2.0 def test_method(): for i in xrange(1000): def func(*a,**kw): pass func(some_parm = None) test_method() This does not happen if you call f without keyword params (using the *a params is OK). If this is indeed a bug, we would like to know how to fix it in the code locally, if that is possible. Also, I am interested in what Ipy flow creates this methods, since I wasn't able to find the function in the code that does this generation Thanks From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Thursday, October 14, 2010 8:31 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? So from an IronPython/DLR perspective the process should stabilize over time. That could take a while - as various Python functions get used repeatedly we'll switch from interpreting them to compiling them. We'll also potentially produce new call site rules which are compiled. That could account for the increase of 16k->18k dynamic methods. That's a 12% increase and could be a reasonable amount of it remained steady from then on. Likewise the # of function codes seems stable but the _codeCount is rising. That might mean that you're defining new functions (via exec/execfile/compile) and they're getting collected - but we still have their (dead) weak references in the code list. Eventually that list should get cleaned out when we hit context._nextCodeCleanup (which should be greater than context._codeCount). I would expect if you were to walk the entire PythonContext._allCodes linked list that you'd see about half of the lists having a dead weak reference. If your windbg-foo is up for writing the script to do this that'd be great but mine is rusty enough I'd need to look it up in the documentation. As for jitted code - all dynamic methods are collectible and any normal RefEmit code is not collectible (there's an option to make assemblies collectible in .NET 4.0, but we don't use it as we generally don't generate that many types). Oh, we do also generate new types for subclasses but you should see that NewTypeMaker._newTypes has a stable count over time because we share these between types w/ common bases. Closures, callbacks, generators should all be fine. If you do .symfix, then .reload, in Windbg does "dt HostCodeHeap" show you the fields of the HostCodeHeap structure? Does !eeheap -loader give you the address of the HostCodeHeap? It might be useful to know what m_allocationCount is on the heap overtime as if that's relatively stable the heap could be getting fragmented. Looking at the CLR code I'm becoming more and more convinced HostCodeHeap is only used for dynamic methods so if that's growing then I'm thinking we would appear to be leaking dynamic methods. The only way I can think of figuring out where the allocations are coming from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow (hopefully this will show up in the public symbols). That may be difficult if you can't attach the debugger to the server but if the issue is blocking the server you could have a breakpoint here which does a stack trace and continues execution so you could inspect the stack trace later. I'd include both "kb" and "!ClrStack" from sos in the stack trace. 2.7 shouldn't really change this - whether .NET 4.0 would or not would depend on if the CLR changed anything here. But I'm not sure - I would assume it wouldn't. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Thursday, October 14, 2010 6:44 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Hi, I've been looking on this for some time, and I'm still don't understand some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET 2.0) a bit better. We have a long running (stateful) application that we cannot simulate or run with a debugger open (so no breakpoints). However, since the application is run on a VM, we can take snapshots of it and then open WinDbg instance and break in the middle of the application. We do this a few hours after the application restarted and again after two days so we can see the difference. This way we saw that Jit Code Heap is increasing by a few hundred MB per day, and the number of HostCodeHeap objects is increasing. We also compared the performance counters for the two snapshots, and saw the Jitted Code Bytes increased from 100MB to 862MB and the number of methods jitted increased from 700K to 6.3M. In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to 424MB. On the other hand, the object types you mentioned stay relatively the same: * DynamicMethods count went from 16K to 18K * FunctionCode count went from 4013 to 4025 * The _codeCount field in the PythonContext went from 4447 to 7800 Here is what we don't understand: 1. Is it normal for the application to keep jitting code and methods forever? Should is stabilize? 2. From the numbers I guess that some of the jitted code IS collected. Which types are collectable and which are not? How can I tell which ones I am using? 3. Are there any specific patterns I should avoid to decrease uncollectable code (or jitting in general). I am using a lot of closures, callbacks and generators. 4. What datatypes that are visible in WinDbg can I use to understand if and why IronPython is generating uncollectable code? 5. Is there a way to trace back the HostCodeHeap objects to my code (or IronPython specific features)? 6. Can I expect an improvement in these issues by moving to Ipy 2.7 and/or .Net 4.0? Thanks From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Wednesday, October 13, 2010 10:01 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs and DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we're continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspx leads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there's exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There's also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn't actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they're compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it's likely that it is one of the things I mentioned (or some other cache that's per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That's a bunch of different things to look at - hopefully it'll give some insight into what's going on and help track down the issue. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Wednesday, October 13, 2010 12:12 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 6:59 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as -X:Debug. In general I'd suggest making this configurable somehow and only turn it on if you're actually debugging. It's unfortunate that we can't offer both debugging & collectability but right now that's simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 9:09 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like -X:Debug? You reckon this could be the cause? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 5:53 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? My guess is that's code in the JIT heap that's building up but I'm not 100% certain. How is your code being executed? Do you have the debug option (-D or -X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 2:26 AM To: Discussion of IronPython Subject: [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronnie.maor at gmail.com Mon Oct 18 22:33:43 2010 From: ronnie.maor at gmail.com (Ronnie Maor) Date: Mon, 18 Oct 2010 22:33:43 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <7c4757f1f0618419ef8a198e7ed1bfb1@mail.gmail.com> Message-ID: thanks - wanted to make sure you didn't miss it. we'll try to reduce the number of places where we're exposed to it in the meantime. On Mon, Oct 18, 2010 at 10:28 PM, Dino Viehland wrote: > Yep, I?ve seen it? still thinking about what should be done here. I > certainly understand the problem and what will cause it but the fix is > probably non trivial (but also probably well contained to one or two > classes). It may take me a day or two to respond w/ something substantial. > We probably need to change our kw-calling to use a pre-compiled rule. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Ronnie Maor > *Sent:* Monday, October 18, 2010 1:25 PM > > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > Can someone from IPy team ack that you saw this? > > The issue is causing us a lot of trouble, so we'd really appreciate it if > you could tell us how to fix - we've already built from source to fix a > previous leak, so no problem building with another patch. > > > > BTW, the default value in the function definition is not needed, it's > calling with named arguments that causes the issue. so this is a slightly > simpler repro: > > > > def test_method(): > > for i in xrange(1000): > > def func(param): pass > > func(param = None) > > > > thanks! > > Ronnie > > > > On Mon, Oct 18, 2010 at 11:35 AM, Idan Zaltzberg > wrote: > > FYI, there is a bit simpler reproduction: > > def test_method(): > > for i in xrange(1000): > > def func(param = None): pass > > func(param = None) > > > > test_method() > > > > So, actually any use of keyword params in closure that are redefined causes > the problem. > > > > *From:* Idan Zaltzberg [mailto:idan at cloudshare.com] > *Sent:* Monday, October 18, 2010 11:10 AM > *To:* 'Discussion of IronPython' > *Subject:* RE: [IronPython] HostCodeHeap leakage? > > > > Ok, I finally succeeded in creating a simple reproduction for this problem. > > The following code generates a 1000 methods (according to the ".NET CLR > JIT" performance counter), on Ipy 2.6.1 .Net 2.0 > > > > def test_method(): > > for i in xrange(1000): > > def func(*a,**kw): pass > > func(some_parm = None) > > > > test_method() > > > > This does not happen if you call f without keyword params (using the *a > params is OK). > > If this is indeed a bug, we would like to know how to fix it in the code > locally, if that is possible. > > Also, I am interested in what Ipy flow creates this methods, since I wasn?t > able to find the function in the code that does this generation > > > > Thanks > > > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland > *Sent:* Thursday, October 14, 2010 8:31 PM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > So from an IronPython/DLR perspective the process should stabilize over > time. That could take a while ? as various Python functions get used > repeatedly we?ll switch from interpreting them to compiling them. We?ll > also potentially produce new call site rules which are compiled. That could > account for the increase of 16k->18k dynamic methods. That?s a 12% increase > and could be a reasonable amount of it remained steady from then on. > Likewise the # of function codes seems stable but the _codeCount is rising. > That might mean that you?re defining new functions (via > exec/execfile/compile) and they?re getting collected - but we still have > their (dead) weak references in the code list. Eventually that list should > get cleaned out when we hit context._nextCodeCleanup (which should be > greater than context._codeCount). > > > > I would expect if you were to walk the entire PythonContext._allCodes > linked list that you?d see about half of the lists having a dead weak > reference. If your windbg-foo is up for writing the script to do this > that?d be great but mine is rusty enough I?d need to look it up in the > documentation. > > > > As for jitted code ? all dynamic methods are collectible and any normal > RefEmit code is not collectible (there?s an option to make assemblies > collectible in .NET 4.0, but we don?t use it as we generally don?t generate > that many types). Oh, we do also generate new types for subclasses but you > should see that NewTypeMaker._newTypes has a stable count over time because > we share these between types w/ common bases. > > > > Closures, callbacks, generators should all be fine. > > > > If you do .symfix, then .reload, in Windbg does ?dt HostCodeHeap? show you > the fields of the HostCodeHeap structure? Does !eeheap ?loader give you the > address of the HostCodeHeap? It might be useful to know what > m_allocationCount is on the heap overtime as if that?s relatively stable the > heap could be getting fragmented. Looking at the CLR code I?m becoming more > and more convinced HostCodeHeap is only used for dynamic methods so if > that?s growing then I?m thinking we would appear to be leaking dynamic > methods. > > > > The only way I can think of figuring out where the allocations are coming > from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow > (hopefully this will show up in the public symbols). That may be difficult > if you can?t attach the debugger to the server but if the issue is blocking > the server you could have a breakpoint here which does a stack trace and > continues execution so you could inspect the stack trace later. I?d include > both ?kb? and ?!ClrStack? from sos in the stack trace. > > > > 2.7 shouldn?t really change this ? whether .NET 4.0 would or not would > depend on if the CLR changed anything here. But I?m not sure ? I would > assume it wouldn?t. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg > *Sent:* Thursday, October 14, 2010 6:44 AM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > Hi, > > > > I've been looking on this for some time, and I'm still don?t understand > some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET > 2.0) a bit better. > > > > We have a long running (stateful) application that we cannot simulate or > run with a debugger open (so no breakpoints). > > However, since the application is run on a VM, we can take snapshots of it > and then open WinDbg instance and break in the middle of the application. > > We do this a few hours after the application restarted and again after two > days so we can see the difference. > > > > This way we saw that Jit Code Heap is increasing by a few hundred MB per > day, and the number of HostCodeHeap objects is increasing. > > We also compared the performance counters for the two snapshots, and saw > the Jitted Code Bytes increased from 100MB to 862MB and the number of > methods jitted increased from 700K to 6.3M. > > In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to > 424MB. > > On the other hand, the object types you mentioned stay relatively the same: > > ? DynamicMethods count went from 16K to 18K > > ? FunctionCode count went from 4013 to 4025 > > ? The _*code*Count field in the PythonContext went from 4447 to > 7800 > > Here is what we don't understand: > > 1. Is it normal for the application to keep jitting code and methods > forever? Should is stabilize? > > 2. From the numbers I guess that some of the jitted code IS > collected. Which types are collectable and which are not? How can I tell > which ones I am using? > > 3. Are there any specific patterns I should avoid to decrease > uncollectable code (or jitting in general). I am using a lot of closures, > callbacks and generators. > > 4. What datatypes that are visible in WinDbg can I use to understand > if and why IronPython is generating uncollectable code? > > 5. Is there a way to trace back the HostCodeHeap objects to my code > (or IronPython specific features)? > > 6. Can I expect an improvement in these issues by moving to Ipy 2.7 > and/or .Net 4.0? > > > > Thanks > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland > *Sent:* Wednesday, October 13, 2010 10:01 PM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > If you build from source you could set some breakpoints in AssemblyGen.cs > in the DefineType method. You can also set one in DelegateUtils.cs *and*DelegateHelpers.cs in DefineDelegateType. I think those are all the places > where we are creating uncollectible types. If we?re continuously hitting > those breakpoints after you believe your app has reached steady state then > something is going wrong. > > > > This code > http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads me to believe the HostCodeHeap might also be used for DynamicMethods. > If that is the case then the other place to look would be if FunctionCode > objects are being re-created repeatedly. That will happen if there?s > exec/eval/compile calls which are happening and if those objects are being > kept alive then we could be growing the heap over time. > > > > There?s also some complicated code which deals with keeping a list of all > code that is alive. We do cleanup this list, and the list is a list of weak > references so it shouldn?t actually keep the code alive, but you could put > some breakpoints at FunctionCode.RegisterFunctionCode and > FunctionCode.CodeCleanup to see if that list is growing boundlessly (which > it would be if something was keeping code objects alive after an > exec/eval/compile). > > > > Another place where code generation could be occurring would be w/ > regexes. If you are dynamically generating reg-exes, or executing a huge > different variety of them over time, and they?re compiled, then the compiled > regexes could be staying in memory. There is a regex cache and you can > clear it by calling re.purge(). But it should only cache up to 100 regexes. > > > > A final possible thing to investigate might be what happens if you throw > away the entire ScriptEngine instance. Here you could try re-cycling the > ScriptEngine say every 6 hours and see if the problem goes away. If that > fixes the problem then it?s likely that it is one of the things I mentioned > (or some other cache that?s per-runtime). At least that would start to > narrow it down vs. some potentially global state (like the subtype list > which is shared across ScriptEngines). > > > > That?s a bunch of different things to look at ? hopefully it?ll give some > insight into what?s going on and help track down the issue. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg > *Sent:* Wednesday, October 13, 2010 12:12 AM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > I tried what you suggested (changed setup.DebugMode = false;) > > But still I get the same behavior: > > The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. > > Is there a way to verify from the IronPython code that DebugMode is off? > > Is there anything else I can do (other startup settings?) to > decrease/understand the increase in HostCodeHeap objects? > > Thanks. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland > *Sent:* Tuesday, October 05, 2010 6:59 PM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this > configurable somehow and only turn it on if you?re actually debugging. It?s > unfortunate that we can?t offer both debugging & collectability but right > now that?s simply a limitation of the CLR and/or our lack of a separate VS > debug engine which can debug Python code. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg > *Sent:* Tuesday, October 05, 2010 9:09 AM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > Im running using the engine from a hosting app. > > We have these lines in the startup: > > ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); > setup.DebugMode = true; > > ScriptRuntime runtime = Python.CreateRuntime(setup.Options); > > engine = runtime.GetEngine("py"); > > > > Is this is the same like ?X:Debug? > > You reckon this could be the cause? > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland > *Sent:* Tuesday, October 05, 2010 5:53 PM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] HostCodeHeap leakage? > > > > My guess is that?s code in the JIT heap that?s building up but I?m not 100% > certain. How is your code being executed? Do you have the debug option (-D > or ?X:Debug) enabled? To support debug mode we need to produce > uncollectible code which could be building up. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg > *Sent:* Tuesday, October 05, 2010 2:26 AM > *To:* Discussion of IronPython > *Subject:* [IronPython] HostCodeHeap leakage? > > > > I am trying to find a memory/"performance" leak in an Ipy application. > > Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is > getting bigger (150MB increase per day). From the !eeheap output it seems > that the increase is due to HostCodeHeap (objects?). > > As I understand these objects might be created by Ipy infra, is that right? > > Is there anyway I can get more info on their content, or prevent them from > growing? > > Thanks > > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kurt at kurtrichardson.com Wed Oct 20 20:45:20 2010 From: kurt at kurtrichardson.com (Kurt A. Richardson) Date: Wed, 20 Oct 2010 11:45:20 -0700 Subject: [IronPython] cannot access protected member GaugeContainer1 without a python subclass of default_aspx Message-ID: <4CBF38C0.6000001@kurtrichardson.com> Hi List I have a (relatively) simple webpage that enables me to display some basic data from my TED5000 energy monitor, as well as switch a fan on and off in my offer utilizing an XBee network. I'm sure everything worked just fine for months and now suddenly the project will not compile. The Default.aspx webpage includes a Dundas GaugeContainer component instantiated as: ... The Dundas Gauge assembly is referenced correctly: <%@ Register Assembly="DundasWebGauge" Namespace="Dundas.Gauges.WebControl" TagPrefix="DGWC" %> The CodeFile (Default.aspx.py) contains some simple functions, but the one that throws an error is part of the Page_Load function: def Page_Load(sender,e): if not IsPostBack: gauge = GaugeContainer1 gauge.Callback += GaugeContainer1_Callback The third line (gauge = GaugeContainer1) results in the following parse error: *Parser Error Message: *cannot access protected member GaugeContainer1 without a python subclass of default_aspx Any ideas what is causing this error? Many thanks in advance for any assistance you can offer. Kind regards, Kurt -------------- next part -------------- An HTML attachment was scrubbed... URL: From kurt at kurtrichardson.com Wed Oct 20 20:47:16 2010 From: kurt at kurtrichardson.com (Kurt A. Richardson) Date: Wed, 20 Oct 2010 11:47:16 -0700 Subject: [IronPython] cannot access protected member GaugeContainer1 without a python subclass of default_aspx In-Reply-To: <4CBF38C0.6000001@kurtrichardson.com> References: <4CBF38C0.6000001@kurtrichardson.com> Message-ID: <4CBF3934.5020501@kurtrichardson.com> PS I'm using IPY 2.7 Alpha 1 with VS2010 .NET 4.0 On 10/20/2010 11:45 AM, Kurt A. Richardson wrote: > Hi List > > I have a (relatively) simple webpage that enables me to display some > basic data from my TED5000 energy monitor, as well as switch a fan on > and off in my offer utilizing an XBee network. I'm sure everything > worked just fine for months and now suddenly the project will not compile. > > The Default.aspx webpage includes a Dundas GaugeContainer component > instantiated as: > > runat="server">... > > The Dundas Gauge assembly is referenced correctly: > > <%@ Register Assembly="DundasWebGauge" > Namespace="Dundas.Gauges.WebControl" TagPrefix="DGWC" %> > > The CodeFile (Default.aspx.py) contains some simple functions, but the > one that throws an error is part of the Page_Load function: > > def Page_Load(sender,e): > if not IsPostBack: > gauge = GaugeContainer1 > gauge.Callback += GaugeContainer1_Callback > > The third line (gauge = GaugeContainer1) results in the following > parse error: > > *Parser Error Message: *cannot access protected member GaugeContainer1 > without a python subclass of default_aspx > > Any ideas what is causing this error? > > Many thanks in advance for any assistance you can offer. > > Kind regards, Kurt > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kurt at kurtrichardson.com Wed Oct 20 20:54:22 2010 From: kurt at kurtrichardson.com (Kurt A. Richardson) Date: Wed, 20 Oct 2010 11:54:22 -0700 Subject: [IronPython] cannot access protected member GaugeContainer1 without a python subclass of default_aspx In-Reply-To: <4CBF3934.5020501@kurtrichardson.com> References: <4CBF38C0.6000001@kurtrichardson.com> <4CBF3934.5020501@kurtrichardson.com> Message-ID: <4CBF3ADE.1090200@kurtrichardson.com> PPS As I went through commenting-out calls to page controls I found that I can't evenset-up the DropDownList I have on the page, let alone the Dundas ones. One thing I noticed is that the page declaration: <%@ Page Language="IronPython" Theme="MSH" EnableSessionState="true" CodeFile="Default.aspx.py" %> has a warning associated with the CodeFile: The language of the file referenced by the 'CodeFile' attribute does not match the language specified by the 'Language' attribute in the current file. On 10/20/2010 11:47 AM, Kurt A. Richardson wrote: > PS I'm using IPY 2.7 Alpha 1 with VS2010 .NET 4.0 > > On 10/20/2010 11:45 AM, Kurt A. Richardson wrote: >> Hi List >> >> I have a (relatively) simple webpage that enables me to display some >> basic data from my TED5000 energy monitor, as well as switch a fan on >> and off in my offer utilizing an XBee network. I'm sure everything >> worked just fine for months and now suddenly the project will not >> compile. >> >> The Default.aspx webpage includes a Dundas GaugeContainer component >> instantiated as: >> >> > runat="server">... >> >> The Dundas Gauge assembly is referenced correctly: >> >> <%@ Register Assembly="DundasWebGauge" >> Namespace="Dundas.Gauges.WebControl" TagPrefix="DGWC" %> >> >> The CodeFile (Default.aspx.py) contains some simple functions, but >> the one that throws an error is part of the Page_Load function: >> >> def Page_Load(sender,e): >> if not IsPostBack: >> gauge = GaugeContainer1 >> gauge.Callback += GaugeContainer1_Callback >> >> The third line (gauge = GaugeContainer1) results in the following >> parse error: >> >> *Parser Error Message: *cannot access protected member >> GaugeContainer1 without a python subclass of default_aspx >> >> Any ideas what is causing this error? >> >> Many thanks in advance for any assistance you can offer. >> >> Kind regards, Kurt >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kurt at kurtrichardson.com Wed Oct 20 21:07:02 2010 From: kurt at kurtrichardson.com (Kurt A. Richardson) Date: Wed, 20 Oct 2010 12:07:02 -0700 Subject: [IronPython] cannot access protected member GaugeContainer1 without a python subclass of default_aspx In-Reply-To: <4CBF3ADE.1090200@kurtrichardson.com> References: <4CBF38C0.6000001@kurtrichardson.com> <4CBF3934.5020501@kurtrichardson.com> <4CBF3ADE.1090200@kurtrichardson.com> Message-ID: <4CBF3DD6.1000107@kurtrichardson.com> Some progress... It seems that the EnableSessionState="true" in the page declaration is causing the problem. If I remove it the page compiles and views OK. Any ideas why? Thanks, Kurt On 10/20/2010 11:54 AM, Kurt A. Richardson wrote: > PPS As I went through commenting-out calls to page controls I found > that I can't evenset-up the DropDownList I have on the page, let alone > the Dundas ones. > > One thing I noticed is that the page declaration: > > <%@ Page Language="IronPython" Theme="MSH" EnableSessionState="true" > CodeFile="Default.aspx.py" %> > > has a warning associated with the CodeFile: > > The language of the file referenced by the 'CodeFile' attribute does > not match the language specified by the 'Language' attribute in the > current file. > > On 10/20/2010 11:47 AM, Kurt A. Richardson wrote: >> PS I'm using IPY 2.7 Alpha 1 with VS2010 .NET 4.0 >> >> On 10/20/2010 11:45 AM, Kurt A. Richardson wrote: >>> Hi List >>> >>> I have a (relatively) simple webpage that enables me to display some >>> basic data from my TED5000 energy monitor, as well as switch a fan >>> on and off in my offer utilizing an XBee network. I'm sure >>> everything worked just fine for months and now suddenly the project >>> will not compile. >>> >>> The Default.aspx webpage includes a Dundas GaugeContainer component >>> instantiated as: >>> >>> >> runat="server">... >>> >>> The Dundas Gauge assembly is referenced correctly: >>> >>> <%@ Register Assembly="DundasWebGauge" >>> Namespace="Dundas.Gauges.WebControl" TagPrefix="DGWC" %> >>> >>> The CodeFile (Default.aspx.py) contains some simple functions, but >>> the one that throws an error is part of the Page_Load function: >>> >>> def Page_Load(sender,e): >>> if not IsPostBack: >>> gauge = GaugeContainer1 >>> gauge.Callback += GaugeContainer1_Callback >>> >>> The third line (gauge = GaugeContainer1) results in the following >>> parse error: >>> >>> *Parser Error Message: *cannot access protected member >>> GaugeContainer1 without a python subclass of default_aspx >>> >>> Any ideas what is causing this error? >>> >>> Many thanks in advance for any assistance you can offer. >>> >>> Kind regards, Kurt >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Thu Oct 21 19:32:36 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 21 Oct 2010 17:32:36 +0000 Subject: [IronPython] cannot access protected member GaugeContainer1 without a python subclass of default_aspx In-Reply-To: <4CBF3DD6.1000107@kurtrichardson.com> References: <4CBF38C0.6000001@kurtrichardson.com> <4CBF3934.5020501@kurtrichardson.com> <4CBF3ADE.1090200@kurtrichardson.com> <4CBF3DD6.1000107@kurtrichardson.com> Message-ID: What this error means is that you're attempting to access a member which is declared as protected on something which is not a Python subclass. This would be the same as: public class Foo { protected void Bar() { } } public class Bar { public Bar() { new Foo().Bar(); } } Here C# gives you the error: "error CS0122: 'Foo.Bar()' is inaccessible due to its protection level" In this case I think what's happening is a member is being generated in a class for the GaugeControl1 - you need to somehow make that member public. I don't actually know how to do that but hopefully you can do something like 'Accessibility="public"' onto the DGWC:GaugeContainer tag. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Kurt A. Richardson Sent: Wednesday, October 20, 2010 12:07 PM To: users at lists.ironpython.com Subject: Re: [IronPython] cannot access protected member GaugeContainer1 without a python subclass of default_aspx Some progress... It seems that the EnableSessionState="true" in the page declaration is causing the problem. If I remove it the page compiles and views OK. Any ideas why? Thanks, Kurt On 10/20/2010 11:54 AM, Kurt A. Richardson wrote: PPS As I went through commenting-out calls to page controls I found that I can't evenset-up the DropDownList I have on the page, let alone the Dundas ones. One thing I noticed is that the page declaration: <%@ Page Language="IronPython" Theme="MSH" EnableSessionState="true" CodeFile="Default.aspx.py" %> has a warning associated with the CodeFile: The language of the file referenced by the 'CodeFile' attribute does not match the language specified by the 'Language' attribute in the current file. On 10/20/2010 11:47 AM, Kurt A. Richardson wrote: PS I'm using IPY 2.7 Alpha 1 with VS2010 .NET 4.0 On 10/20/2010 11:45 AM, Kurt A. Richardson wrote: Hi List I have a (relatively) simple webpage that enables me to display some basic data from my TED5000 energy monitor, as well as switch a fan on and off in my offer utilizing an XBee network. I'm sure everything worked just fine for months and now suddenly the project will not compile. The Default.aspx webpage includes a Dundas GaugeContainer component instantiated as: ... The Dundas Gauge assembly is referenced correctly: <%@ Register Assembly="DundasWebGauge" Namespace="Dundas.Gauges.WebControl" TagPrefix="DGWC" %> The CodeFile (Default.aspx.py) contains some simple functions, but the one that throws an error is part of the Page_Load function: def Page_Load(sender,e): if not IsPostBack: gauge = GaugeContainer1 gauge.Callback += GaugeContainer1_Callback The third line (gauge = GaugeContainer1) results in the following parse error: Parser Error Message: cannot access protected member GaugeContainer1 without a python subclass of default_aspx Any ideas what is causing this error? Many thanks in advance for any assistance you can offer. Kind regards, Kurt -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Thu Oct 21 23:08:40 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 21 Oct 2010 21:08:40 +0000 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <7c4757f1f0618419ef8a198e7ed1bfb1@mail.gmail.com> Message-ID: Here's a fix for this. There will still be some cases w/ function calls consuming memory but it solves the keyword arg problem and it only changes one file. I've also checked the fix in and pushed it to CodePlex so you can just get MetaPythonFunction.cs from there (change #78756 - http://ironpython.codeplex.com/SourceControl/changeset/changes/78756 ). But here's the description if you want to patch it by hand. This won't be in the 2.7B1 release but is checked in now for subsequent releases. So in MetaPythonFunction.cs there's a class FunctionBinderHelper. It needs a new member variable: private bool _needCodeTest; // true if we need to test the code object (This is line 301 in the current 2.7 sources). There's a function GetComplexRestriction, it needs this code added: if (_extractedKeyword) { return BindingRestrictions.GetInstanceRestriction(_func.Expression, _func.Value); } else if (_needCodeTest) { return GetSimpleRestriction().Merge( BindingRestrictions.GetInstanceRestriction( Expression.Property( GetFunctionParam(), "__code__" ), _func.Value.__code__ ) ); } return GetSimpleRestriction(); } And finally in GetArgumentsForRule we need to set _needCodeTest instead of _extractedKeyword for ArgumentType.Named: case ArgumentType.Named: _extractedKeyword = true; _needCodeTest = true; You can also replace the other places where we sign to _extractedKeyword w/ assignmnets to _needCodeTest except for in ExtractDefaultValue. That one still depends upon the actual function object who's defaults are mutable. So as long as the issue here doesn't involve defaults (as in we don't consume any default values) then this will fix the problem. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ronnie Maor Sent: Monday, October 18, 2010 1:34 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? thanks - wanted to make sure you didn't miss it. we'll try to reduce the number of places where we're exposed to it in the meantime. On Mon, Oct 18, 2010 at 10:28 PM, Dino Viehland > wrote: Yep, I've seen it... still thinking about what should be done here. I certainly understand the problem and what will cause it but the fix is probably non trivial (but also probably well contained to one or two classes). It may take me a day or two to respond w/ something substantial. We probably need to change our kw-calling to use a pre-compiled rule. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ronnie Maor Sent: Monday, October 18, 2010 1:25 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Can someone from IPy team ack that you saw this? The issue is causing us a lot of trouble, so we'd really appreciate it if you could tell us how to fix - we've already built from source to fix a previous leak, so no problem building with another patch. BTW, the default value in the function definition is not needed, it's calling with named arguments that causes the issue. so this is a slightly simpler repro: def test_method(): for i in xrange(1000): def func(param): pass func(param = None) thanks! Ronnie On Mon, Oct 18, 2010 at 11:35 AM, Idan Zaltzberg > wrote: FYI, there is a bit simpler reproduction: def test_method(): for i in xrange(1000): def func(param = None): pass func(param = None) test_method() So, actually any use of keyword params in closure that are redefined causes the problem. From: Idan Zaltzberg [mailto:idan at cloudshare.com] Sent: Monday, October 18, 2010 11:10 AM To: 'Discussion of IronPython' Subject: RE: [IronPython] HostCodeHeap leakage? Ok, I finally succeeded in creating a simple reproduction for this problem. The following code generates a 1000 methods (according to the ".NET CLR JIT" performance counter), on Ipy 2.6.1 .Net 2.0 def test_method(): for i in xrange(1000): def func(*a,**kw): pass func(some_parm = None) test_method() This does not happen if you call f without keyword params (using the *a params is OK). If this is indeed a bug, we would like to know how to fix it in the code locally, if that is possible. Also, I am interested in what Ipy flow creates this methods, since I wasn't able to find the function in the code that does this generation Thanks From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Thursday, October 14, 2010 8:31 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? So from an IronPython/DLR perspective the process should stabilize over time. That could take a while - as various Python functions get used repeatedly we'll switch from interpreting them to compiling them. We'll also potentially produce new call site rules which are compiled. That could account for the increase of 16k->18k dynamic methods. That's a 12% increase and could be a reasonable amount of it remained steady from then on. Likewise the # of function codes seems stable but the _codeCount is rising. That might mean that you're defining new functions (via exec/execfile/compile) and they're getting collected - but we still have their (dead) weak references in the code list. Eventually that list should get cleaned out when we hit context._nextCodeCleanup (which should be greater than context._codeCount). I would expect if you were to walk the entire PythonContext._allCodes linked list that you'd see about half of the lists having a dead weak reference. If your windbg-foo is up for writing the script to do this that'd be great but mine is rusty enough I'd need to look it up in the documentation. As for jitted code - all dynamic methods are collectible and any normal RefEmit code is not collectible (there's an option to make assemblies collectible in .NET 4.0, but we don't use it as we generally don't generate that many types). Oh, we do also generate new types for subclasses but you should see that NewTypeMaker._newTypes has a stable count over time because we share these between types w/ common bases. Closures, callbacks, generators should all be fine. If you do .symfix, then .reload, in Windbg does "dt HostCodeHeap" show you the fields of the HostCodeHeap structure? Does !eeheap -loader give you the address of the HostCodeHeap? It might be useful to know what m_allocationCount is on the heap overtime as if that's relatively stable the heap could be getting fragmented. Looking at the CLR code I'm becoming more and more convinced HostCodeHeap is only used for dynamic methods so if that's growing then I'm thinking we would appear to be leaking dynamic methods. The only way I can think of figuring out where the allocations are coming from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow (hopefully this will show up in the public symbols). That may be difficult if you can't attach the debugger to the server but if the issue is blocking the server you could have a breakpoint here which does a stack trace and continues execution so you could inspect the stack trace later. I'd include both "kb" and "!ClrStack" from sos in the stack trace. 2.7 shouldn't really change this - whether .NET 4.0 would or not would depend on if the CLR changed anything here. But I'm not sure - I would assume it wouldn't. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Thursday, October 14, 2010 6:44 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Hi, I've been looking on this for some time, and I'm still don't understand some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET 2.0) a bit better. We have a long running (stateful) application that we cannot simulate or run with a debugger open (so no breakpoints). However, since the application is run on a VM, we can take snapshots of it and then open WinDbg instance and break in the middle of the application. We do this a few hours after the application restarted and again after two days so we can see the difference. This way we saw that Jit Code Heap is increasing by a few hundred MB per day, and the number of HostCodeHeap objects is increasing. We also compared the performance counters for the two snapshots, and saw the Jitted Code Bytes increased from 100MB to 862MB and the number of methods jitted increased from 700K to 6.3M. In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to 424MB. On the other hand, the object types you mentioned stay relatively the same: * DynamicMethods count went from 16K to 18K * FunctionCode count went from 4013 to 4025 * The _codeCount field in the PythonContext went from 4447 to 7800 Here is what we don't understand: 1. Is it normal for the application to keep jitting code and methods forever? Should is stabilize? 2. From the numbers I guess that some of the jitted code IS collected. Which types are collectable and which are not? How can I tell which ones I am using? 3. Are there any specific patterns I should avoid to decrease uncollectable code (or jitting in general). I am using a lot of closures, callbacks and generators. 4. What datatypes that are visible in WinDbg can I use to understand if and why IronPython is generating uncollectable code? 5. Is there a way to trace back the HostCodeHeap objects to my code (or IronPython specific features)? 6. Can I expect an improvement in these issues by moving to Ipy 2.7 and/or .Net 4.0? Thanks From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Wednesday, October 13, 2010 10:01 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs and DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we're continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspx leads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there's exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There's also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn't actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they're compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it's likely that it is one of the things I mentioned (or some other cache that's per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That's a bunch of different things to look at - hopefully it'll give some insight into what's going on and help track down the issue. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Wednesday, October 13, 2010 12:12 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 6:59 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as -X:Debug. In general I'd suggest making this configurable somehow and only turn it on if you're actually debugging. It's unfortunate that we can't offer both debugging & collectability but right now that's simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 9:09 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like -X:Debug? You reckon this could be the cause? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 5:53 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? My guess is that's code in the JIT heap that's building up but I'm not 100% certain. How is your code being executed? Do you have the debug option (-D or -X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 2:26 AM To: Discussion of IronPython Subject: [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Fri Oct 22 01:01:20 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 21 Oct 2010 23:01:20 +0000 Subject: [IronPython] Announcing IronPython 2.7 Beta 1 Message-ID: Hello Python Community, We're pleased to announce the Beta release of IronPython 2.7 which can be downloaded at http://ironpython.codeplex.com/releases/view/48818. This is a major new version of IronPython with a number of significant updates. As the first beta release in the 2.7 series this represents a feature complete release which is compatible w/ CPython 2.7. This is the last release from Microsoft before turning these projects over to the new coordinators [see http://tinyurl.com/24tjztk for more information]. Changes thus far include: * Updates the language to be compatible with CPython 2.7 * Improves integrated Visual Studio support (IronPython Tools for Visual Studio) * Extends CPython 2.7's documentation with useful information pertaining to IronPython * Adds the mmap and signal modules * Includes a number of performance updates and bug fixes * Requires .NET 4.0 and Silverlight 4.0 Python 2.7 includes a number of features backported from the Python 3.0 series. This release implements the new builtin _io module, includes dictionary and set comprehensions, set literals, supports multiple context managers in the with statement, and adds several new functions to the itertools methods, and auto indexing for the new string formatting. There are also numerous updates to the standard library such as ordered dictionaries and the new argparse module. This release also includes a "IronPython Tools for Visual Studio" option within the IronPython installer. This enables one install to get both IronPython and IronPython Visual Studio support assuming you have an existing installation of Visual Studio 2010. This version of IronPython Tools includes a number of bug fixes as improved WPF designer support. The designer fully supports XAML and WPF including data binding to Python classes dynamically. We've also updated the IronPython installer to include documentation based upon the CPython documentation. This new .chm file includes documentation on the Python language and standard library. It's been extended from the normal Python documentation to include IronPython specific topics such as the DLR hosting APIs and extending IronPython from statically typed .NET languages. We flushed out more support for missing built-in modules which CPython includes. This release includes the mmap and signal modules bringing better support for interoperating with unmanaged code. As usual there are a number of bug fixes and performance improvements. This release includes major performance improvements in cPickle, the sum built-in function, and includes support for fast exceptions which do not use the .NET exception mechanism. There have also been improvements to significantly reduce memory usage of the IronPython ASTs. One of the end results of these numerous improvements is that IronPython's startup time has decreased by 10% when compared to IronPython 2.6.1. - The IronPython Team From dinov at microsoft.com Fri Oct 22 01:01:17 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 21 Oct 2010 23:01:17 +0000 Subject: [IronPython] Announcing IronPython 2.6.2 Message-ID: Hello Python Community, We're pleased to announce the final release of IronPython 2.6.2 which can be downloaded at http://ironpython.codeplex.com/releases/view/41236. This is a minor update which fixes the most egregious issues discovered since 2.6.1. After the number of major changes we released in 2.6.1 this release keeps the number of changes to a minimum to ensure that it'll have great compatibility against the previous versions we've shipped. This is the last release from Microsoft before turning these projects over to the new coordinators [see http://tinyurl.com/24tjztk for more information]. The full list of issues fixed includes: 26593 pyc fails if a file contains a call to unicode 26940 Wrong line numbers in traceback when ecoding is specified 27547 Problem with ScriptSource.GetCodeProperties() (Fix type error when parsing invalid text "lambda") 27247 delattr doesn't work on instance attributes of user defined instance Fix a memory leak related to the id dispenser Fix stack overflow with a large number of nested if/elif blocks Special thanks to gjones , cendalc, fuzzyman , sstevenson, Chuck Jacobs ,and Idan Zaltzberg for reporting issues and making this a great release! Happy scripting! - The IronPython Team From dinov at microsoft.com Fri Oct 22 01:08:40 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 21 Oct 2010 23:08:40 +0000 Subject: [IronPython] New Components and Contributors for IronPython and IronRuby Message-ID: This is just in case you miss the tinyurl link in the release notes for 2.6.2 and 2.7B1. I'd encourage everyone to go over and read Jason's blog which contains an announcement about changes to the IronPython and IronRuby projects. http://blogs.msdn.com/b/jasonz/archive/2010/10/21/new-components-and-contributors-for-ironpython-and-ironruby.aspx -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Fri Oct 22 01:09:20 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Thu, 21 Oct 2010 17:09:20 -0600 Subject: [IronPython] The Future of IronPython Message-ID: As Dino recently posted, this is the last release of IronPython by Microsoft. As of today, myself, Michael Foord, Jimmy Schementi, and Miguel de Icaza are the coordinators for the IronPython project (Jimmy and Miguel will also handle IronRuby). More information can be found from Jason Zander: http://blogs.msdn.com/b/jasonz/archive/2010/10/21/new-components-and-contributors-for-ironpython-and-ironruby.aspx. Do note that Zander doesn't actually come out and *say* that Microsoft is no longer funding IronPython/IronRuby, but that's what has happened. Any future participation from MS employees will be on an unofficial, spare-time basis. All that said, I hope this isn't the end of IronPython. There's a good community here, and I think we can do just as good a job as Microsoft did. You can now contribute patches to IronPython, but that means you will have to if you want the it to flourish. A good Open Source community requires a core of regular contributors, but also requires a mass of occasional ones that are just scratching their particular itches. >From here, my goal is to get IronPython 2.7 final out the door in a timely manner. Let's leave discussions about things like source control or external libraries aside for a bit and get a solid release -- then we can start those discussions. I fear that starting a major overhaul will kill any momentum we have and cause the 2.7 release to be unacceptably delayed. I wouldn't have signed up to do this if I thought IronPython didn't have a future. - Jeff From ironpy at hugunin.net Fri Oct 22 01:12:25 2010 From: ironpy at hugunin.net (Jim Hugunin) Date: Thu, 21 Oct 2010 16:12:25 -0700 Subject: [IronPython] My Farewell to IronPython and Microsoft Message-ID: Today marks the end of a crazy six year journey for me at Microsoft. I clearly remember my brutal first 8 months at this company as I worked with lawyers, marketing folks and execs to figure out if and how we could release IronPython as an open source project from Microsoft. The final approval for that release came the night before I was slated to give a keynote talk at the annual Python conference ? with no backup plan. That talk was immense fun and I appreciated the willingness of the Python community to consider IronPython on its merits ? and also to join me in my fantasy that getting to this point had only taken a reasonable two months rather than the actual eight. That first conference started many conversations with the community about what it meant to be true to the Python language. We made some major changes to how .NET methods were exposed based on just those first conversations. Since then we?ve done our best to perform a balancing act between being true to Microsoft and .NET and true to the Python community. That first release of IronPython was clearly broken in many ways. It was released under an "open source" license that was specific to Microsoft and not trusted by the community. Over the years that evolved into an OSI approved license and finally into the well-known and trusted Apache license. Similarly, that first version had huge technical holes such as the fact that all the dynamically generated code it created could never be garbage collected leading to a huge memory leak. This particular issue was fixed by adding powerful DynamicMethods to the .NET 2.0 framework. Through the years, we?ve been making steady progress on these technical and community engagement issues. The culmination for me of the work on IronPython was the creation of the Dynamic Language Runtime (DLR) which brought many of the ideas we developed for IronPython deep into the .NET platform. The coolest part for me is the ability for different languages to interoperate with each other by building on a common layer. I love being able to use IronRuby to call into my favorite Python libraries. The greatest joy was working with the team that added support for the new "dynamic" keyword to C#. I?ve always been a neutral in the ongoing war between the proponents of dynamic and static typing and it was really satisfying to add optional dynamic typing to C# by building on top of its rich static type system. Best of all, I don?t think I?ll ever get over the kick I get out of talking about the static type called "dynamic". Microsoft?s decision to abandon its investment in IronPythonwas a catalyst but not the cause of my leaving the company. While most of you know that I haven't been directly involved in IronPython for quite some time, the decision still provided a spur to cause me to reflect on my time at the company and realize that it was time to explore other career options. There would be something emotionally satisfying about leaving Microsoft in a fit of rage ? preferably involving the illegal deployment of an emergency escape slide. However, I leave feeling respect for the many great people and products produced here. I will suffer some pain when I have to write code in Java now that I?ve learned to love the elegance of C#. I will suffer some frustrations when I have to use Google Docs instead of the finely polished UI in Microsoft Office. More than anything, I will always value the chance that I had to work with and learn valuable lessons from some truly great people. As I leave Microsoft, I?m incredibly excited to be going to work for Google. I like to build projects with small talented teams working on quick cycles driven by iterative feedback from users. I like to have a healthy relationship with Open Source code and communities, and I believe that the future lies in the cloud and the web. These things are all possible to do at Microsoft and IronPython is a testament to that. However, making that happen at Microsoft always felt like trying to fit a square peg into a round hole ? which can be done but only at major cost to both the peg and the hole. I?m excited to be going somewhere that fits my natural instincts for how to build great software and has demonstrated how successful this approach can be. I?m even pretty sure that I?ll grow to love Google Docs as it continues to rapidly improve through great engineering combined with continuous iterative feedback. Given my new employer, I will be throwing my lot in with the Java side of the virtual machine world. I think that C# has truly evolved into a nicer language than Java and that .NET has some cool features that the JVM is missing. However, I also see great things in the Java world both technically with features like the adaptive compilation in HotSpot and more significantly in terms of the vibrant community it has managed to create that adds huge value to the platform. When I weigh them both in the balance, neither side has a clear advantage. I respect Google?s decision to standardize on a uniform set of primary programming languages with Python, JavaScript, Java and C++. I don?t see any reason to push against that set ? particularly if it means I get to consider Python a primary language! This also means that I am leaving the IronPython project. The four people named as initial coordinators are fantastic and it would be a pleasure to work with them. It would also be very satisfying to work on IronPython outside of the corporate constraints I?ve been living in for the past six years. I have great hope for this project in these new hands and look forward to watching their future successes. So long and thanks for all the fish - Jim Hugunin (original of this message is posted at http://hugunin.net/microsoft_farewell.html) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Fri Oct 22 01:19:23 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Thu, 21 Oct 2010 17:19:23 -0600 Subject: [IronPython] My Farewell to IronPython and Microsoft In-Reply-To: References: Message-ID: On Thu, Oct 21, 2010 at 5:12 PM, Jim Hugunin wrote: > This also means that I am leaving the IronPython project. The four people > named as initial coordinators are fantastic and it would be a pleasure to > work with them. It would also be very satisfying to work on IronPython > outside of the corporate constraints I?ve been living in for the past six > years. I have great hope for this project in these new hands and look > forward to watching their future successes. Thank you for everything you've done, Jim - despite your best efforts prove otherwise, .NET turned out to be a pretty darn good platform for dynamic languages after all :). I think we would all love to have you back on the project, if you so desire. Best of luck with your new job! - Jeff From charles.c.strahan at gmail.com Fri Oct 22 02:22:53 2010 From: charles.c.strahan at gmail.com (Charles Strahan) Date: Thu, 21 Oct 2010 19:22:53 -0500 Subject: [IronPython] The Future of IronPython In-Reply-To: References: Message-ID: While I am a little bummed that Microsoft is cutting funding for these Iron* projects, I do think it's great news that the ball is now in our court. What's more concerning is the fact that Microsoft, as of right now, has little incentive (that I'm aware of) to extend the DLR. -Charles On Thu, Oct 21, 2010 at 6:09 PM, Jeff Hardy wrote: > As Dino recently posted, this is the last release of IronPython by > Microsoft. As of today, myself, Michael Foord, Jimmy Schementi, and > Miguel de Icaza are the coordinators for the IronPython project (Jimmy > and Miguel will also handle IronRuby). More information can be found > from Jason Zander: > > http://blogs.msdn.com/b/jasonz/archive/2010/10/21/new-components-and-contributors-for-ironpython-and-ironruby.aspx > . > Do note that Zander doesn't actually come out and *say* that Microsoft > is no longer funding IronPython/IronRuby, but that's what has > happened. Any future participation from MS employees will be on an > unofficial, spare-time basis. > > All that said, I hope this isn't the end of IronPython. There's a good > community here, and I think we can do just as good a job as Microsoft > did. You can now contribute patches to IronPython, but that means you > will have to if you want the it to flourish. A good Open Source > community requires a core of regular contributors, but also requires a > mass of occasional ones that are just scratching their particular > itches. > > From here, my goal is to get IronPython 2.7 final out the door in a > timely manner. Let's leave discussions about things like source > control or external libraries aside for a bit and get a solid release > -- then we can start those discussions. I fear that starting a major > overhaul will kill any momentum we have and cause the 2.7 release to > be unacceptably delayed. > > I wouldn't have signed up to do this if I thought IronPython didn't > have a future. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charles.c.strahan at gmail.com Fri Oct 22 02:31:13 2010 From: charles.c.strahan at gmail.com (Charles Strahan) Date: Thu, 21 Oct 2010 19:31:13 -0500 Subject: [IronPython] My Farewell to IronPython and Microsoft In-Reply-To: References: Message-ID: I'd like to second that - thank you, Jim, for your contributions to the .NET community and for making dynamic languages on the CLR a reality. Congrats on your new job, and best of luck! -Charles On Thu, Oct 21, 2010 at 6:19 PM, Jeff Hardy wrote: > On Thu, Oct 21, 2010 at 5:12 PM, Jim Hugunin wrote: > > This also means that I am leaving the IronPython project. The four people > > named as initial coordinators are fantastic and it would be a pleasure to > > work with them. It would also be very satisfying to work on IronPython > > outside of the corporate constraints I?ve been living in for the past six > > years. I have great hope for this project in these new hands and look > > forward to watching their future successes. > > Thank you for everything you've done, Jim - despite your best efforts > prove otherwise, .NET turned out to be a pretty darn good platform for > dynamic languages after all :). I think we would all love to have you > back on the project, if you so desire. Best of luck with your new job! > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From grauenwolf at gmail.com Fri Oct 22 04:25:41 2010 From: grauenwolf at gmail.com (Jonathan Allen) Date: Thu, 21 Oct 2010 19:25:41 -0700 Subject: [IronPython] The Future of IronPython Message-ID: <4cc0f625.077adc0a.4291.6109@mx.google.com> Are there any important features missing from the DLR? Jonathan -----Original Message----- From: Charles Strahan Sent: Thursday, October 21, 2010 5:22 PM To: Discussion of IronPython Subject: Re: [IronPython] The Future of IronPython While I am a little bummed that Microsoft is cutting funding for these Iron* projects, I do think it's great news that the ball is now in our court. What's more concerning is the fact that Microsoft, as of right now, has little incentive (that I'm aware of) to extend the DLR. -Charles On Thu, Oct 21, 2010 at 6:09 PM, Jeff Hardy wrote: > As Dino recently posted, this is the last release of IronPython by > Microsoft. As of today, myself, Michael Foord, Jimmy Schementi, and > Miguel de Icaza are the coordinators for the IronPython project (Jimmy > and Miguel will also handle IronRuby). More information can be found > from Jason Zander: > > http://blogs.msdn.com/b/jasonz/archive/2010/10/21/new-components-and-contributors-for-ironpython-and-ironruby.aspx > . > Do note that Zander doesn't actually come out and *say* that Microsoft > is no longer funding IronPython/IronRuby, but that's what has > happened. Any future participation from MS employees will be on an > unofficial, spare-time basis. > > All that said, I hope this isn't the end of IronPython. There's a good > community here, and I think we can do just as good a job as Microsoft > did. You can now contribute patches to IronPython, but that means you > will have to if you want the it to flourish. A good Open Source > community requires a core of regular contributors, but also requires a > mass of occasional ones that are just scratching their particular > itches. > > From here, my goal is to get IronPython 2.7 final out the door in a > timely manner. Let's leave discussions about things like source > control or external libraries aside for a bit and get a solid release > -- then we can start those discussions. I fear that starting a major > overhaul will kill any momentum we have and cause the 2.7 release to > be unacceptably delayed. > > I wouldn't have signed up to do this if I thought IronPython didn't > have a future. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > From tony.meyer at gmail.com Fri Oct 22 07:01:33 2010 From: tony.meyer at gmail.com (Tony Meyer) Date: Fri, 22 Oct 2010 18:01:33 +1300 Subject: [IronPython] The Future of IronPython In-Reply-To: <4cc0f625.077adc0a.4291.6109@mx.google.com> References: <4cc0f625.077adc0a.4291.6109@mx.google.com> Message-ID: On Fri, Oct 22, 2010 at 3:25 PM, Jonathan Allen wrote: > Are there any important features missing from the DLR? I would say that the issue is more that if something arose in the future (e.g. when implementing IronPython 3 or as some sort of IronPython-specific enhancement) that needed DLR changes, that would previously have been (I assume) possible, and is now (again, I assume) difficult. I would guess that this puts IronPython in a similar situation (in this respect) to Jython (although since Java is open-source, I suppose there is the possibility of pushing upstream there). I don't know enough about the way that IronPython and the DLR interact to know how likely it is that changes like this would be needed. There doesn't seem to be a CPython equivalent, unless you consider that CPython is unable to change the instruction sets that the CPython interpreter runs on (which is stretching the similarity really). Cheers, Tony From jdhardy at gmail.com Fri Oct 22 07:13:18 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Thu, 21 Oct 2010 23:13:18 -0600 Subject: [IronPython] The Future of IronPython In-Reply-To: <4cc0f625.077adc0a.4291.6109@mx.google.com> References: <4cc0f625.077adc0a.4291.6109@mx.google.com> Message-ID: On Thu, Oct 21, 2010 at 8:25 PM, Jonathan Allen wrote: > Are there any important features missing from the DLR? There are two parts to the DLR: the "inner ring" and the "outer ring". The inner ring is the expression tree extensions, call sites, and some other infrastructure stuff - this is what shipped with .NET 4 was previously in Microsoft.Scripting.Core.dll. The outer ring in the hosting APIs - ScriptEngines, Scopes, etc. It is in the hands of the community now and we can do with it as we please. As far as I know, the inner ring/.NET 4 parts are pretty solid - there may be a few changes, but they're likely pretty minor, and hopefully MS is willing to work with us (and projects like IronJS, Coljure-CLR, & IronScheme) if changes are needed. The outer ring is also pretty solid as well, as far as I know, but it's not a concern since it can be changed at will (except that multiple projects depend on it, so some caution would still be needed). We might have to start calling that part DLR+ or something to avoid confusion. - Jeff From idan at cloudshare.com Fri Oct 22 09:13:39 2010 From: idan at cloudshare.com (Idan Zaltzberg) Date: Fri, 22 Oct 2010 09:13:39 +0200 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: References: <7c4757f1f0618419ef8a198e7ed1bfb1@mail.gmail.com> Message-ID: <781eb71519308c350a2515a76337b7b4@mail.gmail.com> Thanks! We will try it. Just a couple of questions: 1. I guess this will not be in the current release of ipy 2.6.2, right? *2. *I didn?t understand you last comment *"**You can also replace the other places where we sign to _extractedKeyword w/ assignmnets to _needCodeTest except for in ExtractDefaultValue. That one still depends upon the actual function object who?s defaults are mutable. So as long as the issue here doesn?t involve defaults (as in we don?t consume any default values) then this will fix the problem."** *can you please elaborate on that?** *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Thursday, October 21, 2010 11:09 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Here?s a fix for this. There will still be some cases w/ function calls consuming memory but it solves the keyword arg problem and it only changes one file. I?ve also checked the fix in and pushed it to CodePlex so you can just get MetaPythonFunction.cs from there (change #78756 - http://ironpython.codeplex.com/SourceControl/changeset/changes/78756 ). But here?s the description if you want to patch it by hand. This won?t be in the 2.7B1 release but is checked in now for subsequent releases. So in MetaPythonFunction.cs there?s a class FunctionBinderHelper. It needs a new member variable: private bool _needCodeTest; // true if we need to test the code object (This is line 301 in the current 2.7 sources). There?s a function GetComplexRestriction, it needs this code added: if (_extractedKeyword) { return BindingRestrictions.GetInstanceRestriction(_func.Expression, _func.Value); } else if (_needCodeTest) { return GetSimpleRestriction().Merge( BindingRestrictions.GetInstanceRestriction( Expression.Property( GetFunctionParam(), "__code__" ), _func.Value.__code__ ) ); } return GetSimpleRestriction(); } And finally in GetArgumentsForRule we need to set _needCodeTest instead of _extractedKeyword for ArgumentType.Named: case ArgumentType.Named: _extractedKeyword = true; _needCodeTest = true; You can also replace the other places where we sign to _extractedKeyword w/ assignmnets to _needCodeTest *except* for in ExtractDefaultValue. That one still depends upon the actual function object who?s defaults are mutable. So as long as the issue here doesn?t involve defaults (as in we don?t consume any default values) then this will fix the problem. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Ronnie Maor *Sent:* Monday, October 18, 2010 1:34 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? thanks - wanted to make sure you didn't miss it. we'll try to reduce the number of places where we're exposed to it in the meantime. On Mon, Oct 18, 2010 at 10:28 PM, Dino Viehland wrote: Yep, I?ve seen it? still thinking about what should be done here. I certainly understand the problem and what will cause it but the fix is probably non trivial (but also probably well contained to one or two classes). It may take me a day or two to respond w/ something substantial. We probably need to change our kw-calling to use a pre-compiled rule. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Ronnie Maor *Sent:* Monday, October 18, 2010 1:25 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Can someone from IPy team ack that you saw this? The issue is causing us a lot of trouble, so we'd really appreciate it if you could tell us how to fix - we've already built from source to fix a previous leak, so no problem building with another patch. BTW, the default value in the function definition is not needed, it's calling with named arguments that causes the issue. so this is a slightly simpler repro: def test_method(): for i in xrange(1000): def func(param): pass func(param = None) thanks! Ronnie On Mon, Oct 18, 2010 at 11:35 AM, Idan Zaltzberg wrote: FYI, there is a bit simpler reproduction: def test_method(): for i in xrange(1000): def func(param = None): pass func(param = None) test_method() So, actually any use of keyword params in closure that are redefined causes the problem. *From:* Idan Zaltzberg [mailto:idan at cloudshare.com] *Sent:* Monday, October 18, 2010 11:10 AM *To:* 'Discussion of IronPython' *Subject:* RE: [IronPython] HostCodeHeap leakage? Ok, I finally succeeded in creating a simple reproduction for this problem. The following code generates a 1000 methods (according to the ".NET CLR JIT" performance counter), on Ipy 2.6.1 .Net 2.0 def test_method(): for i in xrange(1000): def func(*a,**kw): pass func(some_parm = None) test_method() This does not happen if you call f without keyword params (using the *a params is OK). If this is indeed a bug, we would like to know how to fix it in the code locally, if that is possible. Also, I am interested in what Ipy flow creates this methods, since I wasn?t able to find the function in the code that does this generation Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Thursday, October 14, 2010 8:31 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? So from an IronPython/DLR perspective the process should stabilize over time. That could take a while ? as various Python functions get used repeatedly we?ll switch from interpreting them to compiling them. We?ll also potentially produce new call site rules which are compiled. That could account for the increase of 16k->18k dynamic methods. That?s a 12% increase and could be a reasonable amount of it remained steady from then on. Likewise the # of function codes seems stable but the _codeCount is rising. That might mean that you?re defining new functions (via exec/execfile/compile) and they?re getting collected - but we still have their (dead) weak references in the code list. Eventually that list should get cleaned out when we hit context._nextCodeCleanup (which should be greater than context._codeCount). I would expect if you were to walk the entire PythonContext._allCodes linked list that you?d see about half of the lists having a dead weak reference. If your windbg-foo is up for writing the script to do this that?d be great but mine is rusty enough I?d need to look it up in the documentation. As for jitted code ? all dynamic methods are collectible and any normal RefEmit code is not collectible (there?s an option to make assemblies collectible in .NET 4.0, but we don?t use it as we generally don?t generate that many types). Oh, we do also generate new types for subclasses but you should see that NewTypeMaker._newTypes has a stable count over time because we share these between types w/ common bases. Closures, callbacks, generators should all be fine. If you do .symfix, then .reload, in Windbg does ?dt HostCodeHeap? show you the fields of the HostCodeHeap structure? Does !eeheap ?loader give you the address of the HostCodeHeap? It might be useful to know what m_allocationCount is on the heap overtime as if that?s relatively stable the heap could be getting fragmented. Looking at the CLR code I?m becoming more and more convinced HostCodeHeap is only used for dynamic methods so if that?s growing then I?m thinking we would appear to be leaking dynamic methods. The only way I can think of figuring out where the allocations are coming from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow (hopefully this will show up in the public symbols). That may be difficult if you can?t attach the debugger to the server but if the issue is blocking the server you could have a breakpoint here which does a stack trace and continues execution so you could inspect the stack trace later. I?d include both ?kb? and ?!ClrStack? from sos in the stack trace. 2.7 shouldn?t really change this ? whether .NET 4.0 would or not would depend on if the CLR changed anything here. But I?m not sure ? I would assume it wouldn?t. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Thursday, October 14, 2010 6:44 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Hi, I've been looking on this for some time, and I'm still don?t understand some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET 2.0) a bit better. We have a long running (stateful) application that we cannot simulate or run with a debugger open (so no breakpoints). However, since the application is run on a VM, we can take snapshots of it and then open WinDbg instance and break in the middle of the application. We do this a few hours after the application restarted and again after two days so we can see the difference. This way we saw that Jit Code Heap is increasing by a few hundred MB per day, and the number of HostCodeHeap objects is increasing. We also compared the performance counters for the two snapshots, and saw the Jitted Code Bytes increased from 100MB to 862MB and the number of methods jitted increased from 700K to 6.3M. In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to 424MB. On the other hand, the object types you mentioned stay relatively the same: ? DynamicMethods count went from 16K to 18K ? FunctionCode count went from 4013 to 4025 ? The _*code*Count field in the PythonContext went from 4447 to 7800 Here is what we don't understand: 1. Is it normal for the application to keep jitting code and methods forever? Should is stabilize? 2. From the numbers I guess that some of the jitted code IS collected. Which types are collectable and which are not? How can I tell which ones I am using? 3. Are there any specific patterns I should avoid to decrease uncollectable code (or jitting in general). I am using a lot of closures, callbacks and generators. 4. What datatypes that are visible in WinDbg can I use to understand if and why IronPython is generating uncollectable code? 5. Is there a way to trace back the HostCodeHeap objects to my code (or IronPython specific features)? 6. Can I expect an improvement in these issues by moving to Ipy 2.7 and/or .Net 4.0? Thanks *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Wednesday, October 13, 2010 10:01 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs *and*DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we?re continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspxleads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there?s exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There?s also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn?t actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they?re compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it?s likely that it is one of the things I mentioned (or some other cache that?s per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That?s a bunch of different things to look at ? hopefully it?ll give some insight into what?s going on and help track down the issue. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Wednesday, October 13, 2010 12:12 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 6:59 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as ?X:Debug. In general I?d suggest making this configurable somehow and only turn it on if you?re actually debugging. It?s unfortunate that we can?t offer both debugging & collectability but right now that?s simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 9:09 AM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like ?X:Debug? You reckon this could be the cause? *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland *Sent:* Tuesday, October 05, 2010 5:53 PM *To:* Discussion of IronPython *Subject:* Re: [IronPython] HostCodeHeap leakage? My guess is that?s code in the JIT heap that?s building up but I?m not 100% certain. How is your code being executed? Do you have the debug option (-D or ?X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. *From:* users-bounces at lists.ironpython.com [mailto: users-bounces at lists.ironpython.com] *On Behalf Of *Idan Zaltzberg *Sent:* Tuesday, October 05, 2010 2:26 AM *To:* Discussion of IronPython *Subject:* [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From grauenwolf at gmail.com Fri Oct 22 11:22:09 2010 From: grauenwolf at gmail.com (Jonathan Allen) Date: Fri, 22 Oct 2010 02:22:09 -0700 Subject: [IronPython] The Future of IronPython In-Reply-To: References: <4cc0f625.077adc0a.4291.6109@mx.google.com> Message-ID: <045701cb71ca$9ba25270$d2e6f750$@gmail.com> Thank you for the explanation. Jonathan Allen 619-933-8527 -----Original Message----- From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Jeff Hardy Sent: Thursday, October 21, 2010 10:13 PM To: Discussion of IronPython Subject: Re: [IronPython] The Future of IronPython On Thu, Oct 21, 2010 at 8:25 PM, Jonathan Allen wrote: > Are there any important features missing from the DLR? There are two parts to the DLR: the "inner ring" and the "outer ring". The inner ring is the expression tree extensions, call sites, and some other infrastructure stuff - this is what shipped with .NET 4 was previously in Microsoft.Scripting.Core.dll. The outer ring in the hosting APIs - ScriptEngines, Scopes, etc. It is in the hands of the community now and we can do with it as we please. As far as I know, the inner ring/.NET 4 parts are pretty solid - there may be a few changes, but they're likely pretty minor, and hopefully MS is willing to work with us (and projects like IronJS, Coljure-CLR, & IronScheme) if changes are needed. The outer ring is also pretty solid as well, as far as I know, but it's not a concern since it can be changed at will (except that multiple projects depend on it, so some caution would still be needed). We might have to start calling that part DLR+ or something to avoid confusion. - Jeff _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com From cenovsky at bakalari.cz Fri Oct 22 14:31:08 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Fri, 22 Oct 2010 14:31:08 +0200 Subject: [IronPython] Announcing IronPython 2.6.2 In-Reply-To: References: Message-ID: <4CC1840C.5000401@bakalari.cz> Thank you! -- -- Luk?? On 22.10.2010 1:01, Dino Viehland wrote: > Hello Python Community, > > We're pleased to announce the final release of IronPython 2.6.2 which can be downloaded at http://ironpython.codeplex.com/releases/view/41236. This is a minor update which fixes the most egregious issues discovered since 2.6.1. After the number of major changes we released in 2.6.1 this release keeps the number of changes to a minimum to ensure that it'll have great compatibility against the previous versions we've shipped. This is the last release from Microsoft before turning these projects over to the new coordinators [see http://tinyurl.com/24tjztk for more information]. > > The full list of issues fixed includes: > 26593 pyc fails if a file contains a call to unicode > 26940 Wrong line numbers in traceback when ecoding is specified > 27547 Problem with ScriptSource.GetCodeProperties() (Fix type error when parsing invalid text "lambda") > 27247 delattr doesn't work on instance attributes of user defined instance > Fix a memory leak related to the id dispenser > Fix stack overflow with a large number of nested if/elif blocks > > Special thanks to gjones , cendalc, fuzzyman , sstevenson, Chuck Jacobs ,and Idan Zaltzberg for reporting issues and making this a great release! Happy scripting! > > - The IronPython Team > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > From adal.chiriliuc at gmail.com Fri Oct 22 14:44:28 2010 From: adal.chiriliuc at gmail.com (Adal Chiriliuc) Date: Fri, 22 Oct 2010 15:44:28 +0300 Subject: [IronPython] Announcing IronPython 2.7 Beta 1 In-Reply-To: References: Message-ID: Hi, Thanks for making available a new IronPython build. The 2.7 Beta 1 installer has a few issues through: 1. Changing the default installation folder doesn't work: IronPython will be installed in two places: "C:\Program Files (x86)\IronPython 2.7" and the installation folder that you specified. 2. If you have already have CPython installed, IronPython will overwrite the icon for .py files, but not for .pyc and .pyw files. Please don't replace the CPython icons or at least make this an installer option. Adal On Fri, Oct 22, 2010 at 2:01 AM, Dino Viehland wrote: > Hello Python Community, > > We're pleased to announce the Beta release of IronPython 2.7 which can be downloaded at http://ironpython.codeplex.com/releases/view/48818. This is a major new version of IronPython with a number of significant updates. As the first beta release in the 2.7 series this represents a feature complete release which is compatible w/ CPython 2.7. This is the last release from Microsoft before turning these projects over to the new coordinators [see http://tinyurl.com/24tjztk for more information]. > > Changes thus far include: > * Updates the language to be compatible with CPython 2.7 > * Improves integrated Visual Studio support (IronPython Tools for Visual Studio) > * Extends CPython 2.7's documentation with useful information pertaining to IronPython > * Adds the mmap and signal modules > * Includes a number of performance updates and bug fixes > * Requires .NET 4.0 and Silverlight 4.0 > > Python 2.7 includes a number of features backported from the Python 3.0 series. This release implements the new builtin _io module, includes dictionary and set comprehensions, set literals, supports multiple context managers in the with statement, and adds several new functions to the itertools methods, and auto indexing for the new string formatting. There are also numerous updates to the standard library such as ordered dictionaries and the new argparse module. > > This release also includes a "IronPython Tools for Visual Studio" option within the IronPython installer. This enables one install to get both IronPython and IronPython Visual Studio support assuming you have an existing installation of Visual Studio 2010. This version of IronPython Tools includes a number of bug fixes as improved WPF designer support. The designer fully supports XAML and WPF including data binding to Python classes dynamically. > > We've also updated the IronPython installer to include documentation based upon the CPython documentation. This new .chm file includes documentation on the Python language and standard library. It's been extended from the normal Python documentation to include IronPython specific topics such as the DLR hosting APIs and extending IronPython from statically typed .NET languages. > > We flushed out more support for missing built-in modules which CPython includes. This release includes the mmap and signal modules bringing better support for interoperating with unmanaged code. > > As usual there are a number of bug fixes and performance improvements. This release includes major performance improvements in cPickle, the sum built-in function, and includes support for fast exceptions which do not use the .NET exception mechanism. There have also been improvements to significantly reduce memory usage of the IronPython ASTs. One of the end results of these numerous improvements is that IronPython's startup time has decreased by 10% when compared to IronPython 2.6.1. > > - The IronPython Team > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > From pablodalma93 at hotmail.com Fri Oct 22 15:16:59 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Fri, 22 Oct 2010 10:16:59 -0300 Subject: [IronPython] The Future of IronPython In-Reply-To: References: <4cc0f625.077adc0a.4291.6109@mx.google.com>, Message-ID: Just curious, are Dino and Jimmy still working at Microsoft in other projects or they moved to another company if we might know? Greetings > Date: Thu, 21 Oct 2010 23:13:18 -0600 > From: jdhardy at gmail.com > To: users at lists.ironpython.com > Subject: Re: [IronPython] The Future of IronPython > > On Thu, Oct 21, 2010 at 8:25 PM, Jonathan Allen wrote: > > Are there any important features missing from the DLR? > > There are two parts to the DLR: the "inner ring" and the "outer ring". > The inner ring is the expression tree extensions, call sites, and some > other infrastructure stuff - this is what shipped with .NET 4 was > previously in Microsoft.Scripting.Core.dll. The outer ring in the > hosting APIs - ScriptEngines, Scopes, etc. It is in the hands of the > community now and we can do with it as we please. > > As far as I know, the inner ring/.NET 4 parts are pretty solid - there > may be a few changes, but they're likely pretty minor, and hopefully > MS is willing to work with us (and projects like IronJS, Coljure-CLR, > & IronScheme) if changes are needed. > > The outer ring is also pretty solid as well, as far as I know, but > it's not a concern since it can be changed at will (except that > multiple projects depend on it, so some caution would still be > needed). We might have to start calling that part DLR+ or something to > avoid confusion. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonvspython at gmail.com Fri Oct 22 16:21:04 2010 From: jonvspython at gmail.com (jon vs. python) Date: Fri, 22 Oct 2010 16:21:04 +0200 Subject: [IronPython] The Future of IronPython In-Reply-To: References: <4cc0f625.077adc0a.4291.6109@mx.google.com> Message-ID: Jim moved to Google. I don't know about Dino. On Fri, Oct 22, 2010 at 3:16 PM, Pablo Dalmazzo wrote: > Just curious, are Dino and Jimmy still working at Microsoft in other > projects or they moved to another company if we might know? > > Greetings > > > Date: Thu, 21 Oct 2010 23:13:18 -0600 > > From: jdhardy at gmail.com > > To: users at lists.ironpython.com > > > Subject: Re: [IronPython] The Future of IronPython > > > > On Thu, Oct 21, 2010 at 8:25 PM, Jonathan Allen > wrote: > > > Are there any important features missing from the DLR? > > > > There are two parts to the DLR: the "inner ring" and the "outer ring". > > The inner ring is the expression tree extensions, call sites, and some > > other infrastructure stuff - this is what shipped with .NET 4 was > > previously in Microsoft.Scripting.Core.dll. The outer ring in the > > hosting APIs - ScriptEngines, Scopes, etc. It is in the hands of the > > community now and we can do with it as we please. > > > > As far as I know, the inner ring/.NET 4 parts are pretty solid - there > > may be a few changes, but they're likely pretty minor, and hopefully > > MS is willing to work with us (and projects like IronJS, Coljure-CLR, > > & IronScheme) if changes are needed. > > > > The outer ring is also pretty solid as well, as far as I know, but > > it's not a concern since it can be changed at will (except that > > multiple projects depend on it, so some caution would still be > > needed). We might have to start calling that part DLR+ or something to > > avoid confusion. > > > > - Jeff > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdeville at microsoft.com Fri Oct 22 16:23:39 2010 From: jdeville at microsoft.com (Jim Deville) Date: Fri, 22 Oct 2010 14:23:39 +0000 Subject: [IronPython] The Future of IronPython In-Reply-To: References: <4cc0f625.077adc0a.4291.6109@mx.google.com> Message-ID: <571651983633624F9BA6BDE490351AEE063E74CB@TK5EX14MBXC201.redmond.corp.microsoft.com> Dino is still at MS, Jimmy (Schementi) went to Lab49. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of jon vs. python Sent: Friday, October 22, 2010 7:21 AM To: Discussion of IronPython Subject: Re: [IronPython] The Future of IronPython Jim moved to Google. I don't know about Dino. On Fri, Oct 22, 2010 at 3:16 PM, Pablo Dalmazzo > wrote: Just curious, are Dino and Jimmy still working at Microsoft in other projects or they moved to another company if we might know? Greetings > Date: Thu, 21 Oct 2010 23:13:18 -0600 > From: jdhardy at gmail.com > To: users at lists.ironpython.com > Subject: Re: [IronPython] The Future of IronPython > > On Thu, Oct 21, 2010 at 8:25 PM, Jonathan Allen > wrote: > > Are there any important features missing from the DLR? > > There are two parts to the DLR: the "inner ring" and the "outer ring". > The inner ring is the expression tree extensions, call sites, and some > other infrastructure stuff - this is what shipped with .NET 4 was > previously in Microsoft.Scripting.Core.dll. The outer ring in the > hosting APIs - ScriptEngines, Scopes, etc. It is in the hands of the > community now and we can do with it as we please. > > As far as I know, the inner ring/.NET 4 parts are pretty solid - there > may be a few changes, but they're likely pretty minor, and hopefully > MS is willing to work with us (and projects like IronJS, Coljure-CLR, > & IronScheme) if changes are needed. > > The outer ring is also pretty solid as well, as far as I know, but > it's not a concern since it can be changed at will (except that > multiple projects depend on it, so some caution would still be > needed). We might have to start calling that part DLR+ or something to > avoid confusion. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From curt at hagenlocher.org Fri Oct 22 16:24:57 2010 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Fri, 22 Oct 2010 07:24:57 -0700 Subject: [IronPython] The Future of IronPython In-Reply-To: References: <4cc0f625.077adc0a.4291.6109@mx.google.com> Message-ID: Jimmy and Jim are two different people. Jimmy Schementi left Microsoft almost three months ago and is now working for Lab49. He will continue to be closely involved in IronRuby and IronPython. Jim Hugunin has just left for Google, and is not expected to be involved in IronPython in the future. Dino's still here. On Fri, Oct 22, 2010 at 7:21 AM, jon vs. python wrote: > Jim moved to Google. I don't know about Dino. > > On Fri, Oct 22, 2010 at 3:16 PM, Pablo Dalmazzo wrote: > >> Just curious, are Dino and Jimmy still working at Microsoft in other >> projects or they moved to another company if we might know? >> >> Greetings >> >> > Date: Thu, 21 Oct 2010 23:13:18 -0600 >> > From: jdhardy at gmail.com >> > To: users at lists.ironpython.com >> >> > Subject: Re: [IronPython] The Future of IronPython >> > >> > On Thu, Oct 21, 2010 at 8:25 PM, Jonathan Allen >> wrote: >> > > Are there any important features missing from the DLR? >> > >> > There are two parts to the DLR: the "inner ring" and the "outer ring". >> > The inner ring is the expression tree extensions, call sites, and some >> > other infrastructure stuff - this is what shipped with .NET 4 was >> > previously in Microsoft.Scripting.Core.dll. The outer ring in the >> > hosting APIs - ScriptEngines, Scopes, etc. It is in the hands of the >> > community now and we can do with it as we please. >> > >> > As far as I know, the inner ring/.NET 4 parts are pretty solid - there >> > may be a few changes, but they're likely pretty minor, and hopefully >> > MS is willing to work with us (and projects like IronJS, Coljure-CLR, >> > & IronScheme) if changes are needed. >> > >> > The outer ring is also pretty solid as well, as far as I know, but >> > it's not a concern since it can be changed at will (except that >> > multiple projects depend on it, so some caution would still be >> > needed). We might have to start calling that part DLR+ or something to >> > avoid confusion. >> > >> > - Jeff >> > _______________________________________________ >> > Users mailing list >> > Users at lists.ironpython.com >> > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablodalma93 at hotmail.com Fri Oct 22 16:27:20 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Fri, 22 Oct 2010 11:27:20 -0300 Subject: [IronPython] The Future of IronPython In-Reply-To: References: <4cc0f625.077adc0a.4291.6109@mx.google.com>, , , , Message-ID: What do you mean by here, participating in the IronPython project or working in Microsoft? Date: Fri, 22 Oct 2010 07:24:57 -0700 From: curt at hagenlocher.org To: users at lists.ironpython.com Subject: Re: [IronPython] The Future of IronPython Jimmy and Jim are two different people. Jimmy Schementi left Microsoft almost three months ago and is now working for Lab49. He will continue to be closely involved in IronRuby and IronPython. Jim Hugunin has just left for Google, and is not expected to be involved in IronPython in the future. Dino's still here. On Fri, Oct 22, 2010 at 7:21 AM, jon vs. python wrote: Jim moved to Google. I don't know about Dino. On Fri, Oct 22, 2010 at 3:16 PM, Pablo Dalmazzo wrote: Just curious, are Dino and Jimmy still working at Microsoft in other projects or they moved to another company if we might know? Greetings > Date: Thu, 21 Oct 2010 23:13:18 -0600 > From: jdhardy at gmail.com > To: users at lists.ironpython.com > Subject: Re: [IronPython] The Future of IronPython > > On Thu, Oct 21, 2010 at 8:25 PM, Jonathan Allen wrote: > > Are there any important features missing from the DLR? > > There are two parts to the DLR: the "inner ring" and the "outer ring". > The inner ring is the expression tree extensions, call sites, and some > other infrastructure stuff - this is what shipped with .NET 4 was > previously in Microsoft.Scripting.Core.dll. The outer ring in the > hosting APIs - ScriptEngines, Scopes, etc. It is in the hands of the > community now and we can do with it as we please. > > As far as I know, the inner ring/.NET 4 parts are pretty solid - there > may be a few changes, but they're likely pretty minor, and hopefully > MS is willing to work with us (and projects like IronJS, Coljure-CLR, > & IronScheme) if changes are needed. > > The outer ring is also pretty solid as well, as far as I know, but > it's not a concern since it can be changed at will (except that > multiple projects depend on it, so some caution would still be > needed). We might have to start calling that part DLR+ or something to > avoid confusion. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Larry.Jones at aspentech.com Fri Oct 22 16:29:49 2010 From: Larry.Jones at aspentech.com (Jones, Larry) Date: Fri, 22 Oct 2010 10:29:49 -0400 Subject: [IronPython] My Farewell to IronPython and Microsoft In-Reply-To: References: Message-ID: +1. Great job to you and to the entire team. -----Original Message----- From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Jeff Hardy Sent: Thursday, October 21, 2010 6:19 PM To: Discussion of IronPython Subject: Re: [IronPython] My Farewell to IronPython and Microsoft On Thu, Oct 21, 2010 at 5:12 PM, Jim Hugunin wrote: > This also means that I am leaving the IronPython project. The four people > named as initial coordinators are fantastic and it would be a pleasure to > work with them. It would also be very satisfying to work on IronPython > outside of the corporate constraints I've been living in for the past six > years. I have great hope for this project in these new hands and look > forward to watching their future successes. Thank you for everything you've done, Jim - despite your best efforts prove otherwise, .NET turned out to be a pretty darn good platform for dynamic languages after all :). I think we would all love to have you back on the project, if you so desire. Best of luck with your new job! - Jeff _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify the sender and permanently delete the original copy and any copy of any e-mail, and any printout thereof. From curt at hagenlocher.org Fri Oct 22 16:30:25 2010 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Fri, 22 Oct 2010 07:30:25 -0700 Subject: [IronPython] The Future of IronPython In-Reply-To: References: <4cc0f625.077adc0a.4291.6109@mx.google.com> Message-ID: I meant "working at Microsoft". I expect he'll tell us what he's working on at some point. :) On Fri, Oct 22, 2010 at 7:27 AM, Pablo Dalmazzo wrote: > What do you mean by here, participating in the IronPython project or > working in Microsoft? > > ------------------------------ > Date: Fri, 22 Oct 2010 07:24:57 -0700 > From: curt at hagenlocher.org > To: users at lists.ironpython.com > Subject: Re: [IronPython] The Future of IronPython > > Jimmy and Jim are two different people. Jimmy Schementi left Microsoft > almost three months ago and is now working for Lab49. He will continue to be > closely involved in IronRuby and IronPython. Jim Hugunin has just left for > Google, and is not expected to be involved in IronPython in the future. > Dino's still here. > > On Fri, Oct 22, 2010 at 7:21 AM, jon vs. python wrote: > > Jim moved to Google. I don't know about Dino. > > On Fri, Oct 22, 2010 at 3:16 PM, Pablo Dalmazzo wrote: > > Just curious, are Dino and Jimmy still working at Microsoft in other > projects or they moved to another company if we might know? > > Greetings > > > Date: Thu, 21 Oct 2010 23:13:18 -0600 > > From: jdhardy at gmail.com > > To: users at lists.ironpython.com > > > Subject: Re: [IronPython] The Future of IronPython > > > > On Thu, Oct 21, 2010 at 8:25 PM, Jonathan Allen > wrote: > > > Are there any important features missing from the DLR? > > > > There are two parts to the DLR: the "inner ring" and the "outer ring". > > The inner ring is the expression tree extensions, call sites, and some > > other infrastructure stuff - this is what shipped with .NET 4 was > > previously in Microsoft.Scripting.Core.dll. The outer ring in the > > hosting APIs - ScriptEngines, Scopes, etc. It is in the hands of the > > community now and we can do with it as we please. > > > > As far as I know, the inner ring/.NET 4 parts are pretty solid - there > > may be a few changes, but they're likely pretty minor, and hopefully > > MS is willing to work with us (and projects like IronJS, Coljure-CLR, > > & IronScheme) if changes are needed. > > > > The outer ring is also pretty solid as well, as far as I know, but > > it's not a concern since it can be changed at will (except that > > multiple projects depend on it, so some caution would still be > > needed). We might have to start calling that part DLR+ or something to > > avoid confusion. > > > > - Jeff > > _______________________________________________ > > Users mailing list > > Users at lists.ironpython.com > > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Larry.Jones at aspentech.com Fri Oct 22 16:31:09 2010 From: Larry.Jones at aspentech.com (Jones, Larry) Date: Fri, 22 Oct 2010 10:31:09 -0400 Subject: [IronPython] The Future of IronPython In-Reply-To: References: Message-ID: Thanks, Jeff, for your past and future work. -----Original Message----- From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Jeff Hardy Sent: Thursday, October 21, 2010 6:09 PM To: Discussion of IronPython Subject: [IronPython] The Future of IronPython As Dino recently posted, this is the last release of IronPython by Microsoft. As of today, myself, Michael Foord, Jimmy Schementi, and Miguel de Icaza are the coordinators for the IronPython project (Jimmy and Miguel will also handle IronRuby). More information can be found from Jason Zander: http://blogs.msdn.com/b/jasonz/archive/2010/10/21/new-components-and-con tributors-for-ironpython-and-ironruby.aspx. Do note that Zander doesn't actually come out and *say* that Microsoft is no longer funding IronPython/IronRuby, but that's what has happened. Any future participation from MS employees will be on an unofficial, spare-time basis. All that said, I hope this isn't the end of IronPython. There's a good community here, and I think we can do just as good a job as Microsoft did. You can now contribute patches to IronPython, but that means you will have to if you want the it to flourish. A good Open Source community requires a core of regular contributors, but also requires a mass of occasional ones that are just scratching their particular itches. From here, my goal is to get IronPython 2.7 final out the door in a timely manner. Let's leave discussions about things like source control or external libraries aside for a bit and get a solid release -- then we can start those discussions. I fear that starting a major overhaul will kill any momentum we have and cause the 2.7 release to be unacceptably delayed. I wouldn't have signed up to do this if I thought IronPython didn't have a future. - Jeff _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify the sender and permanently delete the original copy and any copy of any e-mail, and any printout thereof. From vernondcole at gmail.com Fri Oct 22 16:46:36 2010 From: vernondcole at gmail.com (Vernon Cole) Date: Fri, 22 Oct 2010 08:46:36 -0600 Subject: [IronPython] Announcing IronPython 2.7 Beta 1 In-Reply-To: References: Message-ID: Your issue #2 is not a bug -- or at least it is the same as all other installers. Every version of Python which you install will take over as the default "Open" command for .py files -- and will supply it's own icon for them. When you are trying out a new Python version, you must remember to re-install your chosen default version afterward, if you expect to double-click a .py and get a certain version of Python to start. The same thing happens with Python 3.1 and 2.7 for example. I use so many different versions that I have given up on the re-install trick. I now use a bunch of .bat files (in a directory on my search path) to pick the version I want. For example, the file "py23.bat" contains: c:\python23\python.exe %1 %2 %3 %4 %5 %6 %7 %8 %9 and "py31.bat" contains: c:\python31\python.exe %1 %2 %3 %4 %5 %6 %7 %8 %9 and "ipy.bat" contains: "c:\program files\IronPython 2.7\ipy.exe" %1 %2 %3 %4 %5 %6 %7 %8 %9 The file "python.bat" points to my chosen default version for this week. So to test"myTestScript" under IronPython 2.7, I type: ipy myTestScript.py arg1 arg2 The same little c:\utils directory also contains such handy tools as tail.exe, unix2dos.exe, less.exe, and 2to3.bat, which contains: c:\python26\python.exe c:\python26\Tools\Scripts\2to3.py %1 %2 %3 %4 %5 %6 %7 %8 %9 You get the picture... Those of us who's brains are stuck in console mode do this sort of thing. -- Vernon On Fri, Oct 22, 2010 at 6:44 AM, Adal Chiriliuc wrote: > Hi, > > Thanks for making available a new IronPython build. The 2.7 Beta 1 > installer has a few issues through: > > 1. Changing the default installation folder doesn't work: IronPython > will be installed in two places: "C:\Program Files (x86)\IronPython > 2.7" and the installation folder that you specified. > > 2. If you have already have CPython installed, IronPython will > overwrite the icon for .py files, but not for .pyc and .pyw files. > Please don't replace the CPython icons or at least make this an > installer option. > > Adal > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at voidspace.org.uk Fri Oct 22 16:50:28 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Fri, 22 Oct 2010 15:50:28 +0100 Subject: [IronPython] Announcing IronPython 2.7 Beta 1 In-Reply-To: References: Message-ID: <4CC1A4B4.10909@voidspace.org.uk> On 22/10/2010 15:46, Vernon Cole wrote: > Your issue #2 is not a bug -- or at least it is the same as all other > installers. > Well, it's *new* as the IronPython installer has never done this before - and I would certainly question whether it is desirable for it to be the *default* for IronPython. IronPython doesn't use .pyc files (and not really .pyw either I think) so it isn't surprising it doesn't change those. I would really prefer that IronPython doesn't seize the association with .py files without explicit instruction from the user . Michael Foord > Every version of Python which you install will take over as the > default "Open" command for .py files -- and will supply it's own icon > for them. When you are trying out a new Python version, you must > remember to re-install your chosen default version afterward, if you > expect to double-click a .py and get a certain version of Python to > start. The same thing happens with Python 3.1 and 2.7 for example. I > use so many different versions that I have given up on the re-install > trick. I now use a bunch of .bat files (in a directory on my search > path) to pick the version I want. > > For example, the file "py23.bat" contains: > c:\python23\python.exe %1 %2 %3 %4 %5 %6 %7 %8 %9 > > and "py31.bat" contains: > c:\python31\python.exe %1 %2 %3 %4 %5 %6 %7 %8 %9 > > and "ipy.bat" contains: > "c:\program files\IronPython 2.7\ipy.exe" %1 %2 %3 %4 %5 %6 %7 %8 %9 > > The file "python.bat" points to my chosen default version for this week. > > So to test"myTestScript" under IronPython 2.7, I type: > ipy myTestScript.py arg1 arg2 > > The same little c:\utils directory also contains such handy tools as > tail.exe, unix2dos.exe, less.exe, and 2to3.bat, which contains: > c:\python26\python.exe c:\python26\Tools\Scripts\2to3.py %1 %2 %3 %4 > %5 %6 %7 %8 %9 > > You get the picture... Those of us who's brains are stuck in console > mode do this sort of thing. > -- > Vernon > > On Fri, Oct 22, 2010 at 6:44 AM, Adal Chiriliuc > > wrote: > > Hi, > > Thanks for making available a new IronPython build. The 2.7 Beta 1 > installer has a few issues through: > > 1. Changing the default installation folder doesn't work: IronPython > will be installed in two places: "C:\Program Files (x86)\IronPython > 2.7" and the installation folder that you specified. > > 2. If you have already have CPython installed, IronPython will > overwrite the icon for .py files, but not for .pyc and .pyw files. > Please don't replace the CPython icons or at least make this an > installer option. > > Adal > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cenovsky at bakalari.cz Fri Oct 22 16:59:22 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Fri, 22 Oct 2010 16:59:22 +0200 Subject: [IronPython] Announcing IronPython 2.7 Beta 1 In-Reply-To: References: Message-ID: <4CC1A6CA.100@bakalari.cz> I think there is an option to register .py extensions in the CPython installer. But it is hidden under some Extra settings button. -- -- Luk?s( On 22.10.2010 16:46, Vernon Cole wrote: > Your issue #2 is not a bug -- or at least it is the same as all other > installers. > > Every version of Python which you install will take over as the > default "Open" command for .py files -- and will supply it's own icon > for them. When you are trying out a new Python version, you must > remember to re-install your chosen default version afterward, if you > expect to double-click a .py and get a certain version of Python to > start. The same thing happens with Python 3.1 and 2.7 for example. I > use so many different versions that I have given up on the re-install > trick. I now use a bunch of .bat files (in a directory on my search > path) to pick the version I want. > > For example, the file "py23.bat" contains: > c:\python23\python.exe %1 %2 %3 %4 %5 %6 %7 %8 %9 > > and "py31.bat" contains: > c:\python31\python.exe %1 %2 %3 %4 %5 %6 %7 %8 %9 > > and "ipy.bat" contains: > "c:\program files\IronPython 2.7\ipy.exe" %1 %2 %3 %4 %5 %6 %7 %8 %9 > > The file "python.bat" points to my chosen default version for this week. > > So to test"myTestScript" under IronPython 2.7, I type: > ipy myTestScript.py arg1 arg2 > > The same little c:\utils directory also contains such handy tools as > tail.exe, unix2dos.exe, less.exe, and 2to3.bat, which contains: > c:\python26\python.exe c:\python26\Tools\Scripts\2to3.py %1 %2 %3 %4 > %5 %6 %7 %8 %9 > > You get the picture... Those of us who's brains are stuck in console > mode do this sort of thing. > -- > Vernon > > On Fri, Oct 22, 2010 at 6:44 AM, Adal Chiriliuc > > wrote: > > Hi, > > Thanks for making available a new IronPython build. The 2.7 Beta 1 > installer has a few issues through: > > 1. Changing the default installation folder doesn't work: IronPython > will be installed in two places: "C:\Program Files (x86)\IronPython > 2.7" and the installation folder that you specified. > > 2. If you have already have CPython installed, IronPython will > overwrite the icon for .py files, but not for .pyc and .pyw files. > Please don't replace the CPython icons or at least make this an > installer option. > > Adal > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Fri Oct 22 17:07:34 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Fri, 22 Oct 2010 10:07:34 -0500 Subject: [IronPython] Announcing IronPython 2.7 Beta 1 In-Reply-To: <4CC1A6CA.100@bakalari.cz> References: <4CC1A6CA.100@bakalari.cz> Message-ID: On Fri, Oct 22, 2010 at 09:59, Lukas Cenovsky wrote: > I think there is an option to register .py extensions in the CPython > installer. But it is hidden under some Extra settings button. > > -- > -- Luk?? > It's currently an option during the customization step, right after you choose where to install. The compilation to pyc files is probably the hidden part you are thinking of (accessible via the "Advanced" button on that same step). -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxyaffe at gmail.com Fri Oct 22 17:28:41 2010 From: maxyaffe at gmail.com (Max Yaffe) Date: Fri, 22 Oct 2010 11:28:41 -0400 Subject: [IronPython] Thank you all In-Reply-To: References: Message-ID: <7993FB06FBE14EB38343F91BAAC341FC@Gamry.com> Thank you Jim, Jimmy, Dino, and everyone else for all your hard work. And everyone else, too. So who's going to be working on Iron-C#? Max > Date: Thu, 21 Oct 2010 16:12:25 -0700 > From: Jim Hugunin > > Today marks the end of a crazy six year journey for me at > Microsoft. > ************************************* From jdhardy at gmail.com Fri Oct 22 17:34:11 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Fri, 22 Oct 2010 09:34:11 -0600 Subject: [IronPython] Thank you all In-Reply-To: <7993FB06FBE14EB38343F91BAAC341FC@Gamry.com> References: <7993FB06FBE14EB38343F91BAAC341FC@Gamry.com> Message-ID: On Fri, Oct 22, 2010 at 9:28 AM, Max Yaffe wrote: > So who's going to be working on Iron-C#? Mono already has a hostable C# compiler ... is that what you're thinking of? - Jeff From slide.o.mix at gmail.com Fri Oct 22 17:42:40 2010 From: slide.o.mix at gmail.com (Slide) Date: Fri, 22 Oct 2010 08:42:40 -0700 Subject: [IronPython] Thank you all In-Reply-To: References: <7993FB06FBE14EB38343F91BAAC341FC@Gamry.com> Message-ID: On Fri, Oct 22, 2010 at 8:34 AM, Jeff Hardy wrote: > On Fri, Oct 22, 2010 at 9:28 AM, Max Yaffe wrote: >> So who's going to be working on Iron-C#? > > Mono already has a hostable C# compiler ... is that what you're thinking of? > > - Jeff Miguel tweeted about turning it into a full fledged scripting language called S# at one point. slide From vernondcole at gmail.com Fri Oct 22 17:56:41 2010 From: vernondcole at gmail.com (Vernon Cole) Date: Fri, 22 Oct 2010 09:56:41 -0600 Subject: [IronPython] The Future of IronPython In-Reply-To: References: Message-ID: Jeff: Can your version of zlib be slipped in to v2.7 before final now? (please please pretty please) I would really like distutils, etc to actually work. -- Vernon Cole On Thu, Oct 21, 2010 at 5:09 PM, Jeff Hardy wrote: > As Dino recently posted, this is the last release of IronPython by > Microsoft. As of today, myself, Michael Foord, Jimmy Schementi, and > Miguel de Icaza are the coordinators for the IronPython project (Jimmy > and Miguel will also handle IronRuby). More information can be found > from Jason Zander: > > http://blogs.msdn.com/b/jasonz/archive/2010/10/21/new-components-and-contributors-for-ironpython-and-ironruby.aspx > . > Do note that Zander doesn't actually come out and *say* that Microsoft > is no longer funding IronPython/IronRuby, but that's what has > happened. Any future participation from MS employees will be on an > unofficial, spare-time basis. > > All that said, I hope this isn't the end of IronPython. There's a good > community here, and I think we can do just as good a job as Microsoft > did. You can now contribute patches to IronPython, but that means you > will have to if you want the it to flourish. A good Open Source > community requires a core of regular contributors, but also requires a > mass of occasional ones that are just scratching their particular > itches. > > From here, my goal is to get IronPython 2.7 final out the door in a > timely manner. Let's leave discussions about things like source > control or external libraries aside for a bit and get a solid release > -- then we can start those discussions. I fear that starting a major > overhaul will kill any momentum we have and cause the 2.7 release to > be unacceptably delayed. > > I wouldn't have signed up to do this if I thought IronPython didn't > have a future. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Fri Oct 22 18:15:30 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Fri, 22 Oct 2010 10:15:30 -0600 Subject: [IronPython] The Future of IronPython In-Reply-To: References: Message-ID: On Fri, Oct 22, 2010 at 9:56 AM, Vernon Cole wrote: > Jeff: > ? Can your version of zlib be slipped in to v2.7 before final now? (please > please pretty please) Well, the first issue is that it depends on another open source component (Component Ace's Zlib.Net - http://componentace.com/zlib_.NET.htm) with a different licence, and I don't know what the legal implications are of that. The second issue is that we (me and Michael) want to get a stable 2.7 out as soon as possible, and were not planning on adding community modules until the 2.7.1 release. However, zlib support is also the single highest voted issue on codeplex, and the module passes all CPython tests for zlib and gzip, so we might make an exception. Michael? - Jeff From dinov at microsoft.com Fri Oct 22 18:23:04 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 22 Oct 2010 16:23:04 +0000 Subject: [IronPython] The Future of IronPython In-Reply-To: References: <4cc0f625.077adc0a.4291.6109@mx.google.com> Message-ID: Yes, I'm still at Microsoft and I'm still working on something Python related - I even get to write some Python code now from time to time :). But I think that's about all I can say right now but I intend to be at PyCon this year still and hopefully will be giving a talk. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Curt Hagenlocher Sent: Friday, October 22, 2010 7:30 AM To: Discussion of IronPython Subject: Re: [IronPython] The Future of IronPython I meant "working at Microsoft". I expect he'll tell us what he's working on at some point. :) On Fri, Oct 22, 2010 at 7:27 AM, Pablo Dalmazzo > wrote: What do you mean by here, participating in the IronPython project or working in Microsoft? ________________________________ Date: Fri, 22 Oct 2010 07:24:57 -0700 From: curt at hagenlocher.org To: users at lists.ironpython.com Subject: Re: [IronPython] The Future of IronPython Jimmy and Jim are two different people. Jimmy Schementi left Microsoft almost three months ago and is now working for Lab49. He will continue to be closely involved in IronRuby and IronPython. Jim Hugunin has just left for Google, and is not expected to be involved in IronPython in the future. Dino's still here. On Fri, Oct 22, 2010 at 7:21 AM, jon vs. python > wrote: Jim moved to Google. I don't know about Dino. On Fri, Oct 22, 2010 at 3:16 PM, Pablo Dalmazzo > wrote: Just curious, are Dino and Jimmy still working at Microsoft in other projects or they moved to another company if we might know? Greetings > Date: Thu, 21 Oct 2010 23:13:18 -0600 > From: jdhardy at gmail.com > To: users at lists.ironpython.com > Subject: Re: [IronPython] The Future of IronPython > > On Thu, Oct 21, 2010 at 8:25 PM, Jonathan Allen > wrote: > > Are there any important features missing from the DLR? > > There are two parts to the DLR: the "inner ring" and the "outer ring". > The inner ring is the expression tree extensions, call sites, and some > other infrastructure stuff - this is what shipped with .NET 4 was > previously in Microsoft.Scripting.Core.dll. The outer ring in the > hosting APIs - ScriptEngines, Scopes, etc. It is in the hands of the > community now and we can do with it as we please. > > As far as I know, the inner ring/.NET 4 parts are pretty solid - there > may be a few changes, but they're likely pretty minor, and hopefully > MS is willing to work with us (and projects like IronJS, Coljure-CLR, > & IronScheme) if changes are needed. > > The outer ring is also pretty solid as well, as far as I know, but > it's not a concern since it can be changed at will (except that > multiple projects depend on it, so some caution would still be > needed). We might have to start calling that part DLR+ or something to > avoid confusion. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Fri Oct 22 18:29:40 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 22 Oct 2010 16:29:40 +0000 Subject: [IronPython] HostCodeHeap leakage? In-Reply-To: <781eb71519308c350a2515a76337b7b4@mail.gmail.com> References: <7c4757f1f0618419ef8a198e7ed1bfb1@mail.gmail.com> <781eb71519308c350a2515a76337b7b4@mail.gmail.com> Message-ID: On 1 following the instructions below this should apply pretty cleanly to the 2.6.2 sources. If there's any issues let me know. On 2 - This starts to get hard to explain, but what this change is doing is altering the restrictions we apply when producing a rule. Before the change we say "this rule applies to just this individual function object." But because function objects can be created again and again and again this means we'll end up doing the code gen to call a function each time we see a new function object. The change makes us instead say "this rule applies to every function object that shares the same code object." Because arguments are defined by their code objects this is a much better level of sharing. But because defaults are provided by function objects instead of code objects whenever we have a call which default w/ default we still need to restrict it to the individual function (there's probably something smarter we can do here as well - for example restricting to the function and the # of defaults). With that setup in mind there's a couple of other places where we depend upon the code object but restrict to the individual function. Those can also be switched over which might help you as well. Those are all of the assignments to _extractedKeyword except for the one in ExtractDefaultValue because that's the one that depends upon the function object. So if you just search through MetaPythonFunction you can change all of the assignments to _extractedKeyword except for the one in ExtractDefaultValue to assignments to _needCodeTest (in the current sources I renamed _extractedKeyword to _extractedDefault so it makes more sense). From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Friday, October 22, 2010 12:14 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Thanks! We will try it. Just a couple of questions: 1. I guess this will not be in the current release of ipy 2.6.2, right? 2. I didn't understand you last comment "You can also replace the other places where we sign to _extractedKeyword w/ assignmnets to _needCodeTest except for in ExtractDefaultValue. That one still depends upon the actual function object who's defaults are mutable. So as long as the issue here doesn't involve defaults (as in we don't consume any default values) then this will fix the problem." can you please elaborate on that? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Thursday, October 21, 2010 11:09 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Here's a fix for this. There will still be some cases w/ function calls consuming memory but it solves the keyword arg problem and it only changes one file. I've also checked the fix in and pushed it to CodePlex so you can just get MetaPythonFunction.cs from there (change #78756 - http://ironpython.codeplex.com/SourceControl/changeset/changes/78756 ). But here's the description if you want to patch it by hand. This won't be in the 2.7B1 release but is checked in now for subsequent releases. So in MetaPythonFunction.cs there's a class FunctionBinderHelper. It needs a new member variable: private bool _needCodeTest; // true if we need to test the code object (This is line 301 in the current 2.7 sources). There's a function GetComplexRestriction, it needs this code added: if (_extractedKeyword) { return BindingRestrictions.GetInstanceRestriction(_func.Expression, _func.Value); } else if (_needCodeTest) { return GetSimpleRestriction().Merge( BindingRestrictions.GetInstanceRestriction( Expression.Property( GetFunctionParam(), "__code__" ), _func.Value.__code__ ) ); } return GetSimpleRestriction(); } And finally in GetArgumentsForRule we need to set _needCodeTest instead of _extractedKeyword for ArgumentType.Named: case ArgumentType.Named: _extractedKeyword = true; _needCodeTest = true; You can also replace the other places where we sign to _extractedKeyword w/ assignmnets to _needCodeTest except for in ExtractDefaultValue. That one still depends upon the actual function object who's defaults are mutable. So as long as the issue here doesn't involve defaults (as in we don't consume any default values) then this will fix the problem. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ronnie Maor Sent: Monday, October 18, 2010 1:34 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? thanks - wanted to make sure you didn't miss it. we'll try to reduce the number of places where we're exposed to it in the meantime. On Mon, Oct 18, 2010 at 10:28 PM, Dino Viehland > wrote: Yep, I've seen it... still thinking about what should be done here. I certainly understand the problem and what will cause it but the fix is probably non trivial (but also probably well contained to one or two classes). It may take me a day or two to respond w/ something substantial. We probably need to change our kw-calling to use a pre-compiled rule. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Ronnie Maor Sent: Monday, October 18, 2010 1:25 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Can someone from IPy team ack that you saw this? The issue is causing us a lot of trouble, so we'd really appreciate it if you could tell us how to fix - we've already built from source to fix a previous leak, so no problem building with another patch. BTW, the default value in the function definition is not needed, it's calling with named arguments that causes the issue. so this is a slightly simpler repro: def test_method(): for i in xrange(1000): def func(param): pass func(param = None) thanks! Ronnie On Mon, Oct 18, 2010 at 11:35 AM, Idan Zaltzberg > wrote: FYI, there is a bit simpler reproduction: def test_method(): for i in xrange(1000): def func(param = None): pass func(param = None) test_method() So, actually any use of keyword params in closure that are redefined causes the problem. From: Idan Zaltzberg [mailto:idan at cloudshare.com] Sent: Monday, October 18, 2010 11:10 AM To: 'Discussion of IronPython' Subject: RE: [IronPython] HostCodeHeap leakage? Ok, I finally succeeded in creating a simple reproduction for this problem. The following code generates a 1000 methods (according to the ".NET CLR JIT" performance counter), on Ipy 2.6.1 .Net 2.0 def test_method(): for i in xrange(1000): def func(*a,**kw): pass func(some_parm = None) test_method() This does not happen if you call f without keyword params (using the *a params is OK). If this is indeed a bug, we would like to know how to fix it in the code locally, if that is possible. Also, I am interested in what Ipy flow creates this methods, since I wasn't able to find the function in the code that does this generation Thanks From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Thursday, October 14, 2010 8:31 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? So from an IronPython/DLR perspective the process should stabilize over time. That could take a while - as various Python functions get used repeatedly we'll switch from interpreting them to compiling them. We'll also potentially produce new call site rules which are compiled. That could account for the increase of 16k->18k dynamic methods. That's a 12% increase and could be a reasonable amount of it remained steady from then on. Likewise the # of function codes seems stable but the _codeCount is rising. That might mean that you're defining new functions (via exec/execfile/compile) and they're getting collected - but we still have their (dead) weak references in the code list. Eventually that list should get cleaned out when we hit context._nextCodeCleanup (which should be greater than context._codeCount). I would expect if you were to walk the entire PythonContext._allCodes linked list that you'd see about half of the lists having a dead weak reference. If your windbg-foo is up for writing the script to do this that'd be great but mine is rusty enough I'd need to look it up in the documentation. As for jitted code - all dynamic methods are collectible and any normal RefEmit code is not collectible (there's an option to make assemblies collectible in .NET 4.0, but we don't use it as we generally don't generate that many types). Oh, we do also generate new types for subclasses but you should see that NewTypeMaker._newTypes has a stable count over time because we share these between types w/ common bases. Closures, callbacks, generators should all be fine. If you do .symfix, then .reload, in Windbg does "dt HostCodeHeap" show you the fields of the HostCodeHeap structure? Does !eeheap -loader give you the address of the HostCodeHeap? It might be useful to know what m_allocationCount is on the heap overtime as if that's relatively stable the heap could be getting fragmented. Looking at the CLR code I'm becoming more and more convinced HostCodeHeap is only used for dynamic methods so if that's growing then I'm thinking we would appear to be leaking dynamic methods. The only way I can think of figuring out where the allocations are coming from is to put a breakpoint on mscorwks!HostCodeHeap::AllocMemory_NoThrow (hopefully this will show up in the public symbols). That may be difficult if you can't attach the debugger to the server but if the issue is blocking the server you could have a breakpoint here which does a stack trace and continues execution so you could inspect the stack trace later. I'd include both "kb" and "!ClrStack" from sos in the stack trace. 2.7 shouldn't really change this - whether .NET 4.0 would or not would depend on if the CLR changed anything here. But I'm not sure - I would assume it wouldn't. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Thursday, October 14, 2010 6:44 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Hi, I've been looking on this for some time, and I'm still don't understand some things. Maybe I should begin by explaining our usage of Ipy (2.6.1 .NET 2.0) a bit better. We have a long running (stateful) application that we cannot simulate or run with a debugger open (so no breakpoints). However, since the application is run on a VM, we can take snapshots of it and then open WinDbg instance and break in the middle of the application. We do this a few hours after the application restarted and again after two days so we can see the difference. This way we saw that Jit Code Heap is increasing by a few hundred MB per day, and the number of HostCodeHeap objects is increasing. We also compared the performance counters for the two snapshots, and saw the Jitted Code Bytes increased from 100MB to 862MB and the number of methods jitted increased from 700K to 6.3M. In the WinDbg we saw that the Jitted Code Heap size increases from 126MB to 424MB. On the other hand, the object types you mentioned stay relatively the same: * DynamicMethods count went from 16K to 18K * FunctionCode count went from 4013 to 4025 * The _codeCount field in the PythonContext went from 4447 to 7800 Here is what we don't understand: 1. Is it normal for the application to keep jitting code and methods forever? Should is stabilize? 2. From the numbers I guess that some of the jitted code IS collected. Which types are collectable and which are not? How can I tell which ones I am using? 3. Are there any specific patterns I should avoid to decrease uncollectable code (or jitting in general). I am using a lot of closures, callbacks and generators. 4. What datatypes that are visible in WinDbg can I use to understand if and why IronPython is generating uncollectable code? 5. Is there a way to trace back the HostCodeHeap objects to my code (or IronPython specific features)? 6. Can I expect an improvement in these issues by moving to Ipy 2.7 and/or .Net 4.0? Thanks From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Wednesday, October 13, 2010 10:01 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? If you build from source you could set some breakpoints in AssemblyGen.cs in the DefineType method. You can also set one in DelegateUtils.cs and DelegateHelpers.cs in DefineDelegateType. I think those are all the places where we are creating uncollectible types. If we're continuously hitting those breakpoints after you believe your app has reached steady state then something is going wrong. This code http://www.koders.com/cpp/fid5CC8EACFCC85496B49B8CF83BD05AB36DE691E90.aspx leads me to believe the HostCodeHeap might also be used for DynamicMethods. If that is the case then the other place to look would be if FunctionCode objects are being re-created repeatedly. That will happen if there's exec/eval/compile calls which are happening and if those objects are being kept alive then we could be growing the heap over time. There's also some complicated code which deals with keeping a list of all code that is alive. We do cleanup this list, and the list is a list of weak references so it shouldn't actually keep the code alive, but you could put some breakpoints at FunctionCode.RegisterFunctionCode and FunctionCode.CodeCleanup to see if that list is growing boundlessly (which it would be if something was keeping code objects alive after an exec/eval/compile). Another place where code generation could be occurring would be w/ regexes. If you are dynamically generating reg-exes, or executing a huge different variety of them over time, and they're compiled, then the compiled regexes could be staying in memory. There is a regex cache and you can clear it by calling re.purge(). But it should only cache up to 100 regexes. A final possible thing to investigate might be what happens if you throw away the entire ScriptEngine instance. Here you could try re-cycling the ScriptEngine say every 6 hours and see if the problem goes away. If that fixes the problem then it's likely that it is one of the things I mentioned (or some other cache that's per-runtime). At least that would start to narrow it down vs. some potentially global state (like the subtype list which is shared across ScriptEngines). That's a bunch of different things to look at - hopefully it'll give some insight into what's going on and help track down the issue. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Wednesday, October 13, 2010 12:12 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? I tried what you suggested (changed setup.DebugMode = false;) But still I get the same behavior: The "Jit Code Heap" increases from about 17MB to 230MB in 2 days. Is there a way to verify from the IronPython code that DebugMode is off? Is there anything else I can do (other startup settings?) to decrease/understand the increase in HostCodeHeap objects? Thanks. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 6:59 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Yep, DebugMode is the same as -X:Debug. In general I'd suggest making this configurable somehow and only turn it on if you're actually debugging. It's unfortunate that we can't offer both debugging & collectability but right now that's simply a limitation of the CLR and/or our lack of a separate VS debug engine which can debug Python code. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 9:09 AM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? Im running using the engine from a hosting app. We have these lines in the startup: ScriptRuntimeSetup setup = new ScriptRuntimeSetup(); setup.DebugMode = true; ScriptRuntime runtime = Python.CreateRuntime(setup.Options); engine = runtime.GetEngine("py"); Is this is the same like -X:Debug? You reckon this could be the cause? From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Tuesday, October 05, 2010 5:53 PM To: Discussion of IronPython Subject: Re: [IronPython] HostCodeHeap leakage? My guess is that's code in the JIT heap that's building up but I'm not 100% certain. How is your code being executed? Do you have the debug option (-D or -X:Debug) enabled? To support debug mode we need to produce uncollectible code which could be building up. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Idan Zaltzberg Sent: Tuesday, October 05, 2010 2:26 AM To: Discussion of IronPython Subject: [IronPython] HostCodeHeap leakage? I am trying to find a memory/"performance" leak in an Ipy application. Using WINDBG (!eeheap -loader), we noticed the that the LoaderHeap is getting bigger (150MB increase per day). From the !eeheap output it seems that the increase is due to HostCodeHeap (objects?). As I understand these objects might be created by Ipy infra, is that right? Is there anyway I can get more info on their content, or prevent them from growing? Thanks _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at voidspace.org.uk Fri Oct 22 18:37:49 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Fri, 22 Oct 2010 17:37:49 +0100 Subject: [IronPython] The Future of IronPython In-Reply-To: References: Message-ID: <4CC1BDDD.3030404@voidspace.org.uk> On 22/10/2010 17:15, Jeff Hardy wrote: > On Fri, Oct 22, 2010 at 9:56 AM, Vernon Cole wrote: >> Jeff: >> Can your version of zlib be slipped in to v2.7 before final now? (please >> please pretty please) > Well, the first issue is that it depends on another open source > component (Component Ace's Zlib.Net - > http://componentace.com/zlib_.NET.htm) with a different licence, and I > don't know what the legal implications are of that. > > The second issue is that we (me and Michael) want to get a stable 2.7 > out as soon as possible, and were not planning on adding community > modules until the 2.7.1 release. > > However, zlib support is also the single highest voted issue on > codeplex, and the module passes all CPython tests for zlib and gzip, > so we might make an exception. Michael? If the licenses *are* compatible, so we can ship a fully working zlib module, then fine (and great). If it takes a while to work it out I would still rather we get 2.7 out before we expend a great deal of effort on these issues. All the best, Michael Foord > - Jeff -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From dinov at microsoft.com Fri Oct 22 18:49:13 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 22 Oct 2010 16:49:13 +0000 Subject: [IronPython] Announcing IronPython 2.7 Beta 1 In-Reply-To: References: Message-ID: Adal wrote: > 1. Changing the default installation folder doesn't work: IronPython will be > installed in two places: "C:\Program Files (x86)\IronPython 2.7" and the > installation folder that you specified. > > 2. If you have already have CPython installed, IronPython will overwrite the > icon for .py files, but not for .pyc and .pyw files. > Please don't replace the CPython icons or at least make this an installer > option. Oh, wow, do I get to be the first person to say patches are welcome? :) There were a bunch of changes to the MSI builder to get shared merged modules between Python and Ruby, to support merge modules so apps could use them for IronPython re-disting, and for getting the MSI build infrastructure pushed out publicly instead of just being internal only stuff. These are probably pretty easy to track down as the MSI is nice and isolated at $/IronPython/IronPython_Main/Msi/Python. For the first one it's probably just looking at how the different files are setup. For the 2nd we already detect if someone else has registered the Python file type and don't replace the default (double click) action. So the icon registration probably just needs to move from the Comp_PyFileRegistration to Comp_PyFileExistance component if this is the desired behavior. Anyway I'd suggest opening an issue on the issue tracker for these. From miguel at novell.com Fri Oct 22 18:57:51 2010 From: miguel at novell.com (Miguel de Icaza) Date: Fri, 22 Oct 2010 12:57:51 -0400 Subject: [IronPython] Thank you all In-Reply-To: References: <7993FB06FBE14EB38343F91BAAC341FC@Gamry.com> Message-ID: Hello, Miguel tweeted about turning it into a full fledged scripting language > called S# at one point. > This comes in a number of steps. First, we want to have our C# compiler support the same hosting APIs that the current Iron* languages support. Second, we are going to have an option to have our C# compiler recognize a different language that we are calling "S#". It is not too far from C#, it is basically C# with the features that we wanted to get into the language for many years, and we feel that the time is right to get these out to the public. We have an internal draft document, and once it is more baked we will circulate it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Fri Oct 22 19:01:19 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Fri, 22 Oct 2010 11:01:19 -0600 Subject: [IronPython] The Future of IronPython In-Reply-To: <4CC1BDDD.3030404@voidspace.org.uk> References: <4CC1BDDD.3030404@voidspace.org.uk> Message-ID: On Fri, Oct 22, 2010 at 10:37 AM, Michael Foord wrote: > If the licenses *are* compatible, so we can ship a fully working zlib > module, then fine (and great). It looks like the normal 3-clause BSD licence: http://componentace.com/data/ZLIB_.NET/license.txt, so it should be OK. - Jeff From michael at voidspace.org.uk Fri Oct 22 19:21:54 2010 From: michael at voidspace.org.uk (Michael Foord) Date: Fri, 22 Oct 2010 18:21:54 +0100 Subject: [IronPython] The Future of IronPython In-Reply-To: References: <4CC1BDDD.3030404@voidspace.org.uk> Message-ID: <4CC1C832.8090602@voidspace.org.uk> On 22/10/2010 18:01, Jeff Hardy wrote: > On Fri, Oct 22, 2010 at 10:37 AM, Michael Foord > wrote: >> If the licenses *are* compatible, so we can ship a fully working zlib >> module, then fine (and great). > It looks like the normal 3-clause BSD licence: > http://componentace.com/data/ZLIB_.NET/license.txt, so it should be > OK. > In that case there should be no problem including it, so long as we also include a copy of the license. Michael > - Jeff -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From cenovsky at bakalari.cz Sun Oct 24 12:45:05 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Sun, 24 Oct 2010 12:45:05 +0200 Subject: [IronPython] The Future of IronPython In-Reply-To: References: Message-ID: <4CC40E31.6040404@bakalari.cz> Thank you Jeff you are taking the responsibility for the final 2.7 release. *I'd like to help. What is the best place to start?* -- -- Luk?s( On 22.10.2010 1:09, Jeff Hardy wrote: > As Dino recently posted, this is the last release of IronPython by > Microsoft. As of today, myself, Michael Foord, Jimmy Schementi, and > Miguel de Icaza are the coordinators for the IronPython project (Jimmy > and Miguel will also handle IronRuby). More information can be found > from Jason Zander: > http://blogs.msdn.com/b/jasonz/archive/2010/10/21/new-components-and-contributors-for-ironpython-and-ironruby.aspx. > Do note that Zander doesn't actually come out and *say* that Microsoft > is no longer funding IronPython/IronRuby, but that's what has > happened. Any future participation from MS employees will be on an > unofficial, spare-time basis. > > All that said, I hope this isn't the end of IronPython. There's a good > community here, and I think we can do just as good a job as Microsoft > did. You can now contribute patches to IronPython, but that means you > will have to if you want the it to flourish. A good Open Source > community requires a core of regular contributors, but also requires a > mass of occasional ones that are just scratching their particular > itches. > > > From here, my goal is to get IronPython 2.7 final out the door in a > timely manner. Let's leave discussions about things like source > control or external libraries aside for a bit and get a solid release > -- then we can start those discussions. I fear that starting a major > overhaul will kill any momentum we have and cause the 2.7 release to > be unacceptably delayed. > > I wouldn't have signed up to do this if I thought IronPython didn't > have a future. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfromaniello at gmail.com Mon Oct 25 16:38:07 2010 From: jfromaniello at gmail.com (=?ISO-8859-1?Q?Jos=E9_F=2E_Romaniello?=) Date: Mon, 25 Oct 2010 11:38:07 -0300 Subject: [IronPython] trying to build IronPythonTools locally Message-ID: Hi all, I am trying to build IronPython Tools on my machine, but i get this exception: ------ Build started: Project: IronPythonTools, Configuration: Debug Any CPU ------ IronPythonTools -> IronPythonTools -> IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.dll Creating intermediate PkgDef file. Creating VSIX Container... IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.vsix C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\VSSDK\Microsoft.VsSDK.targets(420,5): error : Exception has been thrown by the target of an invocation. ========== Build: 11 succeeded or up-to-date, 1 failed, 0 skipped ========== I tried with a reset of the Experimental Hive but I have still the same problem. Thank you in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From curt at hagenlocher.org Mon Oct 25 17:14:43 2010 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Mon, 25 Oct 2010 08:14:43 -0700 Subject: [IronPython] trying to build IronPythonTools locally In-Reply-To: References: Message-ID: This is the "uninstall previous version" task. Is it possible that you've previously installed the extension when running as Administrator and that your current user id doesn't have the rights to delete the files? This might be true if, for instance, you previously installed from the MSI. On Mon, Oct 25, 2010 at 7:38 AM, Jos? F. Romaniello wrote: > Hi all, I am trying to build IronPython Tools on my machine, but i get this > exception: > > ------ Build started: Project: IronPythonTools, Configuration: Debug Any > CPU ------ > IronPythonTools -> > IronPythonTools -> > IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.dll > Creating intermediate PkgDef file. > Creating VSIX Container... > IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.vsix > C:\Program Files > (x86)\MSBuild\Microsoft\VisualStudio\v10.0\VSSDK\Microsoft.VsSDK.targets(420,5): > error : Exception has been thrown by the target of an invocation. > ========== Build: 11 succeeded or up-to-date, 1 failed, 0 skipped > ========== > > I tried with a reset of the Experimental Hive but I have still the same > problem. > > Thank you in advance! > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfromaniello at gmail.com Mon Oct 25 18:01:47 2010 From: jfromaniello at gmail.com (=?ISO-8859-1?Q?Jos=E9_F=2E_Romaniello?=) Date: Mon, 25 Oct 2010 13:01:47 -0300 Subject: [IronPython] trying to build IronPythonTools locally In-Reply-To: References: Message-ID: Oh thanks, you are right! I can build now. Another question, is there any unit tests for IronPythonTools? I am using the IronPythonTools.sln in the trunk. I just want to dig in to the code and see if i can contribute with few bugs I found in the highlighting and code completion. I've some experience doing this same thing for other extension ( http://hqladdin.codeplex.com) thank you 2010/10/25 Curt Hagenlocher > This is the "uninstall previous version" task. Is it possible that you've > previously installed the extension when running as Administrator and that > your current user id doesn't have the rights to delete the files? This might > be true if, for instance, you previously installed from the MSI. > > On Mon, Oct 25, 2010 at 7:38 AM, Jos? F. Romaniello < > jfromaniello at gmail.com> wrote: > >> Hi all, I am trying to build IronPython Tools on my machine, but i get >> this exception: >> >> ------ Build started: Project: IronPythonTools, Configuration: Debug Any >> CPU ------ >> IronPythonTools -> >> IronPythonTools -> >> IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.dll >> Creating intermediate PkgDef file. >> Creating VSIX Container... >> IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.vsix >> C:\Program Files >> (x86)\MSBuild\Microsoft\VisualStudio\v10.0\VSSDK\Microsoft.VsSDK.targets(420,5): >> error : Exception has been thrown by the target of an invocation. >> ========== Build: 11 succeeded or up-to-date, 1 failed, 0 skipped >> ========== >> >> I tried with a reset of the Experimental Hive but I have still the same >> problem. >> >> Thank you in advance! >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Mon Oct 25 19:32:01 2010 From: dinov at microsoft.com (Dino Viehland) Date: Mon, 25 Oct 2010 17:32:01 +0000 Subject: [IronPython] trying to build IronPythonTools locally In-Reply-To: References: Message-ID: The tests are rather lacking but some do exist. There's 2 test projects: AnalysisTest and UnitTests. AnalysisTest contains tests which are directly targeting the analysis engine. UnitTests includes some mocks for various VS interfaces and handles things like code completion. I also have a set of tests that I would run through by hand. I've pasted those below in case anyone wants to automate them. REPL Window: View->IronPython Command Line "def f(): pass" + 2 ENTERS f( should bring signature help up "x = 42" "x." should bring up intellisense completion "x " should not being up any completions. Up-up should scroll back to def f(): pass Enter in a middle of a line should insert new line Escape should clear both lines "x=42" left left ctrl-enter should commit assignment while True: pass / Right Click -> Break Execution (or Ctrl-Break) should break execution Closing window + re-opening should still have contents Ctrl-Enter on previous input should paste input to end of buffer (doing it again should paste again - appending onto previous input) Type some text, hit Ctrl-Enter, should execute current line Type a function definition, go to next line, type pass, navigate left, hit ctrl-enter, should immediately execute func def. Context Menu Cut/Copy/Paste works Context Menu Reset repl works x = 42; x.car[enter] - should type "car" not complete to "conjugate" x = 42; x.conjugate[enter] - should respect enter completes option, and should respect enter at end of word completes option. When it does execute the text the output should be on the next line. Define function "def f():\r\n print 'hi'", scroll back up to history, add print "hello" to 2nd line, enter, scroll back through both function definitions Enter "while True: pass", then hit up/down arrow, should move the caret in the edit buffer Type "raise Exception()", hit enter, raise Exception() should have appropriate syntax color highlighting. Type: "def f():[ENTER]print 'hi'[ENTER][ENTER]", hit up arrow, function should be brought back, cursor should be right after 'hi'. Press ENTER again, type "print 'hello'"[ENTER][ENTER] and function should be committed. Type "import sys", commit, then type "sys", and hit backspace a couple of times. No exceptions should occur Tools->Options->Text Editor->IronPython->Advanced->Options. Add "Frames=True", go to repl, restart repl, import sys, make sure sys._getframe is present Go back to options, add ";RecursionLimit=50", restart repl, import sys, sys.getrecursionlimit() should return 50, "def f(): f()", call f(), should get a max recursion overflowed exception stack trace Debug->Execute * in REPL In Python file should execute text in REPL window Variables assigned in file should be in scope Send to Python Repl from edit buffer should send text to REPL window Should open REPL if not already open Should not send enter unless selected in text + View->IronPython Command Line tests import System / System.Environment.Exit(0); Kill remote process Start project, make sure CWD (System.Environment.CurrentDirectory) is project CWD Close project, open .py file, start .py file, make sure CWD is .py files directory Navigation Bar: Verify displays and greys out 1st item when at top of file not in class/def Verify lhs properly transitions into class and func and back out (going back to dimmed & 1st item selected) Verify function in class properly transitions when moving in and out (going back to dimmed & 1st item selected) "class " and "def " - functions and classes w null names shouldn't break the navigation bar (shouldn't display in navigation bar at all?) Completion: Verify various left hand sides which need to be successfully parsed: X[0]. X(0). X(0, 0). X(0, *args). X(0, **args). x.foo. x.foo[0]. x.foo(0). x.foo(y()). Verify from import completions Verify import completions Verify Ctrl-J/Ctrl-Space reflects all imported members "2." Shouldn't bring up completions (typing a float) Comma shouldn't bring up signature help in tuples, lists, and dictionaries from modulethatdoesnot exist import import modulethatdoesnotexist Signature Help: ( brings up signature help Comma navigiates to next parameter Comma when overloads present and exceeds # of args navigates to next overload Too many args properly goes to params array (both when params array has more and less arguments vs another overload) Type: def(f, x): pass, then do f(, backspace over (, signature help should be dismissed. Type (f( backspace over (, help should be dismissed, new help should be created for previous call. Object Browser: Object browser properly reports class and function members Signature help is properly displayed Double clicking goes to definition Outlining: Classes and functions are properly outlined Collapsing functions/classes works Collapsing functions/classes with weird ending white space collapses in a sane manner Templates: Run through each template, create a new project, run it, run it in debug mode. Create each file template in applicable project Brace matching: Test at beginning and end, should high light both braces: (), [], {} ( ), [ ], { } ([), [(], {(} ( () ), [ [] ], { {} } Test ( and ) at beginning and end of file, beginning and end of line Find All References: Ctrl-K-Ctrl-R at start of editor buffer Ctrl-K-Ctrl-R at end of editor buffer Ctrl-K Ctrl-R at start of identifier Ctrl-K Ctrl-R at end of identifier Editor: Select a region and press enter should clear the region Select a region and press backspace - should clear the region Enter some text on a line, hit enter, enter 12 characters of white space on the line, move left 8 characters, hit backspace, should delete 4 spaces to the left. Enter "abcd" on a line followed by a bunch of spaces. Then move back to the "d", press backspace, should delete the d. WPF Designer: Drag all controls onto the designer surface successfully (note Image fails if the project is on a network share due to a WPF bug - it fails in C# as well) Open .xaml file, close .py file, double click object to add event handler, should open .py file and add event handler Add ? control, click on Name over in properties, change name. Go to code and dot through name on Window object. Name should be available in intellisense. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Jos? F. Romaniello Sent: Monday, October 25, 2010 9:02 AM To: Discussion of IronPython Subject: Re: [IronPython] trying to build IronPythonTools locally Oh thanks, you are right! I can build now. Another question, is there any unit tests for IronPythonTools? I am using the IronPythonTools.sln in the trunk. I just want to dig in to the code and see if i can contribute with few bugs I found in the highlighting and code completion. I've some experience doing this same thing for other extension (http://hqladdin.codeplex.com) thank you 2010/10/25 Curt Hagenlocher > This is the "uninstall previous version" task. Is it possible that you've previously installed the extension when running as Administrator and that your current user id doesn't have the rights to delete the files? This might be true if, for instance, you previously installed from the MSI. On Mon, Oct 25, 2010 at 7:38 AM, Jos? F. Romaniello > wrote: Hi all, I am trying to build IronPython Tools on my machine, but i get this exception: ------ Build started: Project: IronPythonTools, Configuration: Debug Any CPU ------ IronPythonTools -> IronPythonTools -> IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.dll Creating intermediate PkgDef file. Creating VSIX Container... IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.vsix C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\VSSDK\Microsoft.VsSDK.targets(420,5): error : Exception has been thrown by the target of an invocation. ========== Build: 11 succeeded or up-to-date, 1 failed, 0 skipped ========== I tried with a reset of the Experimental Hive but I have still the same problem. Thank you in advance! _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Mon Oct 25 19:36:55 2010 From: dinov at microsoft.com (Dino Viehland) Date: Mon, 25 Oct 2010 17:36:55 +0000 Subject: [IronPython] trying to build IronPythonTools locally In-Reply-To: References: Message-ID: Oh, I should add both of those projects are both standalone EXEs which can be run to run all of the tests and they're also VS unit tests. So if you have some fancy version of VS w/ unit testing then you can just run all the tests from within VS using Test->Run->All Tests in Solution. If you don't have the fancy version you can just them from the command line. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Monday, October 25, 2010 10:32 AM To: Discussion of IronPython Subject: Re: [IronPython] trying to build IronPythonTools locally The tests are rather lacking but some do exist. There's 2 test projects: AnalysisTest and UnitTests. AnalysisTest contains tests which are directly targeting the analysis engine. UnitTests includes some mocks for various VS interfaces and handles things like code completion. I also have a set of tests that I would run through by hand. I've pasted those below in case anyone wants to automate them. REPL Window: View->IronPython Command Line "def f(): pass" + 2 ENTERS f( should bring signature help up "x = 42" "x." should bring up intellisense completion "x " should not being up any completions. Up-up should scroll back to def f(): pass Enter in a middle of a line should insert new line Escape should clear both lines "x=42" left left ctrl-enter should commit assignment while True: pass / Right Click -> Break Execution (or Ctrl-Break) should break execution Closing window + re-opening should still have contents Ctrl-Enter on previous input should paste input to end of buffer (doing it again should paste again - appending onto previous input) Type some text, hit Ctrl-Enter, should execute current line Type a function definition, go to next line, type pass, navigate left, hit ctrl-enter, should immediately execute func def. Context Menu Cut/Copy/Paste works Context Menu Reset repl works x = 42; x.car[enter] - should type "car" not complete to "conjugate" x = 42; x.conjugate[enter] - should respect enter completes option, and should respect enter at end of word completes option. When it does execute the text the output should be on the next line. Define function "def f():\r\n print 'hi'", scroll back up to history, add print "hello" to 2nd line, enter, scroll back through both function definitions Enter "while True: pass", then hit up/down arrow, should move the caret in the edit buffer Type "raise Exception()", hit enter, raise Exception() should have appropriate syntax color highlighting. Type: "def f():[ENTER]print 'hi'[ENTER][ENTER]", hit up arrow, function should be brought back, cursor should be right after 'hi'. Press ENTER again, type "print 'hello'"[ENTER][ENTER] and function should be committed. Type "import sys", commit, then type "sys", and hit backspace a couple of times. No exceptions should occur Tools->Options->Text Editor->IronPython->Advanced->Options. Add "Frames=True", go to repl, restart repl, import sys, make sure sys._getframe is present Go back to options, add ";RecursionLimit=50", restart repl, import sys, sys.getrecursionlimit() should return 50, "def f(): f()", call f(), should get a max recursion overflowed exception stack trace Debug->Execute * in REPL In Python file should execute text in REPL window Variables assigned in file should be in scope Send to Python Repl from edit buffer should send text to REPL window Should open REPL if not already open Should not send enter unless selected in text + View->IronPython Command Line tests import System / System.Environment.Exit(0); Kill remote process Start project, make sure CWD (System.Environment.CurrentDirectory) is project CWD Close project, open .py file, start .py file, make sure CWD is .py files directory Navigation Bar: Verify displays and greys out 1st item when at top of file not in class/def Verify lhs properly transitions into class and func and back out (going back to dimmed & 1st item selected) Verify function in class properly transitions when moving in and out (going back to dimmed & 1st item selected) "class " and "def " - functions and classes w null names shouldn't break the navigation bar (shouldn't display in navigation bar at all?) Completion: Verify various left hand sides which need to be successfully parsed: X[0]. X(0). X(0, 0). X(0, *args). X(0, **args). x.foo. x.foo[0]. x.foo(0). x.foo(y()). Verify from import completions Verify import completions Verify Ctrl-J/Ctrl-Space reflects all imported members "2." Shouldn't bring up completions (typing a float) Comma shouldn't bring up signature help in tuples, lists, and dictionaries from modulethatdoesnot exist import import modulethatdoesnotexist Signature Help: ( brings up signature help Comma navigiates to next parameter Comma when overloads present and exceeds # of args navigates to next overload Too many args properly goes to params array (both when params array has more and less arguments vs another overload) Type: def(f, x): pass, then do f(, backspace over (, signature help should be dismissed. Type (f( backspace over (, help should be dismissed, new help should be created for previous call. Object Browser: Object browser properly reports class and function members Signature help is properly displayed Double clicking goes to definition Outlining: Classes and functions are properly outlined Collapsing functions/classes works Collapsing functions/classes with weird ending white space collapses in a sane manner Templates: Run through each template, create a new project, run it, run it in debug mode. Create each file template in applicable project Brace matching: Test at beginning and end, should high light both braces: (), [], {} ( ), [ ], { } ([), [(], {(} ( () ), [ [] ], { {} } Test ( and ) at beginning and end of file, beginning and end of line Find All References: Ctrl-K-Ctrl-R at start of editor buffer Ctrl-K-Ctrl-R at end of editor buffer Ctrl-K Ctrl-R at start of identifier Ctrl-K Ctrl-R at end of identifier Editor: Select a region and press enter should clear the region Select a region and press backspace - should clear the region Enter some text on a line, hit enter, enter 12 characters of white space on the line, move left 8 characters, hit backspace, should delete 4 spaces to the left. Enter "abcd" on a line followed by a bunch of spaces. Then move back to the "d", press backspace, should delete the d. WPF Designer: Drag all controls onto the designer surface successfully (note Image fails if the project is on a network share due to a WPF bug - it fails in C# as well) Open .xaml file, close .py file, double click object to add event handler, should open .py file and add event handler Add ? control, click on Name over in properties, change name. Go to code and dot through name on Window object. Name should be available in intellisense. From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Jos? F. Romaniello Sent: Monday, October 25, 2010 9:02 AM To: Discussion of IronPython Subject: Re: [IronPython] trying to build IronPythonTools locally Oh thanks, you are right! I can build now. Another question, is there any unit tests for IronPythonTools? I am using the IronPythonTools.sln in the trunk. I just want to dig in to the code and see if i can contribute with few bugs I found in the highlighting and code completion. I've some experience doing this same thing for other extension (http://hqladdin.codeplex.com) thank you 2010/10/25 Curt Hagenlocher > This is the "uninstall previous version" task. Is it possible that you've previously installed the extension when running as Administrator and that your current user id doesn't have the rights to delete the files? This might be true if, for instance, you previously installed from the MSI. On Mon, Oct 25, 2010 at 7:38 AM, Jos? F. Romaniello > wrote: Hi all, I am trying to build IronPython Tools on my machine, but i get this exception: ------ Build started: Project: IronPythonTools, Configuration: Debug Any CPU ------ IronPythonTools -> IronPythonTools -> IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.dll Creating intermediate PkgDef file. Creating VSIX Container... IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.vsix C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\VSSDK\Microsoft.VsSDK.targets(420,5): error : Exception has been thrown by the target of an invocation. ========== Build: 11 succeeded or up-to-date, 1 failed, 0 skipped ========== I tried with a reset of the Experimental Hive but I have still the same problem. Thank you in advance! _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cenovsky at bakalari.cz Mon Oct 25 19:48:30 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Mon, 25 Oct 2010 19:48:30 +0200 Subject: [IronPython] Pyc.py platform differences Message-ID: <4CC5C2EE.7030000@bakalari.cz> An HTML attachment was scrubbed... URL: From jfromaniello at gmail.com Mon Oct 25 19:49:26 2010 From: jfromaniello at gmail.com (=?ISO-8859-1?Q?Jos=E9_F=2E_Romaniello?=) Date: Mon, 25 Oct 2010 14:49:26 -0300 Subject: [IronPython] trying to build IronPythonTools locally In-Reply-To: References: Message-ID: GREAT, i think i got it! Thank you very much. 2010/10/25 Dino Viehland > Oh, I should add both of those projects are both standalone EXEs which > can be run to run all of the tests and they?re also VS unit tests. So if > you have some fancy version of VS w/ unit testing then you can just run all > the tests from within VS using Test->Run->All Tests in Solution. If you > don?t have the fancy version you can just them from the command line. > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Dino Viehland > *Sent:* Monday, October 25, 2010 10:32 AM > > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] trying to build IronPythonTools locally > > > > The tests are rather lacking but some do exist. There?s 2 test projects: > AnalysisTest and UnitTests. AnalysisTest contains tests which are directly > targeting the analysis engine. UnitTests includes some mocks for various VS > interfaces and handles things like code completion. I also have a set of > tests that I would run through by hand. I?ve pasted those below in case > anyone wants to automate them. > > > > REPL Window: > > View->IronPython Command Line > > ?def f(): pass? + 2 ENTERS > > f( should bring signature help up > > ?x = 42? > > ?x.? should bring up intellisense completion > > ?x ? should not being up any completions. > > Up-up should scroll back to def f(): pass > > Enter in a middle of a line should insert new line > > Escape should clear both lines > > ?x=42? left left ctrl-enter should commit > assignment > > while True: pass / Right Click -> Break Execution > (or Ctrl-Break) should break execution > > Closing window + re-opening should still have > contents > > Ctrl-Enter on previous input should paste input to > end of buffer (doing it again should paste again ? appending onto previous > input) > > Type some text, hit Ctrl-Enter, should execute > current line > > Type a function definition, go to next line, type > pass, navigate left, hit ctrl-enter, should immediately execute func def. > > Context Menu Cut/Copy/Paste works > > Context Menu Reset repl works > > x = 42; x.car[enter] ? should type ?car? not > complete to ?conjugate? > > x = 42; x.conjugate[enter] ? should respect enter > completes option, and should respect enter at end of word completes option. > When it does execute the text the output should be on the next line. > > Define function ?def f():\r\n print ?hi??, scroll back up to history, > add print ?hello? to 2nd line, enter, scroll back through both function > definitions > > Enter ?while True: pass?, then hit up/down arrow, > should move the caret in the edit buffer > > Type ?raise Exception()?, hit enter, raise > Exception() should have appropriate syntax color highlighting. > > Type: ?def f():[ENTER]print ?hi?[ENTER][ENTER]?, > hit up arrow, function should be brought back, cursor should be right after > ?hi?. Press ENTER again, type ?print ?hello??[ENTER][ENTER] and function > should be committed. > > Type ?import sys?, commit, then type ?sys?, and hit > backspace a couple of times. No exceptions should occur > > Tools->Options->Text > Editor->IronPython->Advanced->Options. > > Add ?Frames=True?, go to repl, restart > repl, import sys, make sure sys._getframe is present > > Go back to options, add > ?;RecursionLimit=50?, restart repl, import sys, sys.getrecursionlimit() > should return 50, ?def f(): f()?, call f(), should get a max recursion > overflowed exception stack trace > > Debug->Execute * in REPL > > In Python file should execute text in REPL window > > Variables assigned in file should be in scope > > Send to Python Repl from edit buffer should send > text to REPL window > > Should open REPL if not already open > > Should not send enter unless selected > in text > > + View->IronPython Command Line tests > > import System / System.Environment.Exit(0); > > Kill remote process > > Start project, make sure CWD > (System.Environment.CurrentDirectory) is project CWD > > Close project, open .py file, start .py file, make sure CWD is .py files > directory > > > > Navigation Bar: > > Verify displays and greys out 1st item when at top of file not > in class/def > > Verify lhs properly transitions into class and func and back > out (going back to dimmed & 1st item selected) > > Verify function in class properly transitions when moving in > and out (going back to dimmed & 1st item selected) > > ?class ? and ?def ? ? functions and classes w null names > shouldn?t break the navigation bar (shouldn?t display in navigation bar at > all?) > > > > > > Completion: > > Verify various left hand sides which need to be successfully > parsed: > > X[0]. > > X(0). > > X(0, 0). > > X(0, *args). > > X(0, **args). > > x.foo. > > x.foo[0]. > > x.foo(0). > > x.foo(y()). > > Verify from import completions > > Verify import completions > > Verify Ctrl-J/Ctrl-Space reflects all imported members > > ?2.? Shouldn?t bring up completions (typing a float) > > Comma shouldn?t bring up signature help in tuples, lists, and > dictionaries > > from modulethatdoesnot exist import > > import modulethatdoesnotexist > > > > Signature Help: > > ( brings up signature help > > Comma navigiates to next parameter > > Comma when overloads present and exceeds # of args navigates to > next overload > > Too many args properly goes to params array (both when params > array has more and less arguments vs another overload) > > Type: def(f, x): pass, then do f(, backspace over (, signature help should > be dismissed. Type (f( backspace over (, help should be dismissed, new help > should be created for previous call. > > > > Object Browser: > > Object browser properly reports class and function members > > Signature help is properly displayed > > Double clicking goes to definition > > > > Outlining: > > Classes and functions are properly outlined > > Collapsing functions/classes works > > Collapsing functions/classes with weird ending white space > collapses in a sane manner > > > > Templates: > > Run through each template, create a new project, run it, run it > in debug mode. > > Create each file template in applicable project > > > > Brace matching: > > Test at beginning and end, should high light both braces: > > (), [], {} > > ( ), [ ], { } > > ([), [(], {(} > > ( () ), [ [] ], { {} } > > Test ( and ) at beginning and end of file, beginning and end of > line > > > > Find All References: > > Ctrl-K-Ctrl-R at start of editor buffer > > Ctrl-K-Ctrl-R at end of editor buffer > > Ctrl-K Ctrl-R at start of identifier > > Ctrl-K Ctrl-R at end of identifier > > > > > > Editor: > > Select a region and press enter should clear the region > > Select a region and press backspace ? should clear the region > > Enter some text on a line, hit enter, enter 12 characters of > white space on the line, move left 8 characters, hit backspace, should > delete 4 spaces to the left. > > Enter ?abcd? on a line followed by a bunch of spaces. Then > move back to the ?d?, press backspace, should delete the d. > > > > WPF Designer: > Drag all controls onto the designer surface successfully (note > Image fails if the project is on a network share due to a WPF bug ? it fails > in C# as well) > > Open .xaml file, close .py file, double click object to add > event handler, should open .py file and add event handler > > Add ? control, click on Name over in properties, change name. > Go to code and dot through name on Window object. Name should be available > in intellisense. > > > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Jos? F. Romaniello > *Sent:* Monday, October 25, 2010 9:02 AM > *To:* Discussion of IronPython > *Subject:* Re: [IronPython] trying to build IronPythonTools locally > > > > Oh thanks, you are right! > > I can build now. > > Another question, is there any unit tests for IronPythonTools? I am using > the IronPythonTools.sln in the trunk. > > > > I just want to dig in to the code and see if i can contribute with few bugs > I found in the highlighting and code completion. > > > > I've some experience doing this same thing for other extension ( > http://hqladdin.codeplex.com) > > > > thank you > > > > 2010/10/25 Curt Hagenlocher > > This is the "uninstall previous version" task. Is it possible that you've > previously installed the extension when running as Administrator and that > your current user id doesn't have the rights to delete the files? This might > be true if, for instance, you previously installed from the MSI. > > On Mon, Oct 25, 2010 at 7:38 AM, Jos? F. Romaniello < > jfromaniello at gmail.com> wrote: > > Hi all, I am trying to build IronPython Tools on my machine, but i get > this exception: > > > > ------ Build started: Project: IronPythonTools, Configuration: Debug Any > CPU ------ > > IronPythonTools -> > > IronPythonTools -> > > IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.dll > > Creating intermediate PkgDef file. > > Creating VSIX Container... > > IronPythonTools -> D:\pos\IronPython\bin\Debug\IronPythonTools.vsix > > C:\Program Files > (x86)\MSBuild\Microsoft\VisualStudio\v10.0\VSSDK\Microsoft.VsSDK.targets(420,5): > error : Exception has been thrown by the target of an invocation. > > ========== Build: 11 succeeded or up-to-date, 1 failed, 0 skipped > ========== > > > > I tried with a reset of the Experimental Hive but I have still the same > problem. > > > > Thank you in advance! > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pablodalma93 at hotmail.com Tue Oct 26 14:20:52 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Tue, 26 Oct 2010 09:20:52 -0300 Subject: [IronPython] Thank you all In-Reply-To: References: , <7993FB06FBE14EB38343F91BAAC341FC@Gamry.com>, , , Message-ID: isnt there already a language for .NET called S# or it's the same language you're talking about? Greetings Date: Fri, 22 Oct 2010 12:57:51 -0400 From: miguel at novell.com To: users at lists.ironpython.com CC: miguel.novell at gmail.com Subject: Re: [IronPython] Thank you all Hello, Miguel tweeted about turning it into a full fledged scripting language called S# at one point. This comes in a number of steps. First, we want to have our C# compiler support the same hosting APIs that the current Iron* languages support. Second, we are going to have an option to have our C# compiler recognize a different language that we are calling "S#". It is not too far from C#, it is basically C# with the features that we wanted to get into the language for many years, and we feel that the time is right to get these out to the public. We have an internal draft document, and once it is more baked we will circulate it. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce.bromberek at gmail.com Tue Oct 26 23:44:04 2010 From: bruce.bromberek at gmail.com (Bruce Bromberek) Date: Tue, 26 Oct 2010 16:44:04 -0500 Subject: [IronPython] DynamicWebServicesHelper Sample Script Message-ID: I've used this sample before on a simple webservice that provides the WSDL file via a URL. I now need to use this with a more complex webservice where the WSDL and XSD files are distributed and stored locally. SOAPUI has no problem loading the local WSDL file and constructing requests. WSDL.exe will work its magic if I specify the XSD files along with the WSDL file on the commandline. DynamicWebServiceHelpers errors out on just the WSDL and chokes on more than one parameter. Is there a simple solution I'm missing? Bruce From ted at tedneward.com Thu Oct 28 03:16:01 2010 From: ted at tedneward.com (Ted Neward) Date: Wed, 27 Oct 2010 18:16:01 -0700 Subject: [IronPython] Thank you all In-Reply-To: References: , <7993FB06FBE14EB38343F91BAAC341FC@Gamry.com>, , , Message-ID: <003a01cb763d$b370fc40$1a52f4c0$@tedneward.com> A long long time ago there was some bits for a Smalltalk port to .NET called S#. I have no idea if that's still alive or not. I could never find a good working download link for it. Ted Neward Java, .NET, XML Services Consulting, Teaching, Speaking, Writing http://www.tedneward.com From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Pablo Dalmazzo Sent: Tuesday, October 26, 2010 5:21 AM To: IronPython Mailing list Subject: Re: [IronPython] Thank you all isnt there already a language for .NET called S# or it's the same language you're talking about? Greetings _____ Date: Fri, 22 Oct 2010 12:57:51 -0400 From: miguel at novell.com To: users at lists.ironpython.com CC: miguel.novell at gmail.com Subject: Re: [IronPython] Thank you all Hello, Miguel tweeted about turning it into a full fledged scripting language called S# at one point. This comes in a number of steps. First, we want to have our C# compiler support the same hosting APIs that the current Iron* languages support. Second, we are going to have an option to have our C# compiler recognize a different language that we are calling "S#". It is not too far from C#, it is basically C# with the features that we wanted to get into the language for many years, and we feel that the time is right to get these out to the public. We have an internal draft document, and once it is more baked we will circulate it. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From noreply at badoo.com Thu Oct 28 07:54:31 2010 From: noreply at badoo.com (Badoo) Date: Thu, 28 Oct 2010 05:54:31 +0000 Subject: [IronPython] =?utf-8?q?=C2=A1Luis_Cota_te_ha_dejado_un_mensaje_en?= =?utf-8?q?_Badoo!?= Message-ID: ?Tienes un nuevo mensaje en Badoo! Luis Cota te dej? un mensaje. Haz click en este enlace para verlo: http://us1.badoo.com/lcota/in/8xdPsrQeS6k/?lang_id=7 M?s gente que tambi?n te est? esperando: Pablo (Rafaela, Argentina) Uge (Rafaela, Argentina) Franco (Rafaela, Argentina) http://us1.badoo.com/lcota/in/8xdPsrQeS6k/?lang_id=7 Si al hacer click sobre el enlace, no funciona, copia y pega la direcci?n en tu barra del navegador. Este email es parte del procedimiento para que leas los mensajes de Luis Cota. Si has recibido este email por equivocaci?n, por favor, ign?ralo. Tras un corto periodo de tiempo el mensaje sera eliminado del sistema. ?Divi?rtete! El Equipo de Badoo Has recibido este email porque un usuario de Badoo te ha dejado un mensaje en Badoo. Este mensaje es autom?tico. Las respuestas a este mensaje no estan controladas y no ser?n contestadas. Si no quieres recibir m?s mensajes de Badoo, h?znoslo saber: http://us1.badoo.com/impersonation.phtml?lang_id=7&mail_code=63&email=users%40lists.ironpython.com&secret=&invite_id=435186&user_id=1093354112 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Thu Oct 28 07:57:34 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Wed, 27 Oct 2010 23:57:34 -0600 Subject: [IronPython] The Road to 2.7 (Was: The Future of IronPython) Message-ID: On Sun, Oct 24, 2010 at 4:45 AM, Lukas Cenovsky wrote: > I'd like to help. What is the best place to start? Right now, we need to identify what's not working in 2.7B1 that needs to be in 2.7 final. The best thing to do would be to identify any issues that are causing you pain and bring them up on the list. Then we can decide what meets the bar for a 2.7 release (with issues that have patches getting priority, of course!). Dino, are there any issues that you know are in 2.7B1 that must be fixed for 2.7 final? Dino has provided some instructions on contributing to IronPython: http://ironpython.codeplex.com/wikipage?title=Respository%20Instructions&referringTitle=Home. We need people to run through that and if there's anything it doesn't cover (I do intend to add subversion instructions directly at some point), and run the test suite as well. Also, doing all of that on Mono, to see what work needs to be done there. Besides knowing what needs to be done, we need a rough timeline. I would like to see a release before the end of the year, or at least a solid release candidate with a possible release early next year. The idea behind an aggressive schedule is to focus on getting the features we have solid and not worry about adding new features or libraries (with the possible exception of zlib). That said, this all just my desires, and I really want to get an idea of what every one else is thinking. - Jeff From jdhardy at gmail.com Thu Oct 28 09:27:22 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Thu, 28 Oct 2010 01:27:22 -0600 Subject: [IronPython] The elephant in the room: source control for IronPython Message-ID: Currently, IronPython is hosted in a TFS repository on CodePlex (http://ironpython.codeplex.com/), which was a copy of MS's internal TFS repository. CodePlex also provides Subversion access, which makes it much more bearable. CodePlex also hosts our issue tracking and wiki pages, which probably won't change any time soon. IronRuby's source code is hosted on github (http://github.com/ironruby/ironruby). It's also a copy of MS's internal TFS repository, but in git. The interesting part is that IronRuby, IronPython, and the DLR are hosted in the *same* repository, since they evolved together. Thus, both the IronPython CodePlex repo and the IronRuby github repo are basically the same. What this is going to look like in the future is an open question, as is the timeline. Originally, I wanted to focus on the 2.7 release and deal with the source control question later. However, it's been raised in a few places, so I think it's better to get some more feedback on whether we should switch source control methods (and if so, to what?) or just stay on TFS/SVN for the time being. Also up for consideration is whether you consider being part of the same repo as IronRuby is valuable, or whether IronPython should split out on its own. We could, for example, drop the source control from CodePlex and just use the IronRuby github repo - it's already set up and we could start developing tomorrow (although it would probably be renamed 'ironlanguages' or something like that). It's also probably the only option if IronPython and IronRuby are to share a repo, as, so far as I know, the IronRuby guys have no plans on leaving github, which makes sense for them - git is the de facto choice in the Ruby community. In Python, however, it's not so clear-cut - Python itself will be moving to Mercurial soon, and there are plans afoot to eventually put the Python stdlib in a separate repo from Python itself, which will likely also be a Mercurial repository. Thus there are advantages (subrepos, in particular) to being on the same DVCS. On top of that, both Michael Foord and I strongly dislike git - I prefer Mercurial, and I imagine the coffee at Canonical will have Michael singing the praises of bzr fairly soon :). Finally, CodePlex supports Mercurial, and thus everything could remain there if we so wish. However, converting the repo to Mercurial could be a difficult task - the fate of the 1.1, 2.0, and 2.6 branches would have to be decided (include them in the repo, or not? Their structure is radically different from the Main branch). There are folders that could very well be stripped (WiX, Ruby, and *3* CPython installations, not to mention IronRuby) to save space, and with a DVCS once they're in the history everyone has to pay that cost in disk space, forever, even if we later remove them. The fate of the DLR would need to be decided - do we keep a local copy, pull from IronRuby's copy, or make it a third repo altogether? My preference is to stick with TFS/SVN for the time being, get 2.7 out the door (manually syncing up the DLR sources with IronRuby in the meantime), and then look at converting to Mercurial. My second choice would be to work out of IronRuby's git repository, get 2.7 released, and then look at converting to Mercurial. Anything that doesn't eventually involve Mercurial is a lot further down my list :). I would like to see the DLR become a separate project, of which IronRuby and IronPython are just clients, along with IronJS, Clojure-CLR, and any others. I don't think the DLR will change too drastically, but the MS guys who are more familiar might have other plans, and Tomas has said he would prefer them to be together for ease of testing. While the coordinators have discussed this already, I think we need more feedback to get an idea of what we should do, so please share your thoughts. This has a direct bearing on how you will be contributing to IronPython. - Jeff From cenovsky at bakalari.cz Thu Oct 28 09:43:48 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Thu, 28 Oct 2010 09:43:48 +0200 Subject: [IronPython] The Road to 2.7 (Was: The Future of IronPython) In-Reply-To: References: Message-ID: <4CC929B4.5030408@bakalari.cz> On 28.10.2010 7:57, Jeff Hardy wrote: > On Sun, Oct 24, 2010 at 4:45 AM, Lukas Cenovsky wrote: >> I'd like to help. What is the best place to start? > Right now, we need to identify what's not working in 2.7B1 that needs > to be in 2.7 final. The best thing to do would be to identify any > issues that are causing you pain and bring them up on the list. Then > we can decide what meets the bar for a 2.7 release (with issues that > have patches getting priority, of course!). Dino, are there any issues > that you know are in 2.7B1 that must be fixed for 2.7 final? I have not found any issue so far. > Dino has provided some instructions on contributing to IronPython: > http://ironpython.codeplex.com/wikipage?title=Respository%20Instructions&referringTitle=Home. > We need people to run through that and if there's anything it doesn't > cover (I do intend to add subversion instructions directly at some > point), and run the test suite as well. Also, doing all of that on > Mono, to see what work needs to be done there. I'll go through it during the weekend for .NET (not Mono). > Besides knowing what needs to be done, we need a rough timeline. I > would like to see a release before the end of the year, or at least a > solid release candidate with a possible release early next year. The > idea behind an aggressive schedule is to focus on getting the features > we have solid and not worry about adding new features or libraries > (with the possible exception of zlib). That said, this all just my > desires, and I really want to get an idea of what every one else is > thinking. I agree with this schedule. It was quite a pain waiting for 2.6.2 bugfixes so I am for quicker releases. -- -- Luk?? From s.j.dower at gmail.com Thu Oct 28 09:53:28 2010 From: s.j.dower at gmail.com (Steve Dower) Date: Thu, 28 Oct 2010 18:53:28 +1100 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: I'll add in a vote for Mercurial (voting always seems to be how to decide on VCS), though I still believe that SVN works better for a contribution/review/patch workflow. Is the plan after 2.7 to start doing 3? That seems like a good opportunity to "start fresh" in a new repository and leave the old stuff where it is, only carrying over the barest minimum. I can't see any movement before 2.7 as being worthwhile. I'd like to see the DLR separated, but doing so does make cross-versioning more difficult. However, following a log that has three separate projects (assuming DLR, IronPy and IronRuby) is messy. Patching potentially gets a lot harder as well. I do intend to contribute, as soon as I stop breaking my own project :) How up to date is the CodePlex issue tracker? On Thu, Oct 28, 2010 at 18:27, Jeff Hardy wrote: > Currently, IronPython is hosted in a TFS repository on CodePlex > (http://ironpython.codeplex.com/), which was a copy of MS's internal > TFS repository. CodePlex also provides Subversion access, which makes > it much more bearable. CodePlex also hosts our issue tracking and wiki > pages, which probably won't change any time soon. > > IronRuby's source code is hosted on github > (http://github.com/ironruby/ironruby). It's also a copy of MS's > internal TFS repository, but in git. > > The interesting part is that IronRuby, IronPython, and the DLR are > hosted in the *same* repository, since they evolved together. Thus, > both the IronPython CodePlex repo and the IronRuby github repo are > basically the same. > > > What this is going to look like in the future is an open question, as > is the timeline. Originally, I wanted to focus on the 2.7 release and > deal with the source control question later. However, it's been raised > in a few places, so I think it's better to get some more feedback on > whether we should switch source control methods (and if so, to what?) > or just stay on TFS/SVN for the time being. Also up for consideration > is whether you consider being part of the same repo as IronRuby is > valuable, or whether IronPython should split out on its own. > > We could, for example, drop the source control from CodePlex and just > use the IronRuby github repo - it's already set up and we could start > developing tomorrow (although it would probably be renamed > 'ironlanguages' or something like that). It's also probably the only > option if IronPython and IronRuby are to share a repo, as, so far as I > know, the IronRuby guys have no plans on leaving github, which makes > sense for them - git is the de facto choice in the Ruby community. > > In Python, however, it's not so clear-cut - Python itself will be > moving to Mercurial soon, and there are plans afoot to eventually put > the Python stdlib in a separate repo from Python itself, which will > likely also be a Mercurial repository. Thus there are advantages > (subrepos, in particular) to being on the same DVCS. On top of that, > both Michael Foord and I strongly dislike git - I prefer Mercurial, > and I imagine the coffee at Canonical will have Michael singing the > praises of bzr fairly soon :). Finally, CodePlex supports Mercurial, > and thus everything could remain there if we so wish. > > However, converting the repo to Mercurial could be a difficult task - > the fate of the 1.1, 2.0, and 2.6 branches would have to be decided > (include them in the repo, or not? Their structure is radically > different from the Main branch). There are folders that could very > well be stripped (WiX, Ruby, and *3* CPython installations, not to > mention IronRuby) to save space, and with a DVCS once they're in the > history everyone has to pay that cost in disk space, forever, even if > we later remove them. The fate of the DLR would need to be decided - > do we keep a local copy, pull from IronRuby's copy, or make it a third > repo altogether? > > My preference is to stick with TFS/SVN for the time being, get 2.7 out > the door (manually syncing up the DLR sources with IronRuby in the > meantime), and then look at converting to Mercurial. My second choice > would be to work out of IronRuby's git repository, get 2.7 released, > and then look at converting to Mercurial. Anything that doesn't > eventually involve Mercurial is a lot further down my list :). > > I would like to see the DLR become a separate project, of which > IronRuby and IronPython are just clients, along with IronJS, > Clojure-CLR, and any others. I don't think the DLR will change too > drastically, but the MS guys who are more familiar might have other > plans, and Tomas has said he would prefer them to be together for ease > of testing. > > While the coordinators have discussed this already, I think we need > more feedback to get an idea of what we should do, so please share > your thoughts. This has a direct bearing on how you will be > contributing to IronPython. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > From cenovsky at bakalari.cz Thu Oct 28 10:06:07 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Thu, 28 Oct 2010 10:06:07 +0200 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: <4CC92EEF.203@bakalari.cz> Hi, I vote for staying wiht the current setup until final 2.7 release. I do not have any experiences wiht DVCS so I don't know whether Mercurial is better than git or vice versa. I use SVN so being on CodePlex which has SVN access is fine for me. There has been a discussion in IronRuby mailing list and it seem their goal is to split current monolithic repo to three parts: IronRuby, IronPython and DLR (see http://rubyforge.org/pipermail/ironruby-core/2010-October/007562.html). The reasons for this makes perfect sense for me too although there are some concerns that testing the interoperability and developing similar parts would be easier in one repo (http://rubyforge.org/pipermail/ironruby-core/2010-October/007493.html). -- -- Luk?? On 28.10.2010 9:27, Jeff Hardy wrote: > Currently, IronPython is hosted in a TFS repository on CodePlex > (http://ironpython.codeplex.com/), which was a copy of MS's internal > TFS repository. CodePlex also provides Subversion access, which makes > it much more bearable. CodePlex also hosts our issue tracking and wiki > pages, which probably won't change any time soon. > > IronRuby's source code is hosted on github > (http://github.com/ironruby/ironruby). It's also a copy of MS's > internal TFS repository, but in git. > > The interesting part is that IronRuby, IronPython, and the DLR are > hosted in the *same* repository, since they evolved together. Thus, > both the IronPython CodePlex repo and the IronRuby github repo are > basically the same. > > > What this is going to look like in the future is an open question, as > is the timeline. Originally, I wanted to focus on the 2.7 release and > deal with the source control question later. However, it's been raised > in a few places, so I think it's better to get some more feedback on > whether we should switch source control methods (and if so, to what?) > or just stay on TFS/SVN for the time being. Also up for consideration > is whether you consider being part of the same repo as IronRuby is > valuable, or whether IronPython should split out on its own. > > We could, for example, drop the source control from CodePlex and just > use the IronRuby github repo - it's already set up and we could start > developing tomorrow (although it would probably be renamed > 'ironlanguages' or something like that). It's also probably the only > option if IronPython and IronRuby are to share a repo, as, so far as I > know, the IronRuby guys have no plans on leaving github, which makes > sense for them - git is the de facto choice in the Ruby community. > > In Python, however, it's not so clear-cut - Python itself will be > moving to Mercurial soon, and there are plans afoot to eventually put > the Python stdlib in a separate repo from Python itself, which will > likely also be a Mercurial repository. Thus there are advantages > (subrepos, in particular) to being on the same DVCS. On top of that, > both Michael Foord and I strongly dislike git - I prefer Mercurial, > and I imagine the coffee at Canonical will have Michael singing the > praises of bzr fairly soon :). Finally, CodePlex supports Mercurial, > and thus everything could remain there if we so wish. > > However, converting the repo to Mercurial could be a difficult task - > the fate of the 1.1, 2.0, and 2.6 branches would have to be decided > (include them in the repo, or not? Their structure is radically > different from the Main branch). There are folders that could very > well be stripped (WiX, Ruby, and *3* CPython installations, not to > mention IronRuby) to save space, and with a DVCS once they're in the > history everyone has to pay that cost in disk space, forever, even if > we later remove them. The fate of the DLR would need to be decided - > do we keep a local copy, pull from IronRuby's copy, or make it a third > repo altogether? > > My preference is to stick with TFS/SVN for the time being, get 2.7 out > the door (manually syncing up the DLR sources with IronRuby in the > meantime), and then look at converting to Mercurial. My second choice > would be to work out of IronRuby's git repository, get 2.7 released, > and then look at converting to Mercurial. Anything that doesn't > eventually involve Mercurial is a lot further down my list :). > > I would like to see the DLR become a separate project, of which > IronRuby and IronPython are just clients, along with IronJS, > Clojure-CLR, and any others. I don't think the DLR will change too > drastically, but the MS guys who are more familiar might have other > plans, and Tomas has said he would prefer them to be together for ease > of testing. > > While the coordinators have discussed this already, I think we need > more feedback to get an idea of what we should do, so please share > your thoughts. This has a direct bearing on how you will be > contributing to IronPython. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > From pablodalma93 at hotmail.com Thu Oct 28 13:17:00 2010 From: pablodalma93 at hotmail.com (Pablo Dalmazzo) Date: Thu, 28 Oct 2010 08:17:00 -0300 Subject: [IronPython] Thank you all In-Reply-To: <003a01cb763d$b370fc40$1a52f4c0$@tedneward.com> References: , , <7993FB06FBE14EB38343F91BAAC341FC@Gamry.com>, , , , , , , , <003a01cb763d$b370fc40$1a52f4c0$@tedneward.com> Message-ID: Oh, I didnt remember where I've seen about a "S#" or what this language was about, it might have been this languagehttp://en.wikipedia.org/wiki/Script.NET which it seems to be known also as "S#" according to that article From: ted at tedneward.com To: users at lists.ironpython.com Date: Wed, 27 Oct 2010 18:16:01 -0700 Subject: Re: [IronPython] Thank you all A long long time ago there was some bits for a Smalltalk port to .NET called S#. I have no idea if that?s still alive or not. I could never find a good working download link for it. Ted NewardJava, .NET, XML ServicesConsulting, Teaching, Speaking, Writinghttp://www.tedneward.com From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Pablo Dalmazzo Sent: Tuesday, October 26, 2010 5:21 AM To: IronPython Mailing list Subject: Re: [IronPython] Thank you all isnt there already a language for .NET called S# or it's the same language you're talking about? GreetingsDate: Fri, 22 Oct 2010 12:57:51 -0400 From: miguel at novell.com To: users at lists.ironpython.com CC: miguel.novell at gmail.com Subject: Re: [IronPython] Thank you all Hello,Miguel tweeted about turning it into a full fledged scripting language called S# at one point. This comes in a number of steps. First, we want to have our C# compiler support the same hosting APIs that the current Iron* languages support. Second, we are going to have an option to have our C# compiler recognize a different language that we are calling "S#". It is not too far from C#, it is basically C# with the features that we wanted to get into the language for many years, and we feel that the time is right to get these out to the public. We have an internal draft document, and once it is more baked we will circulate it. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Thu Oct 28 15:27:34 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Thu, 28 Oct 2010 08:27:34 -0500 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: On Thu, Oct 28, 2010 at 02:27, Jeff Hardy wrote: > > My preference is to stick with TFS/SVN for the time being, get 2.7 out > the door (manually syncing up the DLR sources with IronRuby in the > meantime), and then look at converting to Mercurial. My second choice > would be to work out of IronRuby's git repository, get 2.7 released, > and then look at converting to Mercurial. Anything that doesn't > eventually involve Mercurial is a lot further down my list :). Any path ending with Mercurial makes it more likely that I'll contribute. TFS/SVN through 2.7, then on to Mercurial seems like a reasonable choice and would get a +1 from me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Thu Oct 28 15:34:23 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 28 Oct 2010 09:34:23 -0400 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: <4CC97BDF.3000800@voidspace.org.uk> On 28/10/2010 03:53, Steve Dower wrote: > I'll add in a vote for Mercurial (voting always seems to be how to > decide on VCS), though I still believe that SVN works better for a > contribution/review/patch workflow. Distributed version control systems are very good for distributed development (funny that). Whilst I'm not proposing we use bzr (I would *very* much like us to use mercurial though), our workflow at Canonical with bzr and launchpad is great. You develop in a branch (branching is very easy with dcvs') and push to launchpad. On requesting a merge review you get a great web interface with viewable diff and when the merge is approved you merge back onto trunk. (Branching and merging are substantially easier with hg / bzr / git than with svn.) Having many outstanding branches waiting for review (and keeping them up to date) and starting new branches that depend on other branches is all very easy. Having infrastructure that allows us to easily store feature branches is important for that particular workflow. > Is the plan after 2.7 to start doing 3? That seems like a good > opportunity to "start fresh" in a new repository and leave the old > stuff where it is, only carrying over the barest minimum. I can't see > any movement before 2.7 as being worthwhile. Interesting question. Ideally we would do parallel development but I'm not sure we have the resources for that. > I'd like to see the DLR separated, but doing so does make > cross-versioning more difficult. However, following a log that has > three separate projects (assuming DLR, IronPy and IronRuby) is messy. > Patching potentially gets a lot harder as well. > > I do intend to contribute, as soon as I stop breaking my own project > :) How up to date is the CodePlex issue tracker? The issue tracker has been pretty well maintained. All the best, Michael Foord > On Thu, Oct 28, 2010 at 18:27, Jeff Hardy wrote: >> Currently, IronPython is hosted in a TFS repository on CodePlex >> (http://ironpython.codeplex.com/), which was a copy of MS's internal >> TFS repository. CodePlex also provides Subversion access, which makes >> it much more bearable. CodePlex also hosts our issue tracking and wiki >> pages, which probably won't change any time soon. >> >> IronRuby's source code is hosted on github >> (http://github.com/ironruby/ironruby). It's also a copy of MS's >> internal TFS repository, but in git. >> >> The interesting part is that IronRuby, IronPython, and the DLR are >> hosted in the *same* repository, since they evolved together. Thus, >> both the IronPython CodePlex repo and the IronRuby github repo are >> basically the same. >> >> >> What this is going to look like in the future is an open question, as >> is the timeline. Originally, I wanted to focus on the 2.7 release and >> deal with the source control question later. However, it's been raised >> in a few places, so I think it's better to get some more feedback on >> whether we should switch source control methods (and if so, to what?) >> or just stay on TFS/SVN for the time being. Also up for consideration >> is whether you consider being part of the same repo as IronRuby is >> valuable, or whether IronPython should split out on its own. >> >> We could, for example, drop the source control from CodePlex and just >> use the IronRuby github repo - it's already set up and we could start >> developing tomorrow (although it would probably be renamed >> 'ironlanguages' or something like that). It's also probably the only >> option if IronPython and IronRuby are to share a repo, as, so far as I >> know, the IronRuby guys have no plans on leaving github, which makes >> sense for them - git is the de facto choice in the Ruby community. >> >> In Python, however, it's not so clear-cut - Python itself will be >> moving to Mercurial soon, and there are plans afoot to eventually put >> the Python stdlib in a separate repo from Python itself, which will >> likely also be a Mercurial repository. Thus there are advantages >> (subrepos, in particular) to being on the same DVCS. On top of that, >> both Michael Foord and I strongly dislike git - I prefer Mercurial, >> and I imagine the coffee at Canonical will have Michael singing the >> praises of bzr fairly soon :). Finally, CodePlex supports Mercurial, >> and thus everything could remain there if we so wish. >> >> However, converting the repo to Mercurial could be a difficult task - >> the fate of the 1.1, 2.0, and 2.6 branches would have to be decided >> (include them in the repo, or not? Their structure is radically >> different from the Main branch). There are folders that could very >> well be stripped (WiX, Ruby, and *3* CPython installations, not to >> mention IronRuby) to save space, and with a DVCS once they're in the >> history everyone has to pay that cost in disk space, forever, even if >> we later remove them. The fate of the DLR would need to be decided - >> do we keep a local copy, pull from IronRuby's copy, or make it a third >> repo altogether? >> >> My preference is to stick with TFS/SVN for the time being, get 2.7 out >> the door (manually syncing up the DLR sources with IronRuby in the >> meantime), and then look at converting to Mercurial. My second choice >> would be to work out of IronRuby's git repository, get 2.7 released, >> and then look at converting to Mercurial. Anything that doesn't >> eventually involve Mercurial is a lot further down my list :). >> >> I would like to see the DLR become a separate project, of which >> IronRuby and IronPython are just clients, along with IronJS, >> Clojure-CLR, and any others. I don't think the DLR will change too >> drastically, but the MS guys who are more familiar might have other >> plans, and Tomas has said he would prefer them to be together for ease >> of testing. >> >> While the coordinators have discussed this already, I think we need >> more feedback to get an idea of what we should do, so please share >> your thoughts. This has a direct bearing on how you will be >> contributing to IronPython. >> >> - Jeff >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/ From mark.heath at gmail.com Thu Oct 28 16:34:09 2010 From: mark.heath at gmail.com (Mark Heath) Date: Thu, 28 Oct 2010 15:34:09 +0100 Subject: [IronPython] Using WebClient with Gestalt Message-ID: I'm writing an IronPython Silverlight application and trying to use WebClient to download some xaml so I can use XamlReader.Load to load it. I have the following code in a .py file that is referenced by my main HTML page: wc = WebClient() wc.DownloadStringCompleted += self.xamlDownloaded wc.DownloadStringAsync(Uri('star.xaml',UriKind.Relative)) however, in the completed handler I get a SystemError - not found. I know that the xaml file is present, so I'm wondering if it is looking for it relative to dlr.xap. I didn't get much more success with an absolute Uri - just a SystemError with no message at all, which I guess could be due to Silverlight cross-domain request permissions. Any suggestions for how I can get this working? And another quick question while I'm at it. Are there any known issues with IronPython and data-binding in Silverlight? I converted an IronPython WPF app to Silverlight that was using MVVM, but in Silverlight none of the bindings worked at all. thanks Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From cenovsky at bakalari.cz Thu Oct 28 17:32:31 2010 From: cenovsky at bakalari.cz (Lukas Cenovsky) Date: Thu, 28 Oct 2010 17:32:31 +0200 Subject: [IronPython] Using WebClient with Gestalt In-Reply-To: References: Message-ID: <4CC9978F.20503@bakalari.cz> On 28.10.2010 16:34, Mark Heath wrote: > I'm writing an IronPython Silverlight application and trying to use > WebClient to download some xaml so I can use XamlReader.Load to load > it. I have the following code in a .py file that is referenced by my > main HTML page: > > wc = WebClient() > wc.DownloadStringCompleted += self.xamlDownloaded > wc.DownloadStringAsync(Uri('star.xaml',UriKind.Relative)) > > however, in the completed handler I get a SystemError - not found. I > know that the xaml file is present, so I'm wondering if it is looking > for it relative to dlr.xap. I didn't get much more success with an > absolute Uri - just a SystemError with no message at all, which I > guess could be due to Silverlight cross-domain request permissions. > > Any suggestions for how I can get this working? > > And another quick question while I'm at it. Are there any known issues > with IronPython and data-binding in Silverlight? I converted an > IronPython WPF app to Silverlight that was using MVVM, but in > Silverlight none of the bindings worked at all. For binding, you have to user clrtype.py. See my blog: http://gui-at.blogspot.com/2009/11/inotifypropertychanged-and-databinding.html -- -- Luk?? From dinov at microsoft.com Thu Oct 28 18:05:52 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 28 Oct 2010 16:05:52 +0000 Subject: [IronPython] The Road to 2.7 (Was: The Future of IronPython) In-Reply-To: References: Message-ID: Jeff wrote: > From: users-bounces at lists.ironpython.com [mailto:users- > bounces at lists.ironpython.com] On Behalf Of Jeff Hardy > Sent: Wednesday, October 27, 2010 10:58 PM > To: Discussion of IronPython > Subject: [IronPython] The Road to 2.7 (Was: The Future of IronPython) > > On Sun, Oct 24, 2010 at 4:45 AM, Lukas Cenovsky > wrote: > > I'd like to help. What is the best place to start? > > Right now, we need to identify what's not working in 2.7B1 that needs to be > in 2.7 final. The best thing to do would be to identify any issues that are > causing you pain and bring them up on the list. Then we can decide what > meets the bar for a 2.7 release (with issues that have patches getting priority, > of course!). Dino, are there any issues that you know are in 2.7B1 that must > be fixed for 2.7 final? I think the biggest thing would be to look at the tests which are disabled due to CodePlex bug #28171. Grepping the External.LCA_RESTRICTED\Languages\IronPython\27\Lib\test Directory shows all of the places where things are disabled right now and they all represent incompatibilities w/ CPython 2.7. Some of them may be things that were broken all along and there's just tests for now. Others may be changes in behavior between 2.6 and 2.7. They're also all lumped under this one bug right now which makes it difficult to reason about the individual issues. So there's lots of opportunity to either dive in and fix these, open individual bugs for each of the issues, triage the issues, help identity which ones are more important, see if they've been inadvertently fixed along the way to 2.7, etc... And in general I'd say there's a lot of other bugs in the bug tracker that could be fixed to deliver a really high quality 2.7. From Tomas.Matousek at microsoft.com Thu Oct 28 18:45:15 2010 From: Tomas.Matousek at microsoft.com (Tomas Matousek) Date: Thu, 28 Oct 2010 16:45:15 +0000 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: " following a log that has three separate projects (assuming DLR, IronPy and IronRuby) is messy" We can use git submodules to separate the logs. Tomas -----Original Message----- From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Steve Dower Sent: Thursday, October 28, 2010 12:53 AM To: Discussion of IronPython Subject: Re: [IronPython] The elephant in the room: source control for IronPython I'll add in a vote for Mercurial (voting always seems to be how to decide on VCS), though I still believe that SVN works better for a contribution/review/patch workflow. Is the plan after 2.7 to start doing 3? That seems like a good opportunity to "start fresh" in a new repository and leave the old stuff where it is, only carrying over the barest minimum. I can't see any movement before 2.7 as being worthwhile. I'd like to see the DLR separated, but doing so does make cross-versioning more difficult. However, following a log that has three separate projects (assuming DLR, IronPy and IronRuby) is messy. Patching potentially gets a lot harder as well. I do intend to contribute, as soon as I stop breaking my own project :) How up to date is the CodePlex issue tracker? On Thu, Oct 28, 2010 at 18:27, Jeff Hardy wrote: > Currently, IronPython is hosted in a TFS repository on CodePlex > (http://ironpython.codeplex.com/), which was a copy of MS's internal > TFS repository. CodePlex also provides Subversion access, which makes > it much more bearable. CodePlex also hosts our issue tracking and wiki > pages, which probably won't change any time soon. > > IronRuby's source code is hosted on github > (http://github.com/ironruby/ironruby). It's also a copy of MS's > internal TFS repository, but in git. > > The interesting part is that IronRuby, IronPython, and the DLR are > hosted in the *same* repository, since they evolved together. Thus, > both the IronPython CodePlex repo and the IronRuby github repo are > basically the same. > > > What this is going to look like in the future is an open question, as > is the timeline. Originally, I wanted to focus on the 2.7 release and > deal with the source control question later. However, it's been raised > in a few places, so I think it's better to get some more feedback on > whether we should switch source control methods (and if so, to what?) > or just stay on TFS/SVN for the time being. Also up for consideration > is whether you consider being part of the same repo as IronRuby is > valuable, or whether IronPython should split out on its own. > > We could, for example, drop the source control from CodePlex and just > use the IronRuby github repo - it's already set up and we could start > developing tomorrow (although it would probably be renamed > 'ironlanguages' or something like that). It's also probably the only > option if IronPython and IronRuby are to share a repo, as, so far as I > know, the IronRuby guys have no plans on leaving github, which makes > sense for them - git is the de facto choice in the Ruby community. > > In Python, however, it's not so clear-cut - Python itself will be > moving to Mercurial soon, and there are plans afoot to eventually put > the Python stdlib in a separate repo from Python itself, which will > likely also be a Mercurial repository. Thus there are advantages > (subrepos, in particular) to being on the same DVCS. On top of that, > both Michael Foord and I strongly dislike git - I prefer Mercurial, > and I imagine the coffee at Canonical will have Michael singing the > praises of bzr fairly soon :). Finally, CodePlex supports Mercurial, > and thus everything could remain there if we so wish. > > However, converting the repo to Mercurial could be a difficult task - > the fate of the 1.1, 2.0, and 2.6 branches would have to be decided > (include them in the repo, or not? Their structure is radically > different from the Main branch). There are folders that could very > well be stripped (WiX, Ruby, and *3* CPython installations, not to > mention IronRuby) to save space, and with a DVCS once they're in the > history everyone has to pay that cost in disk space, forever, even if > we later remove them. The fate of the DLR would need to be decided - > do we keep a local copy, pull from IronRuby's copy, or make it a third > repo altogether? > > My preference is to stick with TFS/SVN for the time being, get 2.7 out > the door (manually syncing up the DLR sources with IronRuby in the > meantime), and then look at converting to Mercurial. My second choice > would be to work out of IronRuby's git repository, get 2.7 released, > and then look at converting to Mercurial. Anything that doesn't > eventually involve Mercurial is a lot further down my list :). > > I would like to see the DLR become a separate project, of which > IronRuby and IronPython are just clients, along with IronJS, > Clojure-CLR, and any others. I don't think the DLR will change too > drastically, but the MS guys who are more familiar might have other > plans, and Tomas has said he would prefer them to be together for ease > of testing. > > While the coordinators have discussed this already, I think we need > more feedback to get an idea of what we should do, so please share > your thoughts. This has a direct bearing on how you will be > contributing to IronPython. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com From adal.chiriliuc at gmail.com Thu Oct 28 19:14:18 2010 From: adal.chiriliuc at gmail.com (Adal Chiriliuc) Date: Thu, 28 Oct 2010 20:14:18 +0300 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: I would prefer Mercurial for the simple reason that git is not very well supported on Windows and it's ports are not robust enough. And I would dump pre 2.7 history if it's too much of a hassle to convert. If someone would really need to check ancient history, it could just use the old repository. I would also strive to have a smallish repository. I once tried to check out the WebKit code and gave up after 4 hours of downloading and hundred of thousands of files created on my disk. From vernondcole at gmail.com Thu Oct 28 20:16:30 2010 From: vernondcole at gmail.com (Vernon Cole) Date: Thu, 28 Oct 2010 12:16:30 -0600 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: <4CC97BDF.3000800@voidspace.org.uk> References: <4CC97BDF.3000800@voidspace.org.uk> Message-ID: On Thu, Oct 28, 2010 at 7:34 AM, Michael Foord wrote: > On 28/10/2010 03:53, Steve Dower wrote: > >> I'll add in a vote for Mercurial (voting always seems to be how to >> decide on VCS), though I still believe that SVN works better for a >> contribution/review/patch workflow. >> > > Distributed version control systems are very good for distributed > development (funny that). Whilst I'm not proposing we use bzr (I would > *very* much like us to use mercurial though), our workflow at Canonical with > bzr and launchpad is great. You develop in a branch (branching is very easy > with dcvs') and push to launchpad. On requesting a merge review you get a > great web interface with viewable diff and when the merge is approved you > merge back onto trunk. (Branching and merging are substantially easier with > hg / bzr / git than with svn.) > > Having many outstanding branches waiting for review (and keeping them up to > date) and starting new branches that depend on other branches is all very > easy. Having infrastructure that allows us to easily store feature branches > is important for that particular workflow. > > I agree completely. Right now on my *laptop*, I have 22 bazaar repositories, one CVS, and 4 mercurial. Two of the latter are IPy specific: ironpython.sqlite and adodbapi. PyWin32 will also be going to mercurial as soon as Mark Hammond gets around to it. (If they ever get a Windows version of bzr-hg working I won't even have to remember two slightly different command sets.) The correct choice for the future is hg. > > Is the plan after 2.7 to start doing 3? That seems like a good >> opportunity to "start fresh" in a new repository and leave the old >> stuff where it is, only carrying over the barest minimum. I can't see >> any movement before 2.7 as being worthwhile. >> > > Interesting question. Ideally we would do parallel development but I'm not > sure we have the resources for that. > Python 2.7 is documented to be the LAST of its family. There should not be very much "development", except perhaps filling out the standard library. I would say to put 3.n on a fresh hg tree, and back-port anything necessary into the existing 2.7 tree and infrastructure. No sense in re-inventing that wheel. Much of the difficulty in converting 2.n Python in 3.n has to do with converting between 8-byte strings and Unicode. Since IronPython is already there, I anticipate that there will be minimal problems in 2to3. Parallel development may be largely unnecessary. [Having a piece of published code which runs on all three platforms (2.n, IPy & 3.n with only one trunk version - no branches), I think I can speak with some authority.] -- Vernon Cole -------------- next part -------------- An HTML attachment was scrubbed... URL: From tristanz at gmail.com Thu Oct 28 21:28:33 2010 From: tristanz at gmail.com (Tristan Zajonc) Date: Thu, 28 Oct 2010 15:28:33 -0400 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: <4CC97BDF.3000800@voidspace.org.uk> Message-ID: While both hg and git are fine, I'd vote for git so that we are using the same DVCS as IronRuby, the DLR, and Mono. GitHub also provides excellent community features and is larger than bitbucket. Obviously Michael's and Jeff's opinion probably matter most. I'd suggest just making the call and sticking to it. On Thu, Oct 28, 2010 at 2:16 PM, Vernon Cole wrote: > > On Thu, Oct 28, 2010 at 7:34 AM, Michael Foord wrote: > >> On 28/10/2010 03:53, Steve Dower wrote: >> >>> I'll add in a vote for Mercurial (voting always seems to be how to >>> decide on VCS), though I still believe that SVN works better for a >>> contribution/review/patch workflow. >>> >> >> Distributed version control systems are very good for distributed >> development (funny that). Whilst I'm not proposing we use bzr (I would >> *very* much like us to use mercurial though), our workflow at Canonical with >> bzr and launchpad is great. You develop in a branch (branching is very easy >> with dcvs') and push to launchpad. On requesting a merge review you get a >> great web interface with viewable diff and when the merge is approved you >> merge back onto trunk. (Branching and merging are substantially easier with >> hg / bzr / git than with svn.) >> >> Having many outstanding branches waiting for review (and keeping them up >> to date) and starting new branches that depend on other branches is all very >> easy. Having infrastructure that allows us to easily store feature branches >> is important for that particular workflow. >> >> I agree completely. Right now on my *laptop*, I have 22 bazaar > repositories, one CVS, and 4 mercurial. Two of the latter are IPy specific: > ironpython.sqlite and adodbapi. PyWin32 will also be going to mercurial as > soon as Mark Hammond gets around to it. (If they ever get a Windows version > of bzr-hg working I won't even have to remember two slightly different > command sets.) The correct choice for the future is hg. > >> >> Is the plan after 2.7 to start doing 3? That seems like a good >>> opportunity to "start fresh" in a new repository and leave the old >>> stuff where it is, only carrying over the barest minimum. I can't see >>> any movement before 2.7 as being worthwhile. >>> >> >> Interesting question. Ideally we would do parallel development but I'm not >> sure we have the resources for that. >> > > Python 2.7 is documented to be the LAST of its family. There should not be > very much "development", except perhaps filling out the standard library. > I would say to put 3.n on a fresh hg tree, and back-port anything necessary > into the existing 2.7 tree and infrastructure. No sense in re-inventing that > wheel. > > Much of the difficulty in converting 2.n Python in 3.n has to do with > converting between 8-byte strings and Unicode. Since IronPython is already > there, I anticipate that there will be minimal problems in 2to3. Parallel > development may be largely unnecessary. [Having a piece of published code > which runs on all three platforms (2.n, IPy & 3.n with only one trunk > version - no branches), I think I can speak with some authority.] > -- > Vernon Cole > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From railcar88 at gmail.com Fri Oct 29 00:42:38 2010 From: railcar88 at gmail.com (Scott Scites) Date: Thu, 28 Oct 2010 18:42:38 -0400 Subject: [IronPython] The elephant in the room: source control for IronPython Message-ID: +1 for mercurial and breaking the dlr, ironruby, ironpython, etc. into separate repositories. I prefer to use bitbucket but could live with codeplex as the host. -------------- next part -------------- An HTML attachment was scrubbed... URL: From slide.o.mix at gmail.com Fri Oct 29 00:55:32 2010 From: slide.o.mix at gmail.com (Slide) Date: Thu, 28 Oct 2010 15:55:32 -0700 Subject: [IronPython] ScriptRuntime.ImportModule("foo") vs. import foo Message-ID: I have something like the following code: ScriptEngine engine = Python.CreateEngine(); ScriptScope scope = engine.CreateScope(); engine.Runtime.LoadAssembly(Assembly.GetExecutingAssembly()); // holds the namespace foo // the following line throws an exception engine.Runtime.ImportModule("foo"); // the following if executed works fine engine.Execute("import foo"); I'm basically just trying to make the types in my executing assembly available to the script. I assumed that the ImportModule would see things that had been loaded using LoadAssembly, but this doesn't seem to be the case. Does anyone have examples of doing this that works? Thanks, slide From brendenconolly at gmail.com Fri Oct 29 00:55:30 2010 From: brendenconolly at gmail.com (BC) Date: Fri, 29 Oct 2010 08:55:30 +1000 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: +1 for Mecurial and keeping the projects in separate repo's and I also agree that finishing 2.7 and moving post is a good move. From dinov at microsoft.com Fri Oct 29 01:14:14 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 28 Oct 2010 23:14:14 +0000 Subject: [IronPython] ScriptRuntime.ImportModule("foo") vs. import foo In-Reply-To: References: Message-ID: Slide wrote: > I have something like the following code: > > ScriptEngine engine = Python.CreateEngine(); ScriptScope scope = > engine.CreateScope(); > > engine.Runtime.LoadAssembly(Assembly.GetExecutingAssembly()); // holds > the namespace foo > > // the following line throws an exception > engine.Runtime.ImportModule("foo"); > > // the following if executed works fine > engine.Execute("import foo"); > > > I'm basically just trying to make the types in my executing assembly available > to the script. I assumed that the ImportModule would see things that had > been loaded using LoadAssembly, but this doesn't seem to be the case. > > Does anyone have examples of doing this that works? I think the problem here is that foo is not a PythonModule - it's a NamespaceTracker object instead. ImportModule can currently only support modules. This could be made to support namespace trackers - PythonService.ImportModule would need to be updated to check for a ReflectedNamespace and create a ScriptScope from it. From slide.o.mix at gmail.com Fri Oct 29 01:15:52 2010 From: slide.o.mix at gmail.com (Slide) Date: Thu, 28 Oct 2010 16:15:52 -0700 Subject: [IronPython] All classes that subclass a given type Message-ID: I am trying to find all the classes defined in Python that inherit from a C# class. I can iterate through the types in the ScriptScope using GetVariableName() and then checking if it's a PythonType object, but I don't know how to get the classes that it inherits from. Can anyone help out with this? Also, on a separate note, I would like to override the normal import mechanism to support new file extensions for modules (this is so I can associate the files with my application). Is this possible? Thanks, slide -- slide-o-blog http://slide-o-blog.blogspot.com/ From slide.o.mix at gmail.com Fri Oct 29 01:18:19 2010 From: slide.o.mix at gmail.com (Slide) Date: Thu, 28 Oct 2010 16:18:19 -0700 Subject: [IronPython] ScriptRuntime.ImportModule("foo") vs. import foo In-Reply-To: References: Message-ID: On Thu, Oct 28, 2010 at 4:14 PM, Dino Viehland wrote: > Slide wrote: >> I have something like the following code: >> >> ScriptEngine engine = Python.CreateEngine(); ScriptScope scope = >> engine.CreateScope(); >> >> engine.Runtime.LoadAssembly(Assembly.GetExecutingAssembly()); // holds >> the namespace foo >> >> // the following line throws an exception >> engine.Runtime.ImportModule("foo"); >> >> // the following if executed works fine >> engine.Execute("import foo"); >> >> >> I'm basically just trying to make the types in my executing assembly available >> to the script. I assumed that the ImportModule would see things that had >> been loaded using LoadAssembly, but this doesn't seem to be the case. >> >> Does anyone have examples of doing this that works? > > > I think the problem here is that foo is not a PythonModule - it's a NamespaceTracker > object instead. ?ImportModule can currently only support modules. ?This could be > made to support namespace trackers - PythonService.ImportModule would need > to be updated to check for a ReflectedNamespace and create a ScriptScope from it. > > Is it best to just Execute("import foo") to make it available then? Thanks, slide From dinov at microsoft.com Fri Oct 29 01:22:20 2010 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 28 Oct 2010 23:22:20 +0000 Subject: [IronPython] ScriptRuntime.ImportModule("foo") vs. import foo In-Reply-To: References: Message-ID: Slide wrote: > Is it best to just Execute("import foo") to make it available then? Yeah, probably - at least I don't see a better way right now. From tony.meyer at gmail.com Fri Oct 29 02:06:30 2010 From: tony.meyer at gmail.com (Tony Meyer) Date: Fri, 29 Oct 2010 13:06:30 +1300 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: > What this is going to look like in the future is an open question, as > is the timeline. Originally, I wanted to focus on the 2.7 release and > deal with the source control question later. However, it's been raised > in a few places, so I think it's better to get some more feedback on > whether we should switch source control methods (and if so, to what?) > or just stay on TFS/SVN for the time being. I think this is the type of decision that the "Coordinators" should make themselves based on their own experience/preferences. There's never going to be a consensus on which VCS is the best/most appropriate choice, and going with majority vote isn't necessarily the best idea either. It seems reasonable to assume that the "Coordinators" will be (at least at first) the most frequent contributors, so it has most impact on them as well. I think CPython has demonstrated that changing VCS, while not trivial, is an achievable task, and so if in a few year's time it seems like there is a better choice, there's no reason why a change cannot be made then. Cheers, Tony From noah.gift at gmail.com Fri Oct 29 02:15:07 2010 From: noah.gift at gmail.com (Noah Gift) Date: Thu, 28 Oct 2010 17:15:07 -0700 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: <4CC97BDF.3000800@voidspace.org.uk> References: <4CC97BDF.3000800@voidspace.org.uk> Message-ID: On Thu, Oct 28, 2010 at 6:34 AM, Michael Foord wrote: > On 28/10/2010 03:53, Steve Dower wrote: > >> I'll add in a vote for Mercurial (voting always seems to be how to >> decide on VCS), though I still believe that SVN works better for a >> contribution/review/patch workflow. >> > > Distributed version control systems are very good for distributed > development (funny that). Whilst I'm not proposing we use bzr (I would > *very* much like us to use mercurial though), our workflow at Canonical with > bzr and launchpad is great. You develop in a branch (branching is very easy > with dcvs') and push to launchpad. On requesting a merge review you get a > great web interface with viewable diff and when the merge is approved you > merge back onto trunk. (Branching and merging are substantially easier with > hg / bzr / git than with svn.) > > (From the peanut gallery). One potential benefit to having the project live at Canonical is that Michael works there, and Canonical may be interested in helping officially support it. Stranger things have happened..... -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel at novell.com Fri Oct 29 02:31:48 2010 From: miguel at novell.com (Miguel de Icaza) Date: Thu, 28 Oct 2010 20:31:48 -0400 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: Hello, We could, for example, drop the source control from CodePlex and just > use the IronRuby github repo - it's already set up and we could start > developing tomorrow This is my preference. The GitHub guys have great support for "Organizations", we use that for Mono, and it has nice administration tools for Organizations, allows administrations per module and have lots of hooks and web tools to explore, manage and collaborate as well as providing some 60 different popular hooks for services that can be offered. Moving to Git seems like a no brainer to me: we only have to move IronPython there. If we were to pick another of the open source source code management systems we would be moving both Ruby and Python away. If the concern is the UI for checking code out for Git, there is a transparent bridge that exposes the tree to Subversion which has good Windows clients. MIguel -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony.meyer at gmail.com Fri Oct 29 03:05:36 2010 From: tony.meyer at gmail.com (Tony Meyer) Date: Fri, 29 Oct 2010 14:05:36 +1300 Subject: [IronPython] Installing IronPython 2.7b1 over 2.7a1 Message-ID: Hi, Has anyone else had any issues with this? When I try to do it I get an error "A later version of IronPython 2.7 is already installed". However, the only version of IronPython 2.7 that is installed is alpha 1 ("2.7.0.1"); 2.6 ("2.6.10920.0") is also installed. I'm using the msi installer from codeplex, which says it is "IronPython 2.7 (2.7.0709)". I presume I can fix this by uninstalling a1 and then installing b1 - I haven't had a chance to try that yet. If no-one else has seen this, then I'm inclined to assume this is something odd with my machine. Otherwise I can open a bug. Thanks, Tony From tony.meyer at gmail.com Fri Oct 29 03:07:28 2010 From: tony.meyer at gmail.com (Tony Meyer) Date: Fri, 29 Oct 2010 14:07:28 +1300 Subject: [IronPython] The Road to 2.7 (Was: The Future of IronPython) In-Reply-To: References: Message-ID: > Right now, we need to identify what's not working in 2.7B1 that needs > to be in 2.7 final. The best thing to do would be to identify any > issues that are causing you pain and bring them up on the list. How does the Visual Studio integration fit in to this? Is it a separate sub-project, or just a regular part of IronPython now? IMO it's a really important part of IronPython, and unfortunately still very flawed (at least in a1 - I haven't had a chance to try b1 in depth yet). > Besides knowing what needs to be done, we need a rough timeline. I > would like to see a release before the end of the year, or at least a > solid release candidate with a possible release early next year. +1 for by the end of the year, assuming that's possible. Although 2.7 is the end of 2.x, there's no reason why there can't be reasonably regular IPy 2.7 releases (e.g. filling in standard library gaps). Cheers, Tony From dinov at microsoft.com Fri Oct 29 03:14:28 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 29 Oct 2010 01:14:28 +0000 Subject: [IronPython] Installing IronPython 2.7b1 over 2.7a1 In-Reply-To: References: Message-ID: Tony wrote: > Hi, > > Has anyone else had any issues with this? When I try to do it I get > an error "A later version of IronPython 2.7 is already installed". > However, the only version of IronPython 2.7 that is installed is alpha > 1 ("2.7.0.1"); 2.6 ("2.6.10920.0") is also installed. I'm using the > msi installer from codeplex, which says it is "IronPython 2.7 > (2.7.0709)". > > I presume I can fix this by uninstalling a1 and then installing b1 - I > haven't had a chance to try that yet. > > If no-one else has seen this, then I'm inclined to assume this is > something odd with my machine. Otherwise I can open a bug. I saw someone w/ this issue on twitter or stackoverflow or somewhere like that. Uninstalling A1 and installing B1 should work. Maybe "0709" is considered less than "1" due to the leading zero and we just need to be a little more careful in the future w/ the version numbers. From dinov at microsoft.com Fri Oct 29 03:15:58 2010 From: dinov at microsoft.com (Dino Viehland) Date: Fri, 29 Oct 2010 01:15:58 +0000 Subject: [IronPython] The Road to 2.7 (Was: The Future of IronPython) In-Reply-To: References: Message-ID: Tony wrote: > How does the Visual Studio integration fit in to this? Is it a > separate sub-project, or just a regular part of IronPython now? IMO > it's a really important part of IronPython, and unfortunately still > very flawed (at least in a1 - I haven't had a chance to try b1 in > depth yet). I'd think keeping the integration as part of IronPython and the Windows Installer is the way to go. B1 should be much improved - there were a bunch of bug fixes that went into it although there's still at least a couple of issues still lurking. From jdhardy at gmail.com Fri Oct 29 07:32:16 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Thu, 28 Oct 2010 23:32:16 -0600 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: <4CC97BDF.3000800@voidspace.org.uk> Message-ID: On Thu, Oct 28, 2010 at 12:16 PM, Vernon Cole wrote: > On Thu, Oct 28, 2010 at 7:34 AM, Michael Foord > wrote: >> On 28/10/2010 03:53, Steve Dower wrote: >>> Is the plan after 2.7 to start doing 3? That seems like a good >>> opportunity to "start fresh" in a new repository and leave the old >>> stuff where it is, only carrying over the barest minimum. I can't see >>> any movement before 2.7 as being worthwhile. >> >> Interesting question. Ideally we would do parallel development but I'm not >> sure we have the resources for that. > > Python 2.7 is documented to be the LAST of its family. There should not be > very much "development", except perhaps filling out the standard library. > I would say to put 3.n on a fresh hg tree, and back-port anything necessary > into the existing 2.7 tree and infrastructure. No sense in re-inventing that > wheel. I don't know if I want to lose the history completely, but I'm not sure how much value it has either. Other than that, I'm liking this idea a lot. IronPython 3.2 will be a completely new repo, while 2.7 and below will stay in a legacy one. I don't see there being a ton of major changes to 2.7 anyway, and I doubt it will be around as long as CPython 2.7. IronPython 3 is going to be far more compatible with Python 3 than the 2.x series is. - Jeff From jdhardy at gmail.com Fri Oct 29 07:37:00 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Thu, 28 Oct 2010 23:37:00 -0600 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: On Thu, Oct 28, 2010 at 6:06 PM, Tony Meyer wrote: > I think this is the type of decision that the "Coordinators" should > make themselves based on their own experience/preferences. ?There's > never going to be a consensus on which VCS is the best/most > appropriate choice, and going with majority vote isn't necessarily the > best idea either. I agree completely, but as a benevolent oligarchy, it behooves us to find out what the rest of the community is thinking. Doing the opposite of what the community wants would be a great way to destroy it before we even get started. - Jeff From tony.meyer at gmail.com Fri Oct 29 07:39:39 2010 From: tony.meyer at gmail.com (Tony Meyer) Date: Fri, 29 Oct 2010 18:39:39 +1300 Subject: [IronPython] Installing IronPython 2.7b1 over 2.7a1 In-Reply-To: References: Message-ID: > I saw someone w/ this issue on twitter or stackoverflow or somewhere like > that. ?Uninstalling A1 and installing B1 should work. ?Maybe "0709" is > considered less than "1" due to the leading zero and we just need to be > a little more careful in the future w/ the version numbers. Ok. Sounds like there's no need to open a ticket then. Cheers, Tony From jdhardy at gmail.com Fri Oct 29 08:36:19 2010 From: jdhardy at gmail.com (Jeff Hardy) Date: Fri, 29 Oct 2010 00:36:19 -0600 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: On Thu, Oct 28, 2010 at 6:31 PM, Miguel de Icaza wrote: > Moving to Git seems like a no brainer to me: we only have to move IronPython > there. ? If we were to pick another of the open source source code > management systems we would be moving both Ruby and Python away. Where I'm torn is that IronPython is between a rock and a hard place - on the one hand, I want to work closely with the Mono project (which would strongly imply using github), but I also want to be in sync with the Python community (which has largely embraced hg). Python's separated stdlib will be a Mercurial repo, and if IronPython were in Hg we could easily pull in the stdlib as a subrepo. However, I presume the DLR will stay on github, and we'll need to pull that in as well, which of course would be easier from git. Damned if we do, damned if we don't. I'm leaning towards Mercurial because I prefer it, and it seems like many other people do as well. I know hg can pull from git, but I don't know about the reverse. > If the concern is the UI for checking code out for Git, there is a > transparent bridge that exposes the tree to Subversion which has good > Windows clients. Tortoise-Git is actually very nice now. I don't think git's Windows support is really an issue anymore. - Jeff From niki at vintech.bg Fri Oct 29 09:53:02 2010 From: niki at vintech.bg (niki) Date: Fri, 29 Oct 2010 10:53:02 +0300 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: <4CCA7D5E.8010001@vintech.bg> On 29.10.2010 09:36, Jeff Hardy wrote: > On Thu, Oct 28, 2010 at 6:31 PM, Miguel de Icaza wrote: >> Moving to Git seems like a no brainer to me: we only have to move IronPython >> there. If we were to pick another of the open source source code >> management systems we would be moving both Ruby and Python away. > > Where I'm torn is that IronPython is between a rock and a hard place - > on the one hand, I want to work closely with the Mono project (which > would strongly imply using github), but I also want to be in sync with > the Python community (which has largely embraced hg). > > Python's separated stdlib will be a Mercurial repo, and if IronPython > were in Hg we could easily pull in the stdlib as a subrepo. However, I > presume the DLR will stay on github, and we'll need to pull that in as > well, which of course would be easier from git. > > Damned if we do, damned if we don't. I'm leaning towards Mercurial > because I prefer it, and it seems like many other people do as well. I > know hg can pull from git, but I don't know about the reverse. IIRC mercurial has git support. Niki From noah.gift at gmail.com Fri Oct 29 14:10:44 2010 From: noah.gift at gmail.com (Noah Gift) Date: Fri, 29 Oct 2010 05:10:44 -0700 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: On Thu, Oct 28, 2010 at 11:36 PM, Jeff Hardy wrote: > On Thu, Oct 28, 2010 at 6:31 PM, Miguel de Icaza > wrote: > > Moving to Git seems like a no brainer to me: we only have to move > IronPython > > there. If we were to pick another of the open source source code > > management systems we would be moving both Ruby and Python away. > > Where I'm torn is that IronPython is between a rock and a hard place - > on the one hand, I want to work closely with the Mono project (which > would strongly imply using github), but I also want to be in sync with > the Python community (which has largely embraced hg). > > Python's separated stdlib will be a Mercurial repo, and if IronPython > were in Hg we could easily pull in the stdlib as a subrepo. However, I > presume the DLR will stay on github, and we'll need to pull that in as > well, which of course would be easier from git. > > Damned if we do, damned if we don't. I'm leaning towards Mercurial > because I prefer it, and it seems like many other people do as well. I > know hg can pull from git, but I don't know about the reverse. > > > If the concern is the UI for checking code out for Git, there is a > > transparent bridge that exposes the tree to Subversion which has good > > Windows clients. > > Tortoise-Git is actually very nice now. I don't think git's Windows > support is really an issue anymore. > > Haven't tried it, but this looks interesting: http://www.serpentine.com/blog/2010/10/10/dual-bitbucketgithub-citizenship/ - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > -- Thanks, Noah -------------- next part -------------- An HTML attachment was scrubbed... URL: From Larry.Jones at aspentech.com Fri Oct 29 15:11:00 2010 From: Larry.Jones at aspentech.com (Jones, Larry) Date: Fri, 29 Oct 2010 09:11:00 -0400 Subject: [IronPython] Installing IronPython 2.7b1 over 2.7a1 In-Reply-To: References: Message-ID: I saw the same behavior. I assumed that it was because I was installing a beta version atop an alpha version. Uninstalling the alpha version and re-installing the beta version worked without a hitch. -----Original Message----- From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Dino Viehland Sent: Thursday, October 28, 2010 8:14 PM To: Discussion of IronPython Subject: Re: [IronPython] Installing IronPython 2.7b1 over 2.7a1 Tony wrote: > Hi, > > Has anyone else had any issues with this? When I try to do it I get > an error "A later version of IronPython 2.7 is already installed". > However, the only version of IronPython 2.7 that is installed is alpha > 1 ("2.7.0.1"); 2.6 ("2.6.10920.0") is also installed. I'm using the > msi installer from codeplex, which says it is "IronPython 2.7 > (2.7.0709)". > > I presume I can fix this by uninstalling a1 and then installing b1 - I > haven't had a chance to try that yet. > > If no-one else has seen this, then I'm inclined to assume this is > something odd with my machine. Otherwise I can open a bug. I saw someone w/ this issue on twitter or stackoverflow or somewhere like that. Uninstalling A1 and installing B1 should work. Maybe "0709" is considered less than "1" due to the leading zero and we just need to be a little more careful in the future w/ the version numbers. _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify the sender and permanently delete the original copy and any copy of any e-mail, and any printout thereof. From tristanz at gmail.com Fri Oct 29 20:54:57 2010 From: tristanz at gmail.com (Tristan Zajonc) Date: Fri, 29 Oct 2010 14:54:57 -0400 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: I've used both hg+bitbucket and git+github. In my experience, there is very little difference between hg and git in terms of workflow. I found both to be great and the tools are pretty mature across all platforms. I do think GitHub is rapidly becoming a killer application for open source projects. If you look at the IronRuby repository (http://github.com/ironruby/ironruby), there are already 340 watchers and 109 forks, which is a non-negligible consideration imho. On Fri, Oct 29, 2010 at 8:10 AM, Noah Gift wrote: > > > On Thu, Oct 28, 2010 at 11:36 PM, Jeff Hardy wrote: > >> On Thu, Oct 28, 2010 at 6:31 PM, Miguel de Icaza >> wrote: >> > Moving to Git seems like a no brainer to me: we only have to move >> IronPython >> > there. If we were to pick another of the open source source code >> > management systems we would be moving both Ruby and Python away. >> >> Where I'm torn is that IronPython is between a rock and a hard place - >> on the one hand, I want to work closely with the Mono project (which >> would strongly imply using github), but I also want to be in sync with >> the Python community (which has largely embraced hg). >> >> Python's separated stdlib will be a Mercurial repo, and if IronPython >> were in Hg we could easily pull in the stdlib as a subrepo. However, I >> presume the DLR will stay on github, and we'll need to pull that in as >> well, which of course would be easier from git. >> >> Damned if we do, damned if we don't. I'm leaning towards Mercurial >> because I prefer it, and it seems like many other people do as well. I >> know hg can pull from git, but I don't know about the reverse. >> >> > If the concern is the UI for checking code out for Git, there is a >> > transparent bridge that exposes the tree to Subversion which has good >> > Windows clients. >> >> Tortoise-Git is actually very nice now. I don't think git's Windows >> support is really an issue anymore. >> >> Haven't tried it, but this looks interesting: > > http://www.serpentine.com/blog/2010/10/10/dual-bitbucketgithub-citizenship/ > > > - Jeff >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> > > > > -- > Thanks, > > Noah > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From s.j.dower at gmail.com Sat Oct 30 01:46:00 2010 From: s.j.dower at gmail.com (Steve Dower) Date: Sat, 30 Oct 2010 10:46:00 +1100 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: Are IronPython and the DLR so closely coupled that you *need* the source for both to work on it? Or can you simply develop/test IronPython using the DLR in the GAC? I'd rather have the standard library as a 'default' part of the IronPython checkout than the DLR, primarily because a binary distro of the DLR makes more sense. On Sat, Oct 30, 2010 at 05:54, Tristan Zajonc wrote: > I've used both hg+bitbucket and git+github. ?In my experience, there is very > little difference between hg and git in terms of workflow. ?I found both to > be great and the tools are pretty mature across all platforms. ?I do think > GitHub is rapidly becoming a killer application for open source projects. > ?If you look at the IronRuby repository > (http://github.com/ironruby/ironruby), there are already 340 watchers and > 109 forks, which is a non-negligible consideration imho. > > On Fri, Oct 29, 2010 at 8:10 AM, Noah Gift wrote: >> >> >> On Thu, Oct 28, 2010 at 11:36 PM, Jeff Hardy wrote: >>> >>> On Thu, Oct 28, 2010 at 6:31 PM, Miguel de Icaza >>> wrote: >>> > Moving to Git seems like a no brainer to me: we only have to move >>> > IronPython >>> > there. ? If we were to pick another of the open source source code >>> > management systems we would be moving both Ruby and Python away. >>> >>> Where I'm torn is that IronPython is between a rock and a hard place - >>> on the one hand, I want to work closely with the Mono project (which >>> would strongly imply using github), but I also want to be in sync with >>> the Python community (which has largely embraced hg). >>> >>> Python's separated stdlib will be a Mercurial repo, and if IronPython >>> were in Hg we could easily pull in the stdlib as a subrepo. However, I >>> presume the DLR will stay on github, and we'll need to pull that in as >>> well, which of course would be easier from git. >>> >>> Damned if we do, damned if we don't. I'm leaning towards Mercurial >>> because I prefer it, and it seems like many other people do as well. I >>> know hg can pull from git, but I don't know about the reverse. >>> >>> > If the concern is the UI for checking code out for Git, there is a >>> > transparent bridge that exposes the tree to Subversion which has good >>> > Windows clients. >>> >>> Tortoise-Git is actually very nice now. I don't think git's Windows >>> support is really an issue anymore. >>> >> Haven't tried it, but this looks interesting: >> >> http://www.serpentine.com/blog/2010/10/10/dual-bitbucketgithub-citizenship/ >>> >>> - Jeff >>> _______________________________________________ >>> Users mailing list >>> Users at lists.ironpython.com >>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> >> >> -- >> Thanks, >> >> Noah >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > From fuzzyman at voidspace.org.uk Sat Oct 30 03:26:29 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 29 Oct 2010 21:26:29 -0400 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: Message-ID: <4CCB7445.3090507@voidspace.org.uk> On 29/10/2010 19:46, Steve Dower wrote: > Are IronPython and the DLR so closely coupled that you *need* the > source for both to work on it? Or can you simply develop/test > IronPython using the DLR in the GAC? In general (I believe) the DLR is an integral part of both the IronPython and the IronRuby projects. Switching to a single repository would be convenient because of this. Having to use git, even indirectly, is a high price to pay though. :-) All the best, Michael > I'd rather have the standard library as a 'default' part of the > IronPython checkout than the DLR, primarily because a binary distro of > the DLR makes more sense. > > On Sat, Oct 30, 2010 at 05:54, Tristan Zajonc wrote: >> I've used both hg+bitbucket and git+github. In my experience, there is very >> little difference between hg and git in terms of workflow. I found both to >> be great and the tools are pretty mature across all platforms. I do think >> GitHub is rapidly becoming a killer application for open source projects. >> If you look at the IronRuby repository >> (http://github.com/ironruby/ironruby), there are already 340 watchers and >> 109 forks, which is a non-negligible consideration imho. >> >> On Fri, Oct 29, 2010 at 8:10 AM, Noah Gift wrote: >>> >>> On Thu, Oct 28, 2010 at 11:36 PM, Jeff Hardy wrote: >>>> On Thu, Oct 28, 2010 at 6:31 PM, Miguel de Icaza >>>> wrote: >>>>> Moving to Git seems like a no brainer to me: we only have to move >>>>> IronPython >>>>> there. If we were to pick another of the open source source code >>>>> management systems we would be moving both Ruby and Python away. >>>> Where I'm torn is that IronPython is between a rock and a hard place - >>>> on the one hand, I want to work closely with the Mono project (which >>>> would strongly imply using github), but I also want to be in sync with >>>> the Python community (which has largely embraced hg). >>>> >>>> Python's separated stdlib will be a Mercurial repo, and if IronPython >>>> were in Hg we could easily pull in the stdlib as a subrepo. However, I >>>> presume the DLR will stay on github, and we'll need to pull that in as >>>> well, which of course would be easier from git. >>>> >>>> Damned if we do, damned if we don't. I'm leaning towards Mercurial >>>> because I prefer it, and it seems like many other people do as well. I >>>> know hg can pull from git, but I don't know about the reverse. >>>> >>>>> If the concern is the UI for checking code out for Git, there is a >>>>> transparent bridge that exposes the tree to Subversion which has good >>>>> Windows clients. >>>> Tortoise-Git is actually very nice now. I don't think git's Windows >>>> support is really an issue anymore. >>>> >>> Haven't tried it, but this looks interesting: >>> >>> http://www.serpentine.com/blog/2010/10/10/dual-bitbucketgithub-citizenship/ >>>> - Jeff >>>> _______________________________________________ >>>> Users mailing list >>>> Users at lists.ironpython.com >>>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >>> >>> >>> -- >>> Thanks, >>> >>> Noah >>> >>> _______________________________________________ >>> Users mailing list >>> Users at lists.ironpython.com >>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >>> >> >> _______________________________________________ >> Users mailing list >> Users at lists.ironpython.com >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/ From fuzzyman at voidspace.org.uk Sat Oct 30 03:32:27 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 29 Oct 2010 21:32:27 -0400 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: <4CC97BDF.3000800@voidspace.org.uk> Message-ID: <4CCB75AB.8000605@voidspace.org.uk> On 29/10/2010 01:32, Jeff Hardy wrote: > On Thu, Oct 28, 2010 at 12:16 PM, Vernon Cole wrote: >> On Thu, Oct 28, 2010 at 7:34 AM, Michael Foord >> wrote: >>> On 28/10/2010 03:53, Steve Dower wrote: >>>> Is the plan after 2.7 to start doing 3? That seems like a good >>>> opportunity to "start fresh" in a new repository and leave the old >>>> stuff where it is, only carrying over the barest minimum. I can't see >>>> any movement before 2.7 as being worthwhile. >>> Interesting question. Ideally we would do parallel development but I'm not >>> sure we have the resources for that. >> Python 2.7 is documented to be the LAST of its family. There should not be >> very much "development", except perhaps filling out the standard library. >> I would say to put 3.n on a fresh hg tree, and back-port anything necessary >> into the existing 2.7 tree and infrastructure. No sense in re-inventing that >> wheel. > I don't know if I want to lose the history completely, but I'm not > sure how much value it has either. I'm not sure that losing the history from the repo is such a big loss - so long as the original repo can remain available as a read only resource the information isn't "lost", just available elsewhere. There will be no more features in Python 2.7, not even in the standard library, now that it is in bugfix only maintenance. All the best, Michael > Other than that, I'm liking this > idea a lot. IronPython 3.2 will be a completely new repo, while 2.7 > and below will stay in a legacy one. > > I don't see there being a ton of major changes to 2.7 anyway, and I > doubt it will be around as long as CPython 2.7. IronPython 3 is going > to be far more compatible with Python 3 than the 2.x series is. > > - Jeff > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/ From fuzzyman at voidspace.org.uk Sat Oct 30 03:33:21 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 29 Oct 2010 21:33:21 -0400 Subject: [IronPython] The elephant in the room: source control for IronPython In-Reply-To: References: <4CC97BDF.3000800@voidspace.org.uk> Message-ID: <4CCB75E1.9030904@voidspace.org.uk> On 28/10/2010 20:15, Noah Gift wrote: > > > On Thu, Oct 28, 2010 at 6:34 AM, Michael Foord > > wrote: > > On 28/10/2010 03:53, Steve Dower wrote: > > I'll add in a vote for Mercurial (voting always seems to be how to > decide on VCS), though I still believe that SVN works better for a > contribution/review/patch workflow. > > > Distributed version control systems are very good for distributed > development (funny that). Whilst I'm not proposing we use bzr (I > would *very* much like us to use mercurial though), our workflow > at Canonical with bzr and launchpad is great. You develop in a > branch (branching is very easy with dcvs') and push to launchpad. > On requesting a merge review you get a great web interface with > viewable diff and when the merge is approved you merge back onto > trunk. (Branching and merging are substantially easier with hg / > bzr / git than with svn.) > > > (From the peanut gallery). One potential benefit to having the project > live at Canonical is that Michael works there, and Canonical may be > interested in helping officially support it. Stranger things have > happened..... Hehe, I certainly can't speak for canonical on that score... :-) All the best, Michael > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Sat Oct 30 03:37:14 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 29 Oct 2010 21:37:14 -0400 Subject: [IronPython] All classes that subclass a given type In-Reply-To: References: Message-ID: <4CCB76CA.4090903@voidspace.org.uk> On 28/10/2010 19:15, Slide wrote: > I am trying to find all the classes defined in Python that inherit > from a C# class. I can iterate through the types in the ScriptScope > using GetVariableName() and then checking if it's a PythonType object, > but I don't know how to get the classes that it inherits from. Can > anyone help out with this? > In normal Python you can call Class.__subclasses__() to get a list of subclasses. I haven't tried this with .NET classes on IronPython though. > Also, on a separate note, I would like to override the normal import > mechanism to support new file extensions for modules (this is so I can > associate the files with my application). Is this possible? You can either patch __builtins__.__import__, use the ihooks module, implement a PEP 302 import loader *or* use the IronPython Platform Application Layer to control how IronPython imports are done (this is how IronPython on Silverlight does imports from the xap file). All the best, Michael Foord > Thanks, > > slide > -- http://www.voidspace.org.uk/