From no_reply at codeplex.com  Mon Feb  3 09:22:22 2014
From: no_reply at codeplex.com (CodePlex)
Date: 3 Feb 2014 00:22:22 -0800
Subject: [Ironpython-users] IronPython, Daily Digest 2/2/2014
Message-ID: <CO1MSDNADMCDS2mF1IT00011fd5@discussions.codeplex.com>

Hi ironpython,

Here's your Daily Digest of new issues for project "IronPython".

In today's digest:ISSUES

1. [New issue] reading a file using codecs can fail
2. [New comment] reading a file using codecs can fail

----------------------------------------------

ISSUES

1. [New issue] reading a file using codecs can fail
http://ironpython.codeplex.com/workitem/34951
User paweljasinski has proposed the issue:

"the following will fail given attached file:
import codecs

lines = []
with codecs.open("text-utf8-with-bom.txt", encoding="utf-8-sig") as file_obj:
    for line in file_obj: # fails here
        lines.append(line)

exception:
Traceback (most recent call last):
  File "C:\Program Files (x86)\IronPython 2.7\Lib\encodings\utf_8_sig.py", line 100, in decode
  File "decode-bug.py", line 5, in <module>
  File "C:\Program Files (x86)\IronPython 2.7\Lib\codecs.py", line 684, in next
  File "C:\Program Files (x86)\IronPython 2.7\Lib\codecs.py", line 615, in next
  File "C:\Program Files (x86)\IronPython 2.7\Lib\codecs.py", line 530, in readline
  File "C:\Program Files (x86)\IronPython 2.7\Lib\codecs.py", line 477, in read
UnicodeEncodeError: ('unknown', '\x00', 0, 1, 'failed to decode bytes at index 65')

It will not fail on linux/cpython.
Removing one character from the first line of text-utf8-with-bom.txt file will make it work."-----------------

2. [New comment] reading a file using codecs can fail
http://ironpython.codeplex.com/workitem/34951
User paweljasinski has commented on the issue:

"<p>a better test:<br>```<br>import codecs</p><p>f=open(&quot;text-utf8-with-bom.txt&quot;, &quot;rb&quot;)<br>b=f.read()<br>codecs.utf_8_decode(b)</p><p># remove last character of the file (0x0d)<br>b=b[:-1]<br>print codecs.utf_8_decode(b)[1]</p><p># remove last character of euro code (0x0c)<br>b=b[:-1]<br>print codecs.utf_8_decode(b)[1]</p><p>```<br>produces:<br>```<br>70<br>Traceback (most recent call last):<br>  File &quot;d2_test.py&quot;, line 13, in &lt;module&gt;<br>UnicodeEncodeError: ('unknown', '\x00', 0, 1, 'failed to decode bytes at index 65')<br>```<br>where on linux/cpython it is:<br>73<br>70</p><p></p>"
----------------------------------------------



----------------------------------------------
You are receiving this email because you subscribed to notifications on CodePlex.

To report a bug, request a feature, or add a comment, visit IronPython Issue Tracker. You can unsubscribe or change your issue notification settings on CodePlex.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140203/023889d3/attachment.html>

From m.schaber at codesys.com  Tue Feb  4 09:21:55 2014
From: m.schaber at codesys.com (Markus Schaber)
Date: Tue, 4 Feb 2014 08:21:55 +0000
Subject: [Ironpython-users] Splitting up the IronPython repo
In-Reply-To: <CAF7AXFEEQCqdkVRemgZA2m2FA8b=XmvPBj-Wovh75Z_FyznkiQ@mail.gmail.com>
References: <CAF7AXFEEQCqdkVRemgZA2m2FA8b=XmvPBj-Wovh75Z_FyznkiQ@mail.gmail.com>
Message-ID: <727D8E16AE957149B447FE368139F2B5398580BF@SERVER10>

Hi, Jeff,

I tried to open the IronPython solution file  using Visual Studio 2010 Pro and 2012 Express for Desktop. Both failed to open the IronPython, IronPython and IronPython.SQLite projects with the following error messages:

D:\Contribs\Ironpython\ironpython-only\Src\IronPython\IronPython.csproj : error  : Unable to read the project file 'IronPython.csproj'. 
D:\Contribs\Ironpython\ironpython-only\Src\IronPython\IronPython.csproj(516,3): The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero length.

D:\Contribs\Ironpython\ironpython-only\Src\IronPython.Modules\IronPython.Modules.csproj : error  : Unable to read the project file 'IronPython.Modules.csproj'. 
D:\Contribs\Ironpython\ironpython-only\Src\IronPython.Modules\IronPython.Modules.csproj(149,3): The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero length.

D:\Contribs\Ironpython\ironpython-only\Src\IronPython.SQLite\IronPython.SQLite.csproj : error  : Unable to read the project file 'IronPython.SQLite.csproj'. 
D:\Contribs\Ironpython\ironpython-only\Src\IronPython.SQLite\IronPython.SQLite.csproj(160,3): The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero length.

Some of the properties associated with the solution could not be read.

Best regards

Markus Schaber

CODESYS? a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions

3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915

> -----Urspr?ngliche Nachricht-----
> Von: Ironpython-users [mailto:ironpython-users-
> bounces+m.schaber=codesys.com at python.org] Im Auftrag von Jeff Hardy
> Gesendet: Montag, 20. Januar 2014 10:10
> An: ironpython-users at python.org
> Betreff: [Ironpython-users] Splitting up the IronPython repo
> 
> One of the big issues with working on IronPython is the size of the git
> repository (specifically https://github.com/IronLanguages/main) - git does
> not like really big repos, especially on Windows. Part of the problem is that
> the repository includes:
> 
> * The DLR
> * IronPython
> * IronRuby
> * Two copies of the Python stdlib
> * The Ruby stdlib
> * WiX
> * and a bunch of reference assemblies
> 
> Even on a fast machine, 'git status' takes several seconds to return.
> I believe this is because it was originally a TFS repo, which can scale to
> handle bigger repos by using a bigger server. With git that option doesn't
> exist - if the repo is too big, the only option is to split it up.
> 
> I've created two repos - https://github.com/jdhardy/dlr and
> https://github.com/jdhardy/ironpython-only - that contain just the DLR and
> IronPython, respectively. In them, git calls are nearly instantaneous, which
> makes working with it a lot less painful.
> 
> There are other advantages - the DLR can get its own release cycle and
> packaging, and IP can then depend on a specific version of the DLR.
> Each project has a modified version of the IronPython build system that makes
> it easy to build for other platforms (iOS, Android, Win8, etc. - they still
> need to ported and tested, but the builds are easier).
> 
> I did most of the work using Mono/xbuild, so I know it works there (except
> for a bug in Mono's .NET 4.5 support), but it has some errors on Windows that
> I need to sort out.
> 
> Once I get my Windows box back and get some time to fix the few remaining
> issues, I'll move the repos to the IronLanguages account and use them for
> development of a real DLR release and IronPython 3.0. Any more 2.7 releases
> will come out of the existing repo.
> 
> One downside is that copying patches between 3.0 and 2.7 is going to be extra
> work, but 3.0 will solve so many problems with strings that I think it will
> quickly become the more common target. In general Python
> 3 momentum is picking up so it's a good time (some recent hand-wringing
> notwithstanding) to try and have IronPython 3 in the right spot at the right
> time.
> 
> I haven't really addressed IronRuby because, for intents and purposes, it's
> dead.
> 
> All of that said, if anyone has any objections I'd like to hear them.
> The split repos work well for me but I'm curious if others prefer the
> combined repo.
> 
> - Jeff
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users

From jdhardy at gmail.com  Tue Feb  4 10:01:13 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Tue, 4 Feb 2014 09:01:13 +0000
Subject: [Ironpython-users] Splitting up the IronPython repo
In-Reply-To: <727D8E16AE957149B447FE368139F2B5398580BF@SERVER10>
References: <CAF7AXFEEQCqdkVRemgZA2m2FA8b=XmvPBj-Wovh75Z_FyznkiQ@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B5398580BF@SERVER10>
Message-ID: <CAF7AXFFheby5K-F_A7Pbi1mt7QxLP4Gow1M=-S8vqp=5m=0yyQ@mail.gmail.com>

I did most of the work on my MacBook, just running MSBuild from the
command line; it might still need some tweaking to work from VS. The
main thing is making sure all of the properties are set when just
loading the projects (not using Build.proj as a wrapper). If you look
in the .csproj file, it tries to populate a property (the filename of
the platform-specific properties) and then import it; my guess is that
for some reason that is not getting set properly and staying at ''.

Unfortunately VS and MSBuild are not exactly the same; the environment
and pre-set properties are different depending on whether it is run
from the command line or VS. I do remember opening it in VS 2013
though, to make sure that it worked, but it's possible I changed
something since then or that 2012 and 2013 are slightly different. I
don't want to require 2013, though. (I'd also like to open cleanly in
MonoDevelop/Xamarin Studio, but it tends to mangle the csproj files in
unfortunate ways.)

- Jeff

On Tue, Feb 4, 2014 at 8:21 AM, Markus Schaber <m.schaber at codesys.com> wrote:
> Hi, Jeff,
>
> I tried to open the IronPython solution file  using Visual Studio 2010 Pro and 2012 Express for Desktop. Both failed to open the IronPython, IronPython and IronPython.SQLite projects with the following error messages:
>
> D:\Contribs\Ironpython\ironpython-only\Src\IronPython\IronPython.csproj : error  : Unable to read the project file 'IronPython.csproj'.
> D:\Contribs\Ironpython\ironpython-only\Src\IronPython\IronPython.csproj(516,3): The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero length.
>
> D:\Contribs\Ironpython\ironpython-only\Src\IronPython.Modules\IronPython.Modules.csproj : error  : Unable to read the project file 'IronPython.Modules.csproj'.
> D:\Contribs\Ironpython\ironpython-only\Src\IronPython.Modules\IronPython.Modules.csproj(149,3): The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero length.
>
> D:\Contribs\Ironpython\ironpython-only\Src\IronPython.SQLite\IronPython.SQLite.csproj : error  : Unable to read the project file 'IronPython.SQLite.csproj'.
> D:\Contribs\Ironpython\ironpython-only\Src\IronPython.SQLite\IronPython.SQLite.csproj(160,3): The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero length.
>
> Some of the properties associated with the solution could not be read.
>
> Best regards
>
> Markus Schaber
>
> CODESYS? a trademark of 3S-Smart Software Solutions GmbH
>
> Inspiring Automation Solutions
>
> 3S-Smart Software Solutions GmbH
> Dipl.-Inf. Markus Schaber | Product Development Core Technology
> Memminger Str. 151 | 87439 Kempten | Germany
> Tel. +49-831-54031-979 | Fax +49-831-54031-50
>
> E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
> CODESYS forum: http://forum.codesys.com
>
> Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
>
>> -----Urspr?ngliche Nachricht-----
>> Von: Ironpython-users [mailto:ironpython-users-
>> bounces+m.schaber=codesys.com at python.org] Im Auftrag von Jeff Hardy
>> Gesendet: Montag, 20. Januar 2014 10:10
>> An: ironpython-users at python.org
>> Betreff: [Ironpython-users] Splitting up the IronPython repo
>>
>> One of the big issues with working on IronPython is the size of the git
>> repository (specifically https://github.com/IronLanguages/main) - git does
>> not like really big repos, especially on Windows. Part of the problem is that
>> the repository includes:
>>
>> * The DLR
>> * IronPython
>> * IronRuby
>> * Two copies of the Python stdlib
>> * The Ruby stdlib
>> * WiX
>> * and a bunch of reference assemblies
>>
>> Even on a fast machine, 'git status' takes several seconds to return.
>> I believe this is because it was originally a TFS repo, which can scale to
>> handle bigger repos by using a bigger server. With git that option doesn't
>> exist - if the repo is too big, the only option is to split it up.
>>
>> I've created two repos - https://github.com/jdhardy/dlr and
>> https://github.com/jdhardy/ironpython-only - that contain just the DLR and
>> IronPython, respectively. In them, git calls are nearly instantaneous, which
>> makes working with it a lot less painful.
>>
>> There are other advantages - the DLR can get its own release cycle and
>> packaging, and IP can then depend on a specific version of the DLR.
>> Each project has a modified version of the IronPython build system that makes
>> it easy to build for other platforms (iOS, Android, Win8, etc. - they still
>> need to ported and tested, but the builds are easier).
>>
>> I did most of the work using Mono/xbuild, so I know it works there (except
>> for a bug in Mono's .NET 4.5 support), but it has some errors on Windows that
>> I need to sort out.
>>
>> Once I get my Windows box back and get some time to fix the few remaining
>> issues, I'll move the repos to the IronLanguages account and use them for
>> development of a real DLR release and IronPython 3.0. Any more 2.7 releases
>> will come out of the existing repo.
>>
>> One downside is that copying patches between 3.0 and 2.7 is going to be extra
>> work, but 3.0 will solve so many problems with strings that I think it will
>> quickly become the more common target. In general Python
>> 3 momentum is picking up so it's a good time (some recent hand-wringing
>> notwithstanding) to try and have IronPython 3 in the right spot at the right
>> time.
>>
>> I haven't really addressed IronRuby because, for intents and purposes, it's
>> dead.
>>
>> All of that said, if anyone has any objections I'd like to hear them.
>> The split repos work well for me but I'm curious if others prefer the
>> combined repo.
>>
>> - Jeff
>> _______________________________________________
>> Ironpython-users mailing list
>> Ironpython-users at python.org
>> https://mail.python.org/mailman/listinfo/ironpython-users

From m.schaber at codesys.com  Tue Feb  4 11:40:59 2014
From: m.schaber at codesys.com (Markus Schaber)
Date: Tue, 4 Feb 2014 10:40:59 +0000
Subject: [Ironpython-users] Splitting up the IronPython repo
In-Reply-To: <CAF7AXFFheby5K-F_A7Pbi1mt7QxLP4Gow1M=-S8vqp=5m=0yyQ@mail.gmail.com>
References: <CAF7AXFEEQCqdkVRemgZA2m2FA8b=XmvPBj-Wovh75Z_FyznkiQ@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B5398580BF@SERVER10>
 <CAF7AXFFheby5K-F_A7Pbi1mt7QxLP4Gow1M=-S8vqp=5m=0yyQ@mail.gmail.com>
Message-ID: <727D8E16AE957149B447FE368139F2B53985817F@SERVER10>

Hi, Jeff,

Okay, I'll try to have a look at it later.

I've got mixed results with MonoDevelop/Xamarin Studio in the past, including crashes while trying to load "normal" solution files (which work fine with VS, VS Express, MSBuild, VisualBuild and SharpDevelop)...

Best regards

Markus Schaber

CODESYS? a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions

3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915

> -----Urspr?ngliche Nachricht-----
> Von: Jeff Hardy [mailto:jdhardy at gmail.com]
> Gesendet: Dienstag, 4. Februar 2014 10:01
> An: Markus Schaber
> Cc: ironpython-users at python.org
> Betreff: Re: [Ironpython-users] Splitting up the IronPython repo
> 
> I did most of the work on my MacBook, just running MSBuild from the command
> line; it might still need some tweaking to work from VS. The main thing is
> making sure all of the properties are set when just loading the projects (not
> using Build.proj as a wrapper). If you look in the .csproj file, it tries to
> populate a property (the filename of the platform-specific properties) and
> then import it; my guess is that for some reason that is not getting set
> properly and staying at ''.
> 
> Unfortunately VS and MSBuild are not exactly the same; the environment and
> pre-set properties are different depending on whether it is run from the
> command line or VS. I do remember opening it in VS 2013 though, to make sure
> that it worked, but it's possible I changed something since then or that 2012
> and 2013 are slightly different. I don't want to require 2013, though. (I'd
> also like to open cleanly in MonoDevelop/Xamarin Studio, but it tends to
> mangle the csproj files in unfortunate ways.)
> 
> - Jeff
> 
> On Tue, Feb 4, 2014 at 8:21 AM, Markus Schaber <m.schaber at codesys.com> wrote:
> > Hi, Jeff,
> >
> > I tried to open the IronPython solution file  using Visual Studio 2010 Pro
> and 2012 Express for Desktop. Both failed to open the IronPython, IronPython
> and IronPython.SQLite projects with the following error messages:
> >
> > D:\Contribs\Ironpython\ironpython-only\Src\IronPython\IronPython.csproj :
> error  : Unable to read the project file 'IronPython.csproj'.
> > D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython\IronPython.csproj(516,3): The value "" of the "Project"
> attribute in element <Import> is invalid. Parameter "path" cannot have zero
> length.
> >
> > D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython.Modules\IronPython.Modules.csproj : error  : Unable to
> read the project file 'IronPython.Modules.csproj'.
> > D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython.Modules\IronPython.Modules.csproj(149,3): The value "" of
> the "Project" attribute in element <Import> is invalid. Parameter "path"
> cannot have zero length.
> >
> > D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython.SQLite\IronPython.SQLite.csproj : error  : Unable to read
> the project file 'IronPython.SQLite.csproj'.
> > D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython.SQLite\IronPython.SQLite.csproj(160,3): The value "" of
> the "Project" attribute in element <Import> is invalid. Parameter "path"
> cannot have zero length.
> >
> > Some of the properties associated with the solution could not be read.
> >
> > Best regards
> >
> > Markus Schaber
> >
> > CODESYS? a trademark of 3S-Smart Software Solutions GmbH
> >
> > Inspiring Automation Solutions
> >
> > 3S-Smart Software Solutions GmbH
> > Dipl.-Inf. Markus Schaber | Product Development Core Technology
> > Memminger Str. 151 | 87439 Kempten | Germany Tel. +49-831-54031-979 |
> > Fax +49-831-54031-50
> >
> > E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS
> > store: http://store.codesys.com CODESYS forum:
> > http://forum.codesys.com
> >
> > Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner |
> > Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
> >
> >> -----Urspr?ngliche Nachricht-----
> >> Von: Ironpython-users [mailto:ironpython-users-
> >> bounces+m.schaber=codesys.com at python.org] Im Auftrag von Jeff Hardy
> >> Gesendet: Montag, 20. Januar 2014 10:10
> >> An: ironpython-users at python.org
> >> Betreff: [Ironpython-users] Splitting up the IronPython repo
> >>
> >> One of the big issues with working on IronPython is the size of the
> >> git repository (specifically https://github.com/IronLanguages/main) -
> >> git does not like really big repos, especially on Windows. Part of
> >> the problem is that the repository includes:
> >>
> >> * The DLR
> >> * IronPython
> >> * IronRuby
> >> * Two copies of the Python stdlib
> >> * The Ruby stdlib
> >> * WiX
> >> * and a bunch of reference assemblies
> >>
> >> Even on a fast machine, 'git status' takes several seconds to return.
> >> I believe this is because it was originally a TFS repo, which can
> >> scale to handle bigger repos by using a bigger server. With git that
> >> option doesn't exist - if the repo is too big, the only option is to split
> it up.
> >>
> >> I've created two repos - https://github.com/jdhardy/dlr and
> >> https://github.com/jdhardy/ironpython-only - that contain just the
> >> DLR and IronPython, respectively. In them, git calls are nearly
> >> instantaneous, which makes working with it a lot less painful.
> >>
> >> There are other advantages - the DLR can get its own release cycle
> >> and packaging, and IP can then depend on a specific version of the DLR.
> >> Each project has a modified version of the IronPython build system
> >> that makes it easy to build for other platforms (iOS, Android, Win8,
> >> etc. - they still need to ported and tested, but the builds are easier).
> >>
> >> I did most of the work using Mono/xbuild, so I know it works there
> >> (except for a bug in Mono's .NET 4.5 support), but it has some errors
> >> on Windows that I need to sort out.
> >>
> >> Once I get my Windows box back and get some time to fix the few
> >> remaining issues, I'll move the repos to the IronLanguages account
> >> and use them for development of a real DLR release and IronPython
> >> 3.0. Any more 2.7 releases will come out of the existing repo.
> >>
> >> One downside is that copying patches between 3.0 and 2.7 is going to
> >> be extra work, but 3.0 will solve so many problems with strings that
> >> I think it will quickly become the more common target. In general
> >> Python
> >> 3 momentum is picking up so it's a good time (some recent
> >> hand-wringing
> >> notwithstanding) to try and have IronPython 3 in the right spot at
> >> the right time.
> >>
> >> I haven't really addressed IronRuby because, for intents and
> >> purposes, it's dead.
> >>
> >> All of that said, if anyone has any objections I'd like to hear them.
> >> The split repos work well for me but I'm curious if others prefer the
> >> combined repo.
> >>
> >> - Jeff
> >> _______________________________________________
> >> Ironpython-users mailing list
> >> Ironpython-users at python.org
> >> https://mail.python.org/mailman/listinfo/ironpython-users

From vano at mail.mipt.ru  Tue Feb  4 14:07:22 2014
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Tue, 4 Feb 2014 17:07:22 +0400
Subject: [Ironpython-users] Splitting up the IronPython repo
In-Reply-To: <727D8E16AE957149B447FE368139F2B5398580BF@SERVER10>
References: <CAF7AXFEEQCqdkVRemgZA2m2FA8b=XmvPBj-Wovh75Z_FyznkiQ@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B5398580BF@SERVER10>
Message-ID: <18810510405.20140204170722@mail.mipt.ru>

Have no problems with either IronPython.sln or Git here
at HEAD ed928f6b39bff20a2f60225193b459581b9b756e
with VS 2010 Ultimate and TortoiseGit 1.8.5.0 .

-----Original Message-----
From: Markus Schaber <m.schaber at codesys.com>
Sent: Tuesday, February 4, 2014 12:21
To: Jeff Hardy <jdhardy at gmail.com>
Cc: 
Subject: [Ironpython-users] Splitting up the IronPython repo

Hi, Jeff,

I tried to open the IronPython solution file  using Visual Studio 2010 Pro and 2012 Express for Desktop. Both failed to open the IronPython, IronPython and IronPython.SQLite projects with the following error messages:

D:\Contribs\Ironpython\ironpython-only\Src\IronPython\IronPython.csproj : error  : Unable to read the project file 'IronPython.csproj'. 
D:\Contribs\Ironpython\ironpython-only\Src\IronPython\IronPython.csproj(516,3): The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero length.

D:\Contribs\Ironpython\ironpython-only\Src\IronPython.Modules\IronPython.Modules.csproj : error  : Unable to read the project file 'IronPython.Modules.csproj'. 
D:\Contribs\Ironpython\ironpython-only\Src\IronPython.Modules\IronPython.Modules.csproj(149,3): The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero length.

D:\Contribs\Ironpython\ironpython-only\Src\IronPython.SQLite\IronPython.SQLite.csproj : error  : Unable to read the project file 'IronPython.SQLite.csproj'. 
D:\Contribs\Ironpython\ironpython-only\Src\IronPython.SQLite\IronPython.SQLite.csproj(160,3): The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero length.

Some of the properties associated with the solution could not be read.

Best regards

Markus Schaber

CODESYS? a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions

3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915

> -----Urspr?ngliche Nachricht-----
> Von: Ironpython-users [mailto:ironpython-users-
> bounces+m.schaber=codesys.com at python.org] Im Auftrag von Jeff Hardy
> Gesendet: Montag, 20. Januar 2014 10:10
> An: ironpython-users at python.org
> Betreff: [Ironpython-users] Splitting up the IronPython repo
> 
> One of the big issues with working on IronPython is the size of the git
> repository (specifically https://github.com/IronLanguages/main) - git does
> not like really big repos, especially on Windows. Part of the problem is that
> the repository includes:
> 
> * The DLR
> * IronPython
> * IronRuby
> * Two copies of the Python stdlib
> * The Ruby stdlib
> * WiX
> * and a bunch of reference assemblies
> 
> Even on a fast machine, 'git status' takes several seconds to return.
> I believe this is because it was originally a TFS repo, which can scale to
> handle bigger repos by using a bigger server. With git that option doesn't
> exist - if the repo is too big, the only option is to split it up.
> 
> I've created two repos - https://github.com/jdhardy/dlr and
> https://github.com/jdhardy/ironpython-only - that contain just the DLR and
> IronPython, respectively. In them, git calls are nearly instantaneous, which
> makes working with it a lot less painful.
> 
> There are other advantages - the DLR can get its own release cycle and
> packaging, and IP can then depend on a specific version of the DLR.
> Each project has a modified version of the IronPython build system that makes
> it easy to build for other platforms (iOS, Android, Win8, etc. - they still
> need to ported and tested, but the builds are easier).
> 
> I did most of the work using Mono/xbuild, so I know it works there (except
> for a bug in Mono's .NET 4.5 support), but it has some errors on Windows that
> I need to sort out.
> 
> Once I get my Windows box back and get some time to fix the few remaining
> issues, I'll move the repos to the IronLanguages account and use them for
> development of a real DLR release and IronPython 3.0. Any more 2.7 releases
> will come out of the existing repo.
> 
> One downside is that copying patches between 3.0 and 2.7 is going to be extra
> work, but 3.0 will solve so many problems with strings that I think it will
> quickly become the more common target. In general Python
> 3 momentum is picking up so it's a good time (some recent hand-wringing
> notwithstanding) to try and have IronPython 3 in the right spot at the right
> time.
> 
> I haven't really addressed IronRuby because, for intents and purposes, it's
> dead.
> 
> All of that said, if anyone has any objections I'd like to hear them.
> The split repos work well for me but I'm curious if others prefer the
> combined repo.
> 
> - Jeff
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
_______________________________________________
Ironpython-users mailing list
Ironpython-users at python.org
https://mail.python.org/mailman/listinfo/ironpython-users


From m.schaber at codesys.com  Tue Feb  4 15:59:53 2014
From: m.schaber at codesys.com (Markus Schaber)
Date: Tue, 4 Feb 2014 14:59:53 +0000
Subject: [Ironpython-users] Splitting up the IronPython repo
In-Reply-To: <18810510405.20140204170722@mail.mipt.ru>
References: <CAF7AXFEEQCqdkVRemgZA2m2FA8b=XmvPBj-Wovh75Z_FyznkiQ@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B5398580BF@SERVER10>
 <18810510405.20140204170722@mail.mipt.ru>
Message-ID: <727D8E16AE957149B447FE368139F2B539858219@SERVER10>

Hi,

Von: Ironpython-users [mailto:ironpython-users-
> Have no problems with either IronPython.sln or Git here at HEAD
> ed928f6b39bff20a2f60225193b459581b9b756e
> with VS 2010 Ultimate and TortoiseGit 1.8.5.0 .

It seems you're at a different repository.

On my machine, the IronPython only repository head revision seems to be 
57fc405cde878c4079aeb5bebb35f3d67d91e161
(https://github.com/jdhardy/ironpython-only)

I'm not a GIT expert, so probably I'm doing something wrong...


Best regards

Markus Schaber

CODESYS? a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions

3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915

> -----Original Message-----
> From: Markus Schaber <m.schaber at codesys.com>
> Sent: Tuesday, February 4, 2014 12:21
> To: Jeff Hardy <jdhardy at gmail.com>
> Cc:
> Subject: [Ironpython-users] Splitting up the IronPython repo
> 
> Hi, Jeff,
> 
> I tried to open the IronPython solution file  using Visual Studio 2010 Pro
> and 2012 Express for Desktop. Both failed to open the IronPython, IronPython
> and IronPython.SQLite projects with the following error messages:
> 
> D:\Contribs\Ironpython\ironpython-only\Src\IronPython\IronPython.csproj :
> error  : Unable to read the project file 'IronPython.csproj'.
> D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython\IronPython.csproj(516,3): The value "" of the "Project"
> attribute in element <Import> is invalid. Parameter "path" cannot have zero
> length.
> 
> D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython.Modules\IronPython.Modules.csproj : error  : Unable to
> read the project file 'IronPython.Modules.csproj'.
> D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython.Modules\IronPython.Modules.csproj(149,3): The value "" of
> the "Project" attribute in element <Import> is invalid. Parameter "path"
> cannot have zero length.
> 
> D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython.SQLite\IronPython.SQLite.csproj : error  : Unable to read
> the project file 'IronPython.SQLite.csproj'.
> D:\Contribs\Ironpython\ironpython-
> only\Src\IronPython.SQLite\IronPython.SQLite.csproj(160,3): The value "" of
> the "Project" attribute in element <Import> is invalid. Parameter "path"
> cannot have zero length.
> 
> Some of the properties associated with the solution could not be read.
> 
> Best regards
> 
> Markus Schaber
> 
> CODESYS? a trademark of 3S-Smart Software Solutions GmbH
> 
> Inspiring Automation Solutions
> 
> 3S-Smart Software Solutions GmbH
> Dipl.-Inf. Markus Schaber | Product Development Core Technology Memminger Str.
> 151 | 87439 Kempten | Germany Tel. +49-831-54031-979 | Fax +49-831-54031-50
> 
> E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store:
> http://store.codesys.com CODESYS forum: http://forum.codesys.com
> 
> Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade
> register: Kempten HRB 6186 | Tax ID No.: DE 167014915
> 
> > -----Urspr?ngliche Nachricht-----
> > Von: Ironpython-users [mailto:ironpython-users-
> > bounces+m.schaber=codesys.com at python.org] Im Auftrag von Jeff Hardy
> > Gesendet: Montag, 20. Januar 2014 10:10
> > An: ironpython-users at python.org
> > Betreff: [Ironpython-users] Splitting up the IronPython repo
> >
> > One of the big issues with working on IronPython is the size of the
> > git repository (specifically https://github.com/IronLanguages/main) -
> > git does not like really big repos, especially on Windows. Part of the
> > problem is that the repository includes:
> >
> > * The DLR
> > * IronPython
> > * IronRuby
> > * Two copies of the Python stdlib
> > * The Ruby stdlib
> > * WiX
> > * and a bunch of reference assemblies
> >
> > Even on a fast machine, 'git status' takes several seconds to return.
> > I believe this is because it was originally a TFS repo, which can
> > scale to handle bigger repos by using a bigger server. With git that
> > option doesn't exist - if the repo is too big, the only option is to split
> it up.
> >
> > I've created two repos - https://github.com/jdhardy/dlr and
> > https://github.com/jdhardy/ironpython-only - that contain just the DLR
> > and IronPython, respectively. In them, git calls are nearly
> > instantaneous, which makes working with it a lot less painful.
> >
> > There are other advantages - the DLR can get its own release cycle and
> > packaging, and IP can then depend on a specific version of the DLR.
> > Each project has a modified version of the IronPython build system
> > that makes it easy to build for other platforms (iOS, Android, Win8,
> > etc. - they still need to ported and tested, but the builds are easier).
> >
> > I did most of the work using Mono/xbuild, so I know it works there
> > (except for a bug in Mono's .NET 4.5 support), but it has some errors
> > on Windows that I need to sort out.
> >
> > Once I get my Windows box back and get some time to fix the few
> > remaining issues, I'll move the repos to the IronLanguages account and
> > use them for development of a real DLR release and IronPython 3.0. Any
> > more 2.7 releases will come out of the existing repo.
> >
> > One downside is that copying patches between 3.0 and 2.7 is going to
> > be extra work, but 3.0 will solve so many problems with strings that I
> > think it will quickly become the more common target. In general Python
> > 3 momentum is picking up so it's a good time (some recent
> > hand-wringing
> > notwithstanding) to try and have IronPython 3 in the right spot at the
> > right time.
> >
> > I haven't really addressed IronRuby because, for intents and purposes,
> > it's dead.
> >
> > All of that said, if anyone has any objections I'd like to hear them.
> > The split repos work well for me but I'm curious if others prefer the
> > combined repo.
> >
> > - Jeff
> > _______________________________________________
> > Ironpython-users mailing list
> > Ironpython-users at python.org
> > https://mail.python.org/mailman/listinfo/ironpython-users
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
> 
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users

From olof.bjarnason at gmail.com  Tue Feb  4 13:39:18 2014
From: olof.bjarnason at gmail.com (Olof Bjarnason)
Date: Tue, 4 Feb 2014 12:39:18 +0000
Subject: [Ironpython-users] multiprocessing?
Message-ID: <CA+LucOhoHSGXXr6EzW9Nm7=66si520DKHjOuEeFJ6ZnJ7MSSug@mail.gmail.com>

Hi there, new to IronPython, but 5+ years user of CPython.

I've done some googling already, but cannot seem to find whether or not the
Python std lib module "multiprocessing" is intended to be used in
IronPython or not.

1. Is there a web page describing exactly what python standard modules are
available in the IronPython environment?
2. Is the intention of the IronPython maintainers to at support Python
Standard Library? It does seem like a bit of duplication between .NET
platform and PyStdLib, to say the least. But as a user of both worlds, I'd
like to pick-and-choose the best from each!


Regards

- Olof
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140204/f3e97965/attachment.html>

From jdhardy at gmail.com  Wed Feb  5 01:46:23 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Wed, 5 Feb 2014 00:46:23 +0000
Subject: [Ironpython-users] Splitting up the IronPython repo
In-Reply-To: <727D8E16AE957149B447FE368139F2B53985817F@SERVER10>
References: <CAF7AXFEEQCqdkVRemgZA2m2FA8b=XmvPBj-Wovh75Z_FyznkiQ@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B5398580BF@SERVER10>
 <CAF7AXFFheby5K-F_A7Pbi1mt7QxLP4Gow1M=-S8vqp=5m=0yyQ@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B53985817F@SERVER10>
Message-ID: <CAF7AXFGQqQb-DOnQp_C+_25iUNs1+O9CH1igxpKq5b-xQHpVdg@mail.gmail.com>

I pushed some updates to the repo that should make it work now. One
piece that is currently missing is the DLR assemblies. To get those,
download https://www.dropbox.com/s/bpg05pk50xyadgz/dlr-1.5.0.zip and
copy the 'lib' directory (in dlr-1.5.0) to
Util\References\DynamicLanguageRuntime.1.5.0 (that is, you should have
Util\References\DynamicLanguageRuntime.1.5.0\lib). I haven't tested
2012 but it works in 2013.

- Jeff

On Tue, Feb 4, 2014 at 10:40 AM, Markus Schaber <m.schaber at codesys.com> wrote:
> Hi, Jeff,
>
> Okay, I'll try to have a look at it later.
>
> I've got mixed results with MonoDevelop/Xamarin Studio in the past, including crashes while trying to load "normal" solution files (which work fine with VS, VS Express, MSBuild, VisualBuild and SharpDevelop)...
>
> Best regards
>
> Markus Schaber
>
> CODESYS? a trademark of 3S-Smart Software Solutions GmbH
>
> Inspiring Automation Solutions
>
> 3S-Smart Software Solutions GmbH
> Dipl.-Inf. Markus Schaber | Product Development Core Technology
> Memminger Str. 151 | 87439 Kempten | Germany
> Tel. +49-831-54031-979 | Fax +49-831-54031-50
>
> E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
> CODESYS forum: http://forum.codesys.com
>
> Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
>
>> -----Urspr?ngliche Nachricht-----
>> Von: Jeff Hardy [mailto:jdhardy at gmail.com]
>> Gesendet: Dienstag, 4. Februar 2014 10:01
>> An: Markus Schaber
>> Cc: ironpython-users at python.org
>> Betreff: Re: [Ironpython-users] Splitting up the IronPython repo
>>
>> I did most of the work on my MacBook, just running MSBuild from the command
>> line; it might still need some tweaking to work from VS. The main thing is
>> making sure all of the properties are set when just loading the projects (not
>> using Build.proj as a wrapper). If you look in the .csproj file, it tries to
>> populate a property (the filename of the platform-specific properties) and
>> then import it; my guess is that for some reason that is not getting set
>> properly and staying at ''.
>>
>> Unfortunately VS and MSBuild are not exactly the same; the environment and
>> pre-set properties are different depending on whether it is run from the
>> command line or VS. I do remember opening it in VS 2013 though, to make sure
>> that it worked, but it's possible I changed something since then or that 2012
>> and 2013 are slightly different. I don't want to require 2013, though. (I'd
>> also like to open cleanly in MonoDevelop/Xamarin Studio, but it tends to
>> mangle the csproj files in unfortunate ways.)
>>
>> - Jeff
>>
>> On Tue, Feb 4, 2014 at 8:21 AM, Markus Schaber <m.schaber at codesys.com> wrote:
>> > Hi, Jeff,
>> >
>> > I tried to open the IronPython solution file  using Visual Studio 2010 Pro
>> and 2012 Express for Desktop. Both failed to open the IronPython, IronPython
>> and IronPython.SQLite projects with the following error messages:
>> >
>> > D:\Contribs\Ironpython\ironpython-only\Src\IronPython\IronPython.csproj :
>> error  : Unable to read the project file 'IronPython.csproj'.
>> > D:\Contribs\Ironpython\ironpython-
>> only\Src\IronPython\IronPython.csproj(516,3): The value "" of the "Project"
>> attribute in element <Import> is invalid. Parameter "path" cannot have zero
>> length.
>> >
>> > D:\Contribs\Ironpython\ironpython-
>> only\Src\IronPython.Modules\IronPython.Modules.csproj : error  : Unable to
>> read the project file 'IronPython.Modules.csproj'.
>> > D:\Contribs\Ironpython\ironpython-
>> only\Src\IronPython.Modules\IronPython.Modules.csproj(149,3): The value "" of
>> the "Project" attribute in element <Import> is invalid. Parameter "path"
>> cannot have zero length.
>> >
>> > D:\Contribs\Ironpython\ironpython-
>> only\Src\IronPython.SQLite\IronPython.SQLite.csproj : error  : Unable to read
>> the project file 'IronPython.SQLite.csproj'.
>> > D:\Contribs\Ironpython\ironpython-
>> only\Src\IronPython.SQLite\IronPython.SQLite.csproj(160,3): The value "" of
>> the "Project" attribute in element <Import> is invalid. Parameter "path"
>> cannot have zero length.
>> >
>> > Some of the properties associated with the solution could not be read.
>> >
>> > Best regards
>> >
>> > Markus Schaber
>> >
>> > CODESYS? a trademark of 3S-Smart Software Solutions GmbH
>> >
>> > Inspiring Automation Solutions
>> >
>> > 3S-Smart Software Solutions GmbH
>> > Dipl.-Inf. Markus Schaber | Product Development Core Technology
>> > Memminger Str. 151 | 87439 Kempten | Germany Tel. +49-831-54031-979 |
>> > Fax +49-831-54031-50
>> >
>> > E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS
>> > store: http://store.codesys.com CODESYS forum:
>> > http://forum.codesys.com
>> >
>> > Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner |
>> > Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
>> >
>> >> -----Urspr?ngliche Nachricht-----
>> >> Von: Ironpython-users [mailto:ironpython-users-
>> >> bounces+m.schaber=codesys.com at python.org] Im Auftrag von Jeff Hardy
>> >> Gesendet: Montag, 20. Januar 2014 10:10
>> >> An: ironpython-users at python.org
>> >> Betreff: [Ironpython-users] Splitting up the IronPython repo
>> >>
>> >> One of the big issues with working on IronPython is the size of the
>> >> git repository (specifically https://github.com/IronLanguages/main) -
>> >> git does not like really big repos, especially on Windows. Part of
>> >> the problem is that the repository includes:
>> >>
>> >> * The DLR
>> >> * IronPython
>> >> * IronRuby
>> >> * Two copies of the Python stdlib
>> >> * The Ruby stdlib
>> >> * WiX
>> >> * and a bunch of reference assemblies
>> >>
>> >> Even on a fast machine, 'git status' takes several seconds to return.
>> >> I believe this is because it was originally a TFS repo, which can
>> >> scale to handle bigger repos by using a bigger server. With git that
>> >> option doesn't exist - if the repo is too big, the only option is to split
>> it up.
>> >>
>> >> I've created two repos - https://github.com/jdhardy/dlr and
>> >> https://github.com/jdhardy/ironpython-only - that contain just the
>> >> DLR and IronPython, respectively. In them, git calls are nearly
>> >> instantaneous, which makes working with it a lot less painful.
>> >>
>> >> There are other advantages - the DLR can get its own release cycle
>> >> and packaging, and IP can then depend on a specific version of the DLR.
>> >> Each project has a modified version of the IronPython build system
>> >> that makes it easy to build for other platforms (iOS, Android, Win8,
>> >> etc. - they still need to ported and tested, but the builds are easier).
>> >>
>> >> I did most of the work using Mono/xbuild, so I know it works there
>> >> (except for a bug in Mono's .NET 4.5 support), but it has some errors
>> >> on Windows that I need to sort out.
>> >>
>> >> Once I get my Windows box back and get some time to fix the few
>> >> remaining issues, I'll move the repos to the IronLanguages account
>> >> and use them for development of a real DLR release and IronPython
>> >> 3.0. Any more 2.7 releases will come out of the existing repo.
>> >>
>> >> One downside is that copying patches between 3.0 and 2.7 is going to
>> >> be extra work, but 3.0 will solve so many problems with strings that
>> >> I think it will quickly become the more common target. In general
>> >> Python
>> >> 3 momentum is picking up so it's a good time (some recent
>> >> hand-wringing
>> >> notwithstanding) to try and have IronPython 3 in the right spot at
>> >> the right time.
>> >>
>> >> I haven't really addressed IronRuby because, for intents and
>> >> purposes, it's dead.
>> >>
>> >> All of that said, if anyone has any objections I'd like to hear them.
>> >> The split repos work well for me but I'm curious if others prefer the
>> >> combined repo.
>> >>
>> >> - Jeff
>> >> _______________________________________________
>> >> Ironpython-users mailing list
>> >> Ironpython-users at python.org
>> >> https://mail.python.org/mailman/listinfo/ironpython-users

From no_reply at codeplex.com  Wed Feb  5 09:27:19 2014
From: no_reply at codeplex.com (CodePlex)
Date: 5 Feb 2014 00:27:19 -0800
Subject: [Ironpython-users] IronPython, Daily Digest 2/4/2014
Message-ID: <CO1MSDNADMCDS2GeSJI00015c22@discussions.codeplex.com>

Hi ironpython,

Here's your Daily Digest of new issues for project "IronPython".

In today's digest:ISSUES

1. [New comment] reading a file using codecs can fail

----------------------------------------------

ISSUES

1. [New comment] reading a file using codecs can fail
http://ironpython.codeplex.com/workitem/34951
User paweljasinski has commented on the issue:

"<p>It looks like this is platform specific problem. .net is broken, on mono it doesn't throw, but the number of converted characters is not reported correctly.<br>The problem on .net can be traced back to index parameter of DecoderFallbackBuffer.Fallback which does not follow specification.</p>"
----------------------------------------------



----------------------------------------------
You are receiving this email because you subscribed to notifications on CodePlex.

To report a bug, request a feature, or add a comment, visit IronPython Issue Tracker. You can unsubscribe or change your issue notification settings on CodePlex.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140205/2f446740/attachment.html>

From jdhardy at gmail.com  Wed Feb  5 10:04:29 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Wed, 5 Feb 2014 09:04:29 +0000
Subject: [Ironpython-users] multiprocessing?
In-Reply-To: <CA+LucOhoHSGXXr6EzW9Nm7=66si520DKHjOuEeFJ6ZnJ7MSSug@mail.gmail.com>
References: <CA+LucOhoHSGXXr6EzW9Nm7=66si520DKHjOuEeFJ6ZnJ7MSSug@mail.gmail.com>
Message-ID: <CAF7AXFH9cbteqnq+aSRkfxJLitK1ryEfZ+GX=J5+R2hnQVbKKQ@mail.gmail.com>

On Tue, Feb 4, 2014 at 12:39 PM, Olof Bjarnason
<olof.bjarnason at gmail.com> wrote:
> Hi there, new to IronPython, but 5+ years user of CPython.
>
> I've done some googling already, but cannot seem to find whether or not the
> Python std lib module "multiprocessing" is intended to be used in IronPython
> or not.

I would love to, but it's probably the most difficult module to
support in the entire stdlib. It's also primarily (but not entirely) a
workaround for CPython's GIL, which IronPython doesn't have. Because
multiprocessing and threading implement the same API, in principle you
should be able to use multiprocessing on CPython and threading on
IronPython without much effort.

>
> 1. Is there a web page describing exactly what python standard modules are
> available in the IronPython environment?

Not yet. One of my never-quite-at-the-top-of-my-list tasks is to
create a branch of the Python documentation that contains
IronPython-specific information. Last time I tried I got lost yak
shaving Sphinx and never quite finished it
(https://github.com/IronLanguages/ironpython-docs). Ideally it would
list what version of IronPython things became available (or if they're
not) and have completely rewritten sections for extending & embedding.

> 2. Is the intention of the IronPython maintainers to at support Python
> Standard Library? It does seem like a bit of duplication between .NET
> platform and PyStdLib, to say the least. But as a user of both worlds, I'd
> like to pick-and-choose the best from each!

As much as possible, yes. There are a few cases (like multiprocessing)
where it might be too difficult to be worth the effort, but the vast
majority of it should Just Work.

- Jeff

From olof.bjarnason at gmail.com  Wed Feb  5 10:38:30 2014
From: olof.bjarnason at gmail.com (Olof Bjarnason)
Date: Wed, 5 Feb 2014 09:38:30 +0000
Subject: [Ironpython-users] multiprocessing?
In-Reply-To: <CAF7AXFH9cbteqnq+aSRkfxJLitK1ryEfZ+GX=J5+R2hnQVbKKQ@mail.gmail.com>
References: <CA+LucOhoHSGXXr6EzW9Nm7=66si520DKHjOuEeFJ6ZnJ7MSSug@mail.gmail.com>
 <CAF7AXFH9cbteqnq+aSRkfxJLitK1ryEfZ+GX=J5+R2hnQVbKKQ@mail.gmail.com>
Message-ID: <CA+LucOhV=H8ShNOp=0Cb1WN2yC_asR2K8OWUAG3F8suo9qrS3g@mail.gmail.com>

On 5 February 2014 09:04, Jeff Hardy <jdhardy at gmail.com> wrote:

> On Tue, Feb 4, 2014 at 12:39 PM, Olof Bjarnason
> <olof.bjarnason at gmail.com> wrote:
> > Hi there, new to IronPython, but 5+ years user of CPython.
> >
> > I've done some googling already, but cannot seem to find whether or not
> the
> > Python std lib module "multiprocessing" is intended to be used in
> IronPython
> > or not.
>
> I would love to, but it's probably the most difficult module to
> support in the entire stdlib. It's also primarily (but not entirely) a
> workaround for CPython's GIL, which IronPython doesn't have. Because
> multiprocessing and threading implement the same API, in principle you
> should be able to use multiprocessing on CPython and threading on
> IronPython without much effort.
>
>
I can see multiprocessing being one of the most advanced modules of
PyStdLib, yes.

I guess I could just switch to the threading module then. But does it
support Pool e.g., so I can re-use scripts I already have written?

Is the semantics of thread the same as in Python, being "fake" threading,
not separate processes, in the IPY version of threading?

I guess a follow up on that is "why"? Shouldn't the semantics be as similar
as possible to the PyStdLib?


>
> > 1. Is there a web page describing exactly what python standard modules
> are
> > available in the IronPython environment?
>
> Not yet. One of my never-quite-at-the-top-of-my-list tasks is to
> create a branch of the Python documentation that contains
> IronPython-specific information. Last time I tried I got lost yak
> shaving Sphinx and never quite finished it
> (https://github.com/IronLanguages/ironpython-docs). Ideally it would
> list what version of IronPython things became available (or if they're
> not) and have completely rewritten sections for extending & embedding.
>

I can see this being a huge task to tackle, but it is also essential for
IronPython adoption by CPython devs (as me).

It is definately need to have documentation of what is available and not
from the StdLib.

Also, having a fork of the whole doc seems overkill - and a high
maintenance burden.

I'm asking for a list of modules and their status. Importable? Unit tests
passing? Sounds almost automatable to come with that list come to think of
it...

Is there any CI mechanism for IronPython in place? TravisCI or similar?


>
> > 2. Is the intention of the IronPython maintainers to at support Python
> > Standard Library? It does seem like a bit of duplication between .NET
> > platform and PyStdLib, to say the least. But as a user of both worlds,
> I'd
> > like to pick-and-choose the best from each!
>
> As much as possible, yes. There are a few cases (like multiprocessing)
> where it might be too difficult to be worth the effort, but the vast
> majority of it should Just Work.
>
> - Jeff
>

Thanks for you long answer Jeff!


/Olof
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140205/d88f4aa8/attachment.html>

From jdhardy at gmail.com  Wed Feb  5 10:57:55 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Wed, 5 Feb 2014 09:57:55 +0000
Subject: [Ironpython-users] multiprocessing?
In-Reply-To: <CA+LucOhV=H8ShNOp=0Cb1WN2yC_asR2K8OWUAG3F8suo9qrS3g@mail.gmail.com>
References: <CA+LucOhoHSGXXr6EzW9Nm7=66si520DKHjOuEeFJ6ZnJ7MSSug@mail.gmail.com>
 <CAF7AXFH9cbteqnq+aSRkfxJLitK1ryEfZ+GX=J5+R2hnQVbKKQ@mail.gmail.com>
 <CA+LucOhV=H8ShNOp=0Cb1WN2yC_asR2K8OWUAG3F8suo9qrS3g@mail.gmail.com>
Message-ID: <CAF7AXFFyUPfR2J5BhRoN=hH3+cfTnG+47gtZ5uXeZJ6zCEEXJQ@mail.gmail.com>

On Wed, Feb 5, 2014 at 9:38 AM, Olof Bjarnason <olof.bjarnason at gmail.com> wrote:
>
>
>
> On 5 February 2014 09:04, Jeff Hardy <jdhardy at gmail.com> wrote:
>>
>> On Tue, Feb 4, 2014 at 12:39 PM, Olof Bjarnason
>> <olof.bjarnason at gmail.com> wrote:
>> > Hi there, new to IronPython, but 5+ years user of CPython.
>> >
>> > I've done some googling already, but cannot seem to find whether or not
>> > the
>> > Python std lib module "multiprocessing" is intended to be used in
>> > IronPython
>> > or not.
>>
>> I would love to, but it's probably the most difficult module to
>> support in the entire stdlib. It's also primarily (but not entirely) a
>> workaround for CPython's GIL, which IronPython doesn't have. Because
>> multiprocessing and threading implement the same API, in principle you
>> should be able to use multiprocessing on CPython and threading on
>> IronPython without much effort.
>>
>
> I can see multiprocessing being one of the most advanced modules of
> PyStdLib, yes.

Not only that, but it's fork behaviour might not even be possible
running on the .NET framework. Mono's documentation for fork() has all
sorts of warnings - basically, if it breaks, you get to keep the
pieces. I'm confident it's even possible (and I don't think Jython
supports it either, for much the same reason).

>
> I guess I could just switch to the threading module then. But does it
> support Pool e.g., so I can re-use scripts I already have written?

There is a thread pool in the multiprocessing library that I believe
is pure Python and should work on IronPython (perhaps with some small
patches). I haven't checked to see if concurrent.futures works; it'll
have to for 3.0, but I'm not sure about the 2.7 backport.

>
> Is the semantics of thread the same as in Python, being "fake" threading,
> not separate processes, in the IPY version of threading?

Each Python thread is 1:1 with a .NET thread, which I'm pretty sure
are 1:1 with OS threads. Only one process is used (.NET, not
surprisingly, takes a lot from Windows' computation model, where
threads are cheap and processes are expensive).

>
> I guess a follow up on that is "why"? Shouldn't the semantics be as similar
> as possible to the PyStdLib?

Time and interest. There are other gaps in the stdlib (e.g. expat)
that are more critical, and plenty of other bug fixes that will have a
wider impact. If someone provides the patches to make it work I
wouldn't turn it down, but so far I don't think anyone has the
interest yet.

>
>
>> >
>> > 1. Is there a web page describing exactly what python standard modules
>> > are
>> > available in the IronPython environment?
>>
>> Not yet. One of my never-quite-at-the-top-of-my-list tasks is to
>> create a branch of the Python documentation that contains
>> IronPython-specific information. Last time I tried I got lost yak
>> shaving Sphinx and never quite finished it
>> (https://github.com/IronLanguages/ironpython-docs). Ideally it would
>> list what version of IronPython things became available (or if they're
>> not) and have completely rewritten sections for extending & embedding.
>
>
> I can see this being a huge task to tackle, but it is also essential for
> IronPython adoption by CPython devs (as me).
>
> It is definately need to have documentation of what is available and not
> from the StdLib.

I know. :( It's difficult to balance things that benefit the community
(documentation) vs. things that I want to work on (Py3k features).

>
> Also, having a fork of the whole doc seems overkill - and a high maintenance
> burden.

Not any more work than having a fork of the standard library, which we
already do.  I have some git tricks up my sleeve to make it easy to do
a 3-way merge which will make it easy to bring in upstream changes
while preserving IronPython changes.

>
> I'm asking for a list of modules and their status. Importable? Unit tests
> passing? Sounds almost automatable to come with that list come to think of
> it...
>
> Is there any CI mechanism for IronPython in place? TravisCI or similar?

http://teamcity.codebetter.com/project.html?projectId=project110. Not
in great shape right now, mainly because the IronPython tests are
extremely difficult to run. Another thing on the 3.0 TODO list is
getting that straightened out.

- Jeff

From olof.bjarnason at gmail.com  Wed Feb  5 12:55:46 2014
From: olof.bjarnason at gmail.com (Olof Bjarnason)
Date: Wed, 5 Feb 2014 11:55:46 +0000
Subject: [Ironpython-users] multiprocessing?
In-Reply-To: <CAF7AXFFyUPfR2J5BhRoN=hH3+cfTnG+47gtZ5uXeZJ6zCEEXJQ@mail.gmail.com>
References: <CA+LucOhoHSGXXr6EzW9Nm7=66si520DKHjOuEeFJ6ZnJ7MSSug@mail.gmail.com>
 <CAF7AXFH9cbteqnq+aSRkfxJLitK1ryEfZ+GX=J5+R2hnQVbKKQ@mail.gmail.com>
 <CA+LucOhV=H8ShNOp=0Cb1WN2yC_asR2K8OWUAG3F8suo9qrS3g@mail.gmail.com>
 <CAF7AXFFyUPfR2J5BhRoN=hH3+cfTnG+47gtZ5uXeZJ6zCEEXJQ@mail.gmail.com>
Message-ID: <CA+LucOgWnmrQjhV9e3Sqnvsqvn0e+qiXi8nhndC6EffN7h0-MQ@mail.gmail.com>

On 5 February 2014 09:57, Jeff Hardy <jdhardy at gmail.com> wrote:

> On Wed, Feb 5, 2014 at 9:38 AM, Olof Bjarnason <olof.bjarnason at gmail.com>
> wrote:
> >
> >
> >
> > On 5 February 2014 09:04, Jeff Hardy <jdhardy at gmail.com> wrote:
> >>
> >> On Tue, Feb 4, 2014 at 12:39 PM, Olof Bjarnason
> >> <olof.bjarnason at gmail.com> wrote:
> >> > Hi there, new to IronPython, but 5+ years user of CPython.
> >> >
> >> > I've done some googling already, but cannot seem to find whether or
> not
> >> > the
> >> > Python std lib module "multiprocessing" is intended to be used in
> >> > IronPython
> >> > or not.
> >>
> >> I would love to, but it's probably the most difficult module to
> >> support in the entire stdlib. It's also primarily (but not entirely) a
> >> workaround for CPython's GIL, which IronPython doesn't have. Because
> >> multiprocessing and threading implement the same API, in principle you
> >> should be able to use multiprocessing on CPython and threading on
> >> IronPython without much effort.
> >>
> >
> > I can see multiprocessing being one of the most advanced modules of
> > PyStdLib, yes.
>
> Not only that, but it's fork behaviour might not even be possible
> running on the .NET framework. Mono's documentation for fork() has all
> sorts of warnings - basically, if it breaks, you get to keep the
> pieces. I'm confident it's even possible (and I don't think Jython
> supports it either, for much the same reason).
>

OK. The multiprocessing.Pool is a readable way of doing parallellization in
CPython. Basically run a method and collect results in a list (can it be
simpler than that?). It seems concurrent.futures has a similar syntax using
the ThreadPool.

I had a vague hope threading module would have the same module available
(since their API is supposed to be 1:1), but cannot find it via from
threading import Pool (CPython 2.7.6).

For now I could just write a similar module in IPY myself, since I know
.NET Threads and the Pool api I need for my script. On more general note
it's a bit unfortunate that Pool isn't available in PyStdLib's threading
module.

Pool missing from threading might be more of a PyStdLib issue than
IronPython issue, but it affects me anyway. :)


> >
> > I guess I could just switch to the threading module then. But does it
> > support Pool e.g., so I can re-use scripts I already have written?
>
> There is a thread pool in the multiprocessing library that I believe
> is pure Python and should work on IronPython (perhaps with some small
> patches). I haven't checked to see if concurrent.futures works; it'll
> have to for 3.0, but I'm not sure about the 2.7 backport.
>

Is it's in multprocessing, that isn't importable into IPY. But possibly I
can just copy-patch it to be able to use Pools + threading in IPY.


> >
> > Is the semantics of thread the same as in Python, being "fake" threading,
> > not separate processes, in the IPY version of threading?
>
> Each Python thread is 1:1 with a .NET thread, which I'm pretty sure
> are 1:1 with OS threads. Only one process is used (.NET, not
> surprisingly, takes a lot from Windows' computation model, where
> threads are cheap and processes are expensive).
>

Makes sense nomenclature wise, but not semantically for Python in general;
the question to ask is "would the Python scrip parallell_execution.py give
the same computation result running under both CPython and IronPython?" and
the answer would in general be "No" since Python Threads are running
sequentially/interleaved while the IronPython execution model relies on
.NET threads which is actually not interleaved (at least not in the Python
GIL sense .. it might be in the windows process-thread sense). This is
getting complicated :)


> >
> > I guess a follow up on that is "why"? Shouldn't the semantics be as
> similar
> > as possible to the PyStdLib?
>
> Time and interest. There are other gaps in the stdlib (e.g. expat)
> that are more critical, and plenty of other bug fixes that will have a
> wider impact. If someone provides the patches to make it work I
> wouldn't turn it down, but so far I don't think anyone has the
> interest yet.
>

I see.


>
> >
> >
> >> >
> >> > 1. Is there a web page describing exactly what python standard modules
> >> > are
> >> > available in the IronPython environment?
> >>
> >> Not yet. One of my never-quite-at-the-top-of-my-list tasks is to
> >> create a branch of the Python documentation that contains
> >> IronPython-specific information. Last time I tried I got lost yak
> >> shaving Sphinx and never quite finished it
> >> (https://github.com/IronLanguages/ironpython-docs). Ideally it would
> >> list what version of IronPython things became available (or if they're
> >> not) and have completely rewritten sections for extending & embedding.
> >
> >
> > I can see this being a huge task to tackle, but it is also essential for
> > IronPython adoption by CPython devs (as me).
> >
> > It is definately need to have documentation of what is available and not
> > from the StdLib.
>
> I know. :( It's difficult to balance things that benefit the community
> (documentation) vs. things that I want to work on (Py3k features).
>

Know that balance, it's hard!


>
> >
> > Also, having a fork of the whole doc seems overkill - and a high
> maintenance
> > burden.
>
> Not any more work than having a fork of the standard library, which we
> already do.  I have some git tricks up my sleeve to make it easy to do
> a 3-way merge which will make it easy to bring in upstream changes
> while preserving IronPython changes.
>
> >
> > I'm asking for a list of modules and their status. Importable? Unit tests
> > passing? Sounds almost automatable to come with that list come to think
> of
> > it...
> >
> > Is there any CI mechanism for IronPython in place? TravisCI or similar?
>
> http://teamcity.codebetter.com/project.html?projectId=project110. Not
> in great shape right now, mainly because the IronPython tests are
> extremely difficult to run. Another thing on the 3.0 TODO list is
> getting that straightened out.
>

I get a login screen following that URL...


>
> - Jeff
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140205/29256db8/attachment.html>

From vano at mail.mipt.ru  Wed Feb  5 13:14:49 2014
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Wed, 5 Feb 2014 16:14:49 +0400
Subject: [Ironpython-users] multiprocessing?
In-Reply-To: <CA+LucOgWnmrQjhV9e3Sqnvsqvn0e+qiXi8nhndC6EffN7h0-MQ@mail.gmail.com>
References: <CA+LucOhoHSGXXr6EzW9Nm7=66si520DKHjOuEeFJ6ZnJ7MSSug@mail.gmail.com>
 <CAF7AXFH9cbteqnq+aSRkfxJLitK1ryEfZ+GX=J5+R2hnQVbKKQ@mail.gmail.com>
 <CA+LucOhV=H8ShNOp=0Cb1WN2yC_asR2K8OWUAG3F8suo9qrS3g@mail.gmail.com>
 <CAF7AXFFyUPfR2J5BhRoN=hH3+cfTnG+47gtZ5uXeZJ6zCEEXJQ@mail.gmail.com>
 <CA+LucOgWnmrQjhV9e3Sqnvsqvn0e+qiXi8nhndC6EffN7h0-MQ@mail.gmail.com>
Message-ID: <1611137591.20140205161449@mail.mipt.ru>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140205/d64610cf/attachment.html>

From jdhardy at gmail.com  Wed Feb  5 15:27:27 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Wed, 5 Feb 2014 14:27:27 +0000
Subject: [Ironpython-users] multiprocessing?
In-Reply-To: <CA+LucOgWnmrQjhV9e3Sqnvsqvn0e+qiXi8nhndC6EffN7h0-MQ@mail.gmail.com>
References: <CA+LucOhoHSGXXr6EzW9Nm7=66si520DKHjOuEeFJ6ZnJ7MSSug@mail.gmail.com>
 <CAF7AXFH9cbteqnq+aSRkfxJLitK1ryEfZ+GX=J5+R2hnQVbKKQ@mail.gmail.com>
 <CA+LucOhV=H8ShNOp=0Cb1WN2yC_asR2K8OWUAG3F8suo9qrS3g@mail.gmail.com>
 <CAF7AXFFyUPfR2J5BhRoN=hH3+cfTnG+47gtZ5uXeZJ6zCEEXJQ@mail.gmail.com>
 <CA+LucOgWnmrQjhV9e3Sqnvsqvn0e+qiXi8nhndC6EffN7h0-MQ@mail.gmail.com>
Message-ID: <CAF7AXFFJC04jyk9MDUbWS2DK7MCTuzNJUpPqX30mUyhD1fbusA@mail.gmail.com>

On Wed, Feb 5, 2014 at 11:55 AM, Olof Bjarnason
<olof.bjarnason at gmail.com> wrote:
> OK. The multiprocessing.Pool is a readable way of doing parallellization in
> CPython. Basically run a method and collect results in a list (can it be
> simpler than that?). It seems concurrent.futures has a similar syntax using
> the ThreadPool.
>
> I had a vague hope threading module would have the same module available
> (since their API is supposed to be 1:1), but cannot find it via from
> threading import Pool (CPython 2.7.6).

Try from multiprocessing.pool import ThreadPool. IronPython should
include the Python portions of multiprocessing (namely the 'dummy'
module). If not, please open an issue and I'll include them in 2.7.5.

>
> For now I could just write a similar module in IPY myself, since I know .NET
> Threads and the Pool api I need for my script. On more general note it's a
> bit unfortunate that Pool isn't available in PyStdLib's threading module.
>
> Pool missing from threading might be more of a PyStdLib issue than
> IronPython issue, but it affects me anyway. :)

It's there, it's just in a weird spot and not documented.

>
>>
>> >
>> > I guess I could just switch to the threading module then. But does it
>> > support Pool e.g., so I can re-use scripts I already have written?
>>
>> There is a thread pool in the multiprocessing library that I believe
>> is pure Python and should work on IronPython (perhaps with some small
>> patches). I haven't checked to see if concurrent.futures works; it'll
>> have to for 3.0, but I'm not sure about the 2.7 backport.
>
>
> Is it's in multprocessing, that isn't importable into IPY. But possibly I
> can just copy-patch it to be able to use Pools + threading in IPY.

Also an option, but I would like to make the multiprocessing
ThreadPool usable in IronPython.

>
>>
>> >
>> > Is the semantics of thread the same as in Python, being "fake"
>> > threading,
>> > not separate processes, in the IPY version of threading?
>>
>> Each Python thread is 1:1 with a .NET thread, which I'm pretty sure
>> are 1:1 with OS threads. Only one process is used (.NET, not
>> surprisingly, takes a lot from Windows' computation model, where
>> threads are cheap and processes are expensive).
>
>
> Makes sense nomenclature wise, but not semantically for Python in general;
> the question to ask is "would the Python scrip parallell_execution.py give
> the same computation result running under both CPython and IronPython?" and
> the answer would in general be "No" since Python Threads are running
> sequentially/interleaved while the IronPython execution model relies on .NET
> threads which is actually not interleaved (at least not in the Python GIL
> sense .. it might be in the windows process-thread sense). This is getting
> complicated :)

Threading is always complicated. :)

>> > Is there any CI mechanism for IronPython in place? TravisCI or similar?
>>
>> http://teamcity.codebetter.com/project.html?projectId=project110. Not
>> in great shape right now, mainly because the IronPython tests are
>> extremely difficult to run. Another thing on the 3.0 TODO list is
>> getting that straightened out.
>
>
> I get a login screen following that URL...

Sorry, I forget to mention, just click "login as guest".

- Jeff

From jdhardy at gmail.com  Wed Feb  5 15:28:50 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Wed, 5 Feb 2014 14:28:50 +0000
Subject: [Ironpython-users] multiprocessing?
In-Reply-To: <1611137591.20140205161449@mail.mipt.ru>
References: <CA+LucOhoHSGXXr6EzW9Nm7=66si520DKHjOuEeFJ6ZnJ7MSSug@mail.gmail.com>
 <CAF7AXFH9cbteqnq+aSRkfxJLitK1ryEfZ+GX=J5+R2hnQVbKKQ@mail.gmail.com>
 <CA+LucOhV=H8ShNOp=0Cb1WN2yC_asR2K8OWUAG3F8suo9qrS3g@mail.gmail.com>
 <CAF7AXFFyUPfR2J5BhRoN=hH3+cfTnG+47gtZ5uXeZJ6zCEEXJQ@mail.gmail.com>
 <CA+LucOgWnmrQjhV9e3Sqnvsqvn0e+qiXi8nhndC6EffN7h0-MQ@mail.gmail.com>
 <1611137591.20140205161449@mail.mipt.ru>
Message-ID: <CAF7AXFEk+1KvdMAXWyH=HFqd6r5Wgr+ZPGX8gB--BXfap831Hg@mail.gmail.com>

On Wed, Feb 5, 2014 at 12:14 PM, Ivan Pozdeev <vano at mail.mipt.ru> wrote:
> The usages of multiprocessing.Process are few and far between in
> multiprocessing.Pool.
> This is the most prominent one:
>
>         for i in range(self._processes - len(self._pool)):
>             w = self.Process(target=worker,
>                              args=(self._inqueue, self._outqueue,
>                                    self._initializer,
>                                    self._initargs, self._maxtasksperchild)
>                             )
>             self._pool.append(w)
>
> I guess you can just replace multiprocessing\process.py with a set of
> thunks,
> then happily import the stock mutiprocessing.Pool .

There's a ThreadPool class in pools.py (at the bottom) that does just
that using the multiprocessing.dummy module. If that isn't included in
IronPython, it should be, even if multiprocessing doesn't work 100%.

- Jeff

From doug.blank at gmail.com  Wed Feb  5 19:52:43 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Wed, 5 Feb 2014 13:52:43 -0500
Subject: [Ironpython-users] Building IronRuby
Message-ID: <CAAusYCiTe-H++TBU=s5XsZ+u3hROVnTnKdnLSSsi1SUhhWqS9A@mail.gmail.com>

I was updating the page:

https://ironpython.codeplex.com/wikipage?title=IronPython%20on%20Mono

and got stuck on building IronRuby. I tried many variations (Mono 2.10 on
Ubuntu 13.10) but no luck. Always at least errors like:

PartialTrustDriver.cs(51,63): error CS0103: The name `PermissionState' does
not exist in the current context

and if I set  BaseConfiguration="v2Release", even more errors.

What is a method of building IronRuby now?

-Doug
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140205/72169e52/attachment.html>

From no_reply at codeplex.com  Thu Feb  6 09:29:23 2014
From: no_reply at codeplex.com (CodePlex)
Date: 6 Feb 2014 00:29:23 -0800
Subject: [Ironpython-users] IronPython, Daily Digest 2/5/2014
Message-ID: <CO1MSDNADMCDS2LI9mK00017da2@discussions.codeplex.com>

Hi ironpython,

Here's your Daily Digest of new issues for project "IronPython".

In today's digest:ISSUES

1. [New issue] from multiprocessing.pool import ThreadPool fails

----------------------------------------------

ISSUES

1. [New issue] from multiprocessing.pool import ThreadPool fails
http://ironpython.codeplex.com/workitem/34959
User objarni has proposed the issue:

"Jeff Hardys recommendation on importing ThreadPool from multiprocessing.pool fails on IPY 2.7.4.

Jeffs comment:
Try from multiprocessing.pool import ThreadPool. IronPython should
include the Python portions of multiprocessing (namely the 'dummy'
module). If not, please open an issue and I'll include them in 2.7.5.

My attempt:
IronPython 2.7.4 (2.7.0.40) on .NET 4.0.30319.18408 (32-bit)
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named multiprocessing
>>> from multiprocessing import ThreadPool
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named multiprocessing
>>> from multiprocessing.pool import ThreadPool
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named multiprocessing.pool
>>>
"
----------------------------------------------



----------------------------------------------
You are receiving this email because you subscribed to notifications on CodePlex.

To report a bug, request a feature, or add a comment, visit IronPython Issue Tracker. You can unsubscribe or change your issue notification settings on CodePlex.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140206/0c7c1fd0/attachment.html>

From doug.blank at gmail.com  Fri Feb  7 13:17:20 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Fri, 7 Feb 2014 07:17:20 -0500
Subject: [Ironpython-users] Fwd: pyzmq can be used with ironpython
In-Reply-To: <CA+7con=xD355+DTtTLqpMLZ66BWFmvJXw9QmiAMcmjTwv7ZDiw@mail.gmail.com>
References: <CA+7con=xD355+DTtTLqpMLZ66BWFmvJXw9QmiAMcmjTwv7ZDiw@mail.gmail.com>
Message-ID: <CAAusYChgs0FNPDLEEuziwG=ad6Y9x2mH5ar9Uy85CmY_tyUg9A@mail.gmail.com>

FYI, I started integrating our IronPython (and other languages) system,
named Calico, with IPython in the last few weeks. I ended up writing an
IPython "kernel" (the language backend) in C#:

https://bitbucket.org/ipre/calico/src/master/Source/Calico/ZMQServer.cs

It isn't complete yet (need to add tab-completion and object inspection,
fix a few bugs, such as properly shutting down, and some cleaning up) but
you can run IronPython in the IPython's notebook webbrowser frontend, as
shown here:

http://nbviewer.ipython.org/urls/bitbucket.org/ipre/calico/raw/master/notebooks/Calico%20Overview.ipynb

It uses clrzmq on Mono, but we need to get the low-level libs set for Mac
OSX and Windows.

I've also started attempting to make IPython be an importable library from
our IronPython kernel. Our goals aren't as ambitious as Pawel's... we are
just initially aiming to be able to use parts of IPython from our kernel,
rather than run all of IPython in IronPython. (IPython is made up of three
main components: the ipython startup program, the frontends (console,
qtconsole, and notebook), and the kernel.)  Discussion here:

https://github.com/ipython/ipython/issues/5032

If there are ways to reuse this into a general solution, or if others find
this useful, please let me know.

-Doug

On Fri, Dec 20, 2013 at 7:08 AM, Pawel Jasinski <pawel.jasinski at gmail.com>wrote:

> > For the ipython itself, modifications are ugly (I will push it to my
> > github repo today).
> > Not all usage scenarios of ipython/zmq work. There are unittest in
> > ipython/zmq which I simply gave up.
>
>
> https://github.com/paweljasinski/ipython/tree/iron-wip
>
> --pawel
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140207/b5cbbb1c/attachment.html>

From doug.blank at gmail.com  Fri Feb  7 13:37:31 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Fri, 7 Feb 2014 07:37:31 -0500
Subject: [Ironpython-users] IronPython SQLite interface
Message-ID: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>

Just recently discovered the sqlite3 port for IronPython, but having some
basic trouble:

This works fine:

import sqlite3
sqlite3.connect("test.db")

but this doesn't:

import sqlite3
sqlite3.connect("/home/dblank/test.db")

Any ideas?

-Doug
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140207/75b8bb89/attachment.html>

From slide.o.mix at gmail.com  Fri Feb  7 13:47:45 2014
From: slide.o.mix at gmail.com (Slide)
Date: Fri, 7 Feb 2014 05:47:45 -0700
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
Message-ID: <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>

Do you get a specific error message?


On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com> wrote:

> Just recently discovered the sqlite3 port for IronPython, but having some
> basic trouble:
>
> This works fine:
>
> import sqlite3
> sqlite3.connect("test.db")
>
> but this doesn't:
>
> import sqlite3
> sqlite3.connect("/home/dblank/test.db")
>
> Any ideas?
>
> -Doug
>
>
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>
>


-- 
Website: http://earl-of-code.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140207/68f9cd96/attachment.html>

From doug.blank at gmail.com  Fri Feb  7 14:01:45 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Fri, 7 Feb 2014 08:01:45 -0500
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
 <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
Message-ID: <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>

On Fri, Feb 7, 2014 at 7:47 AM, Slide <slide.o.mix at gmail.com> wrote:

> Do you get a specific error message?
>

Sorry, yes:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
_sqlite3.OperationalError: unable to open database file

Does this work for you? I get this error running latest ipy.exe (Mono 2.10,
Ubuntu 13.10) and IronPython in Calico.

-Doug



>
>
> On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com> wrote:
>
>> Just recently discovered the sqlite3 port for IronPython, but having some
>> basic trouble:
>>
>> This works fine:
>>
>> import sqlite3
>> sqlite3.connect("test.db")
>>
>> but this doesn't:
>>
>> import sqlite3
>> sqlite3.connect("/home/dblank/test.db")
>>
>> Any ideas?
>>
>> -Doug
>>
>>
>> _______________________________________________
>> Ironpython-users mailing list
>> Ironpython-users at python.org
>> https://mail.python.org/mailman/listinfo/ironpython-users
>>
>>
>
>
> --
> Website: http://earl-of-code.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140207/c28b1c2a/attachment.html>

From doug.blank at gmail.com  Fri Feb  7 14:45:20 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Fri, 7 Feb 2014 08:45:20 -0500
Subject: [Ironpython-users] Missing os.getuid()
Message-ID: <CAAusYCj+8j-D98wQBzrnkd=NhE-qh8Nrf=AjkLm5R8cDtgNTNg@mail.gmail.com>

I'm trying to figure out if the missing os.getuid() is a bug in IronPython.

On the one hand, this is to be defined for Unix, where os.name == "posix"
(when running IronPython on Unix).  On the other hand, what should it
return when sys.platform == "cli" and os.name == "nt"?

It seems that many programs in the wild will check to see if os.name is
"posix", and if it is, then it assumes all functions valid for Unix will be
available.

What is IronPython's strategy for situations like this?

-Doug
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140207/f31aaf2a/attachment.html>

From no_reply at codeplex.com  Mon Feb 10 09:26:18 2014
From: no_reply at codeplex.com (CodePlex)
Date: 10 Feb 2014 00:26:18 -0800
Subject: [Ironpython-users] IronPython, Daily Digest 2/9/2014
Message-ID: <CO1MSDNADMCDS2H71yE0001e29b@discussions.codeplex.com>

Hi ironpython,

Here's your Daily Digest of new issues for project "IronPython".

In today's digest:ISSUES

1. [New issue] metaclass interferes with property/partial
2. [New comment] metaclass interferes with property/partial

----------------------------------------------

ISSUES

1. [New issue] metaclass interferes with property/partial
http://ironpython.codeplex.com/workitem/34968
User tfigment has proposed the issue:

"Adding a metaclass causes partial properties to fail due to not passing all arguments to property using the partial.  Same functionality without even a dummy metaclass works."-----------------

2. [New comment] metaclass interferes with property/partial
http://ironpython.codeplex.com/workitem/34968
User tfigment has commented on the issue:

"<p>After addtional testing it looks like its a problem with partial though I cannot determine what it is from code at the moment.</p><p>using the following definition from the python docs instead of &quot;from functools import partial&quot; seems to work better.</p><p>```<br>def partial(func, *args, **keywords):<br>    def newfunc(*fargs, **fkeywords):<br>        newkeywords = keywords.copy()<br>        newkeywords.update(fkeywords)<br>        return func(*(args + fargs), **newkeywords)<br>    newfunc.func = func<br>    newfunc.args = args<br>    newfunc.keywords = keywords<br>    return newfunc<br>```</p>"
----------------------------------------------



----------------------------------------------
You are receiving this email because you subscribed to notifications on CodePlex.

To report a bug, request a feature, or add a comment, visit IronPython Issue Tracker. You can unsubscribe or change your issue notification settings on CodePlex.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140210/cbf62bde/attachment.html>

From jdhardy at gmail.com  Mon Feb 10 11:56:49 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Mon, 10 Feb 2014 10:56:49 +0000
Subject: [Ironpython-users] Missing os.getuid()
In-Reply-To: <CAAusYCj+8j-D98wQBzrnkd=NhE-qh8Nrf=AjkLm5R8cDtgNTNg@mail.gmail.com>
References: <CAAusYCj+8j-D98wQBzrnkd=NhE-qh8Nrf=AjkLm5R8cDtgNTNg@mail.gmail.com>
Message-ID: <CAF7AXFGb6iX1QbpVukgxPT7gAxD=+zb+sbLvFKmq_Pzyhp6zUw@mail.gmail.com>

On Fri, Feb 7, 2014 at 1:45 PM, Doug Blank <doug.blank at gmail.com> wrote:
> I'm trying to figure out if the missing os.getuid() is a bug in IronPython.
>
> On the one hand, this is to be defined for Unix, where os.name == "posix"
> (when running IronPython on Unix).  On the other hand, what should it return
> when sys.platform == "cli" and os.name == "nt"?

On CPython getuid simply doesn't exist if os.name == 'nt'. Seems like
the proper way to go. Arguably the POSIX stuff should not be in 'os',
but that ship has long sailed. :)

>
> It seems that many programs in the wild will check to see if os.name is
> "posix", and if it is, then it assumes all functions valid for Unix will be
> available.
>
> What is IronPython's strategy for situations like this?

Cry :(

Right now IronPython doesn't have a posix module, it just has an nt
module, and renames 'nt' to 'posix' when running on Mono. This is fine
for 90% of the APIs.

The proper fix would be to create a class containing all of the common
stuff implemented using the core .NET APIs (or, more accurately, the
PAL that wraps those APIs), and then separate 'posix' and 'nt' modules
that implement the OS-specific parts using Mono.Posix or P/Invoke.
It's not a huge amount of work but it's enough that it hasn't been
done yet.

- Jeff

From slide.o.mix at gmail.com  Mon Feb 10 14:47:58 2014
From: slide.o.mix at gmail.com (Slide)
Date: Mon, 10 Feb 2014 06:47:58 -0700
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
 <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
 <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>
Message-ID: <CAPiUgVfHcm5E=8JhBMYh-uw=FnDbe0=XzR_XntX7tsgk1eB5aQ@mail.gmail.com>

All the connect call does is this:

int rc = Sqlite3.sqlite3_open(database, out this.db);
if(rc != Sqlite3.SQLITE_OK)
    throw GetSqliteError(this.db, null);

Which makes me thing that sqlite3_open is returning an error code.
What happens if you do the same in CPython?



On Fri, Feb 7, 2014 at 6:01 AM, Doug Blank <doug.blank at gmail.com> wrote:

>
> On Fri, Feb 7, 2014 at 7:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>
>> Do you get a specific error message?
>>
>
> Sorry, yes:
>
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> _sqlite3.OperationalError: unable to open database file
>
> Does this work for you? I get this error running latest ipy.exe (Mono
> 2.10, Ubuntu 13.10) and IronPython in Calico.
>
> -Doug
>
>
>
>>
>>
>> On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>
>>> Just recently discovered the sqlite3 port for IronPython, but having
>>> some basic trouble:
>>>
>>> This works fine:
>>>
>>> import sqlite3
>>> sqlite3.connect("test.db")
>>>
>>> but this doesn't:
>>>
>>> import sqlite3
>>> sqlite3.connect("/home/dblank/test.db")
>>>
>>> Any ideas?
>>>
>>> -Doug
>>>
>>>
>>> _______________________________________________
>>> Ironpython-users mailing list
>>> Ironpython-users at python.org
>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>
>>>
>>
>>
>> --
>> Website: http://earl-of-code.com
>>
>
>


-- 
Website: http://earl-of-code.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140210/65a2701a/attachment.html>

From doug.blank at gmail.com  Mon Feb 10 15:19:49 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Mon, 10 Feb 2014 09:19:49 -0500
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAPiUgVfHcm5E=8JhBMYh-uw=FnDbe0=XzR_XntX7tsgk1eB5aQ@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
 <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
 <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>
 <CAPiUgVfHcm5E=8JhBMYh-uw=FnDbe0=XzR_XntX7tsgk1eB5aQ@mail.gmail.com>
Message-ID: <CAAusYCgT66iVMRq_m9-z___WKnk9HCy568ebf8U4bA68HfS+Yg@mail.gmail.com>

On Mon, Feb 10, 2014 at 8:47 AM, Slide <slide.o.mix at gmail.com> wrote:

> All the connect call does is this:
>
> int rc = Sqlite3.sqlite3_open(database, out this.db);
>
> if(rc != Sqlite3.SQLITE_OK)
>     throw GetSqliteError(this.db, null);
>
>
> Which makes me thing that sqlite3_open is returning an error code. What happens if you do the same in CPython?
>
>
No error, creates a file of length 0, and works:

$ python
Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.connect("/home/dblank/test.db")
<sqlite3.Connection object at 0x7fc3c21b6118>
>>>

$ ls -al /home/dblank/test.db
-rw-r--r-- 1 dblank dblank 0 Feb 10 09:15 /home/dblank/test.db

-Doug



>
>
> On Fri, Feb 7, 2014 at 6:01 AM, Doug Blank <doug.blank at gmail.com> wrote:
>
>>
>> On Fri, Feb 7, 2014 at 7:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>>
>>> Do you get a specific error message?
>>>
>>
>> Sorry, yes:
>>
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>> _sqlite3.OperationalError: unable to open database file
>>
>> Does this work for you? I get this error running latest ipy.exe (Mono
>> 2.10, Ubuntu 13.10) and IronPython in Calico.
>>
>> -Doug
>>
>>
>>
>>>
>>>
>>> On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>>
>>>> Just recently discovered the sqlite3 port for IronPython, but having
>>>> some basic trouble:
>>>>
>>>> This works fine:
>>>>
>>>> import sqlite3
>>>> sqlite3.connect("test.db")
>>>>
>>>> but this doesn't:
>>>>
>>>> import sqlite3
>>>> sqlite3.connect("/home/dblank/test.db")
>>>>
>>>> Any ideas?
>>>>
>>>> -Doug
>>>>
>>>>
>>>> _______________________________________________
>>>> Ironpython-users mailing list
>>>> Ironpython-users at python.org
>>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>>
>>>>
>>>
>>>
>>> --
>>> Website: http://earl-of-code.com
>>>
>>
>>
>
>
> --
> Website: http://earl-of-code.com
>
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140210/71dbcc08/attachment.html>

From doug.blank at gmail.com  Mon Feb 10 23:12:40 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Mon, 10 Feb 2014 17:12:40 -0500
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAAusYCgT66iVMRq_m9-z___WKnk9HCy568ebf8U4bA68HfS+Yg@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
 <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
 <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>
 <CAPiUgVfHcm5E=8JhBMYh-uw=FnDbe0=XzR_XntX7tsgk1eB5aQ@mail.gmail.com>
 <CAAusYCgT66iVMRq_m9-z___WKnk9HCy568ebf8U4bA68HfS+Yg@mail.gmail.com>
Message-ID: <CAAusYCj9xGJBXUK44PW1WdMo4s5PW5v9H_PdnUvBGKujs2y6nQ@mail.gmail.com>

After more testing, it appears that it only fails when the path begins with
a slash "/".

I see:

./c#sqlite/os_win_c.cs:      if ( zConverted.StartsWith( "/" ) &&
!zConverted.StartsWith( "//"

I'll try a couple of more things...

-Doug


On Mon, Feb 10, 2014 at 9:19 AM, Doug Blank <doug.blank at gmail.com> wrote:

>
> On Mon, Feb 10, 2014 at 8:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>
>> All the connect call does is this:
>>
>> int rc = Sqlite3.sqlite3_open(database, out this.db);
>>
>>
>> if(rc != Sqlite3.SQLITE_OK)
>>     throw GetSqliteError(this.db, null);
>>
>>
>>
>> Which makes me thing that sqlite3_open is returning an error code. What happens if you do the same in CPython?
>>
>>
> No error, creates a file of length 0, and works:
>
> $ python
> Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
> [GCC 4.8.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sqlite3
> >>> sqlite3.connect("/home/dblank/test.db")
> <sqlite3.Connection object at 0x7fc3c21b6118>
> >>>
>
> $ ls -al /home/dblank/test.db
> -rw-r--r-- 1 dblank dblank 0 Feb 10 09:15 /home/dblank/test.db
>
> -Doug
>
>
>
>>
>>
>> On Fri, Feb 7, 2014 at 6:01 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>
>>>
>>> On Fri, Feb 7, 2014 at 7:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>>>
>>>> Do you get a specific error message?
>>>>
>>>
>>> Sorry, yes:
>>>
>>> Traceback (most recent call last):
>>>   File "<string>", line 1, in <module>
>>> _sqlite3.OperationalError: unable to open database file
>>>
>>> Does this work for you? I get this error running latest ipy.exe (Mono
>>> 2.10, Ubuntu 13.10) and IronPython in Calico.
>>>
>>> -Doug
>>>
>>>
>>>
>>>>
>>>>
>>>> On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com>wrote:
>>>>
>>>>> Just recently discovered the sqlite3 port for IronPython, but having
>>>>> some basic trouble:
>>>>>
>>>>> This works fine:
>>>>>
>>>>> import sqlite3
>>>>> sqlite3.connect("test.db")
>>>>>
>>>>> but this doesn't:
>>>>>
>>>>> import sqlite3
>>>>> sqlite3.connect("/home/dblank/test.db")
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> -Doug
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ironpython-users mailing list
>>>>> Ironpython-users at python.org
>>>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Website: http://earl-of-code.com
>>>>
>>>
>>>
>>
>>
>> --
>> Website: http://earl-of-code.com
>>
>> _______________________________________________
>> Ironpython-users mailing list
>> Ironpython-users at python.org
>> https://mail.python.org/mailman/listinfo/ironpython-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140210/ef832cb1/attachment.html>

From doug.blank at gmail.com  Mon Feb 10 23:14:43 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Mon, 10 Feb 2014 17:14:43 -0500
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAAusYCj9xGJBXUK44PW1WdMo4s5PW5v9H_PdnUvBGKujs2y6nQ@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
 <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
 <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>
 <CAPiUgVfHcm5E=8JhBMYh-uw=FnDbe0=XzR_XntX7tsgk1eB5aQ@mail.gmail.com>
 <CAAusYCgT66iVMRq_m9-z___WKnk9HCy568ebf8U4bA68HfS+Yg@mail.gmail.com>
 <CAAusYCj9xGJBXUK44PW1WdMo4s5PW5v9H_PdnUvBGKujs2y6nQ@mail.gmail.com>
Message-ID: <CAAusYCg8zv=MoAMJth27zS0SwWz-coS449jkmJ68jKR0r+mpgw@mail.gmail.com>

That must be the issue, as "//home/dblank/name.db" works:

$ ipy.exe
IronPython 2.9.9a0 DEBUG (2.9.0.0) on Mono 4.0.30319.1 (64-bit)
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.connect("//home/dblank/test.db")
<Connection object at 0x000000000000002B>
>>>

-Doug



On Mon, Feb 10, 2014 at 5:12 PM, Doug Blank <doug.blank at gmail.com> wrote:

> After more testing, it appears that it only fails when the path begins
> with a slash "/".
>
> I see:
>
> ./c#sqlite/os_win_c.cs:      if ( zConverted.StartsWith( "/" ) &&
> !zConverted.StartsWith( "//"
>
> I'll try a couple of more things...
>
> -Doug
>
>
> On Mon, Feb 10, 2014 at 9:19 AM, Doug Blank <doug.blank at gmail.com> wrote:
>
>>
>> On Mon, Feb 10, 2014 at 8:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>>
>>> All the connect call does is this:
>>>
>>> int rc = Sqlite3.sqlite3_open(database, out this.db);
>>>
>>>
>>>
>>> if(rc != Sqlite3.SQLITE_OK)
>>>     throw GetSqliteError(this.db, null);
>>>
>>>
>>>
>>>
>>> Which makes me thing that sqlite3_open is returning an error code. What happens if you do the same in CPython?
>>>
>>>
>> No error, creates a file of length 0, and works:
>>
>> $ python
>> Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
>> [GCC 4.8.1] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>  >>> import sqlite3
>> >>> sqlite3.connect("/home/dblank/test.db")
>> <sqlite3.Connection object at 0x7fc3c21b6118>
>> >>>
>>
>> $ ls -al /home/dblank/test.db
>> -rw-r--r-- 1 dblank dblank 0 Feb 10 09:15 /home/dblank/test.db
>>
>> -Doug
>>
>>
>>
>>>
>>>
>>> On Fri, Feb 7, 2014 at 6:01 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>>
>>>>
>>>> On Fri, Feb 7, 2014 at 7:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>>>>
>>>>> Do you get a specific error message?
>>>>>
>>>>
>>>> Sorry, yes:
>>>>
>>>> Traceback (most recent call last):
>>>>   File "<string>", line 1, in <module>
>>>> _sqlite3.OperationalError: unable to open database file
>>>>
>>>> Does this work for you? I get this error running latest ipy.exe (Mono
>>>> 2.10, Ubuntu 13.10) and IronPython in Calico.
>>>>
>>>> -Doug
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com>wrote:
>>>>>
>>>>>> Just recently discovered the sqlite3 port for IronPython, but having
>>>>>> some basic trouble:
>>>>>>
>>>>>> This works fine:
>>>>>>
>>>>>> import sqlite3
>>>>>> sqlite3.connect("test.db")
>>>>>>
>>>>>> but this doesn't:
>>>>>>
>>>>>> import sqlite3
>>>>>> sqlite3.connect("/home/dblank/test.db")
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>> -Doug
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Ironpython-users mailing list
>>>>>> Ironpython-users at python.org
>>>>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Website: http://earl-of-code.com
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Website: http://earl-of-code.com
>>>
>>> _______________________________________________
>>> Ironpython-users mailing list
>>> Ironpython-users at python.org
>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140210/786c74ae/attachment.html>

From slide.o.mix at gmail.com  Mon Feb 10 23:15:24 2014
From: slide.o.mix at gmail.com (Slide)
Date: Mon, 10 Feb 2014 15:15:24 -0700
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAAusYCg8zv=MoAMJth27zS0SwWz-coS449jkmJ68jKR0r+mpgw@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
 <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
 <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>
 <CAPiUgVfHcm5E=8JhBMYh-uw=FnDbe0=XzR_XntX7tsgk1eB5aQ@mail.gmail.com>
 <CAAusYCgT66iVMRq_m9-z___WKnk9HCy568ebf8U4bA68HfS+Yg@mail.gmail.com>
 <CAAusYCj9xGJBXUK44PW1WdMo4s5PW5v9H_PdnUvBGKujs2y6nQ@mail.gmail.com>
 <CAAusYCg8zv=MoAMJth27zS0SwWz-coS449jkmJ68jKR0r+mpgw@mail.gmail.com>
Message-ID: <CAPiUgVc04tQ6hB_0dyjasaE9b86DPhcpEDAYYQQxkDr==uMBMA@mail.gmail.com>

That seems a little odd. Perhaps Jeff can shed some light on it, I think he
wrote the SQLite adaptation layer.


On Mon, Feb 10, 2014 at 3:14 PM, Doug Blank <doug.blank at gmail.com> wrote:

> That must be the issue, as "//home/dblank/name.db" works:
>
> $ ipy.exe
> IronPython 2.9.9a0 DEBUG (2.9.0.0) on Mono 4.0.30319.1 (64-bit)
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sqlite3
> >>> sqlite3.connect("//home/dblank/test.db")
> <Connection object at 0x000000000000002B>
> >>>
>
> -Doug
>
>
>
> On Mon, Feb 10, 2014 at 5:12 PM, Doug Blank <doug.blank at gmail.com> wrote:
>
>> After more testing, it appears that it only fails when the path begins
>> with a slash "/".
>>
>> I see:
>>
>> ./c#sqlite/os_win_c.cs:      if ( zConverted.StartsWith( "/" ) &&
>> !zConverted.StartsWith( "//"
>>
>> I'll try a couple of more things...
>>
>> -Doug
>>
>>
>> On Mon, Feb 10, 2014 at 9:19 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>
>>>
>>> On Mon, Feb 10, 2014 at 8:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>>>
>>>> All the connect call does is this:
>>>>
>>>> int rc = Sqlite3.sqlite3_open(database, out this.db);
>>>>
>>>>
>>>>
>>>>
>>>> if(rc != Sqlite3.SQLITE_OK)
>>>>     throw GetSqliteError(this.db, null);
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Which makes me thing that sqlite3_open is returning an error code. What happens if you do the same in CPython?
>>>>
>>>>
>>> No error, creates a file of length 0, and works:
>>>
>>> $ python
>>> Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
>>> [GCC 4.8.1] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>  >>> import sqlite3
>>> >>> sqlite3.connect("/home/dblank/test.db")
>>> <sqlite3.Connection object at 0x7fc3c21b6118>
>>> >>>
>>>
>>> $ ls -al /home/dblank/test.db
>>> -rw-r--r-- 1 dblank dblank 0 Feb 10 09:15 /home/dblank/test.db
>>>
>>> -Doug
>>>
>>>
>>>
>>>>
>>>>
>>>> On Fri, Feb 7, 2014 at 6:01 AM, Doug Blank <doug.blank at gmail.com>wrote:
>>>>
>>>>>
>>>>> On Fri, Feb 7, 2014 at 7:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>>>>>
>>>>>> Do you get a specific error message?
>>>>>>
>>>>>
>>>>> Sorry, yes:
>>>>>
>>>>> Traceback (most recent call last):
>>>>>   File "<string>", line 1, in <module>
>>>>> _sqlite3.OperationalError: unable to open database file
>>>>>
>>>>> Does this work for you? I get this error running latest ipy.exe (Mono
>>>>> 2.10, Ubuntu 13.10) and IronPython in Calico.
>>>>>
>>>>> -Doug
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com>wrote:
>>>>>>
>>>>>>> Just recently discovered the sqlite3 port for IronPython, but having
>>>>>>> some basic trouble:
>>>>>>>
>>>>>>> This works fine:
>>>>>>>
>>>>>>> import sqlite3
>>>>>>> sqlite3.connect("test.db")
>>>>>>>
>>>>>>> but this doesn't:
>>>>>>>
>>>>>>> import sqlite3
>>>>>>> sqlite3.connect("/home/dblank/test.db")
>>>>>>>
>>>>>>> Any ideas?
>>>>>>>
>>>>>>> -Doug
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Ironpython-users mailing list
>>>>>>> Ironpython-users at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Website: http://earl-of-code.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Website: http://earl-of-code.com
>>>>
>>>> _______________________________________________
>>>> Ironpython-users mailing list
>>>> Ironpython-users at python.org
>>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>>
>>>>
>>>
>>
>


-- 
Website: http://earl-of-code.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140210/461525d3/attachment.html>

From jdhardy at gmail.com  Tue Feb 11 10:39:44 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Tue, 11 Feb 2014 09:39:44 +0000
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAPiUgVc04tQ6hB_0dyjasaE9b86DPhcpEDAYYQQxkDr==uMBMA@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
 <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
 <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>
 <CAPiUgVfHcm5E=8JhBMYh-uw=FnDbe0=XzR_XntX7tsgk1eB5aQ@mail.gmail.com>
 <CAAusYCgT66iVMRq_m9-z___WKnk9HCy568ebf8U4bA68HfS+Yg@mail.gmail.com>
 <CAAusYCj9xGJBXUK44PW1WdMo4s5PW5v9H_PdnUvBGKujs2y6nQ@mail.gmail.com>
 <CAAusYCg8zv=MoAMJth27zS0SwWz-coS449jkmJ68jKR0r+mpgw@mail.gmail.com>
 <CAPiUgVc04tQ6hB_0dyjasaE9b86DPhcpEDAYYQQxkDr==uMBMA@mail.gmail.com>
Message-ID: <CAF7AXFG1bBuS-=kxzzsA8XBx4SfiCfGSJY2iLita2Vc2-W3RQA@mail.gmail.com>

On Mon, Feb 10, 2014 at 10:15 PM, Slide <slide.o.mix at gmail.com> wrote:
> That seems a little odd. Perhaps Jeff can shed some light on it, I think he
> wrote the SQLite adaptation layer.

I did, but the funny code is part of the C#-sqlite library, which is a
port of the C library to C#. (if you're feeling brave, dig through the
code. It's ... interesting.) It looks like the author just assumed
Windows-ish systems in all cases, and Doug has to go and break
everything by trying to run it on Unix. :(

I have no idea what that code is trying to do (context:
https://github.com/IronLanguages/main/blob/master/Languages/IronPython/IronPython.SQLite/c%23sqlite/os_win_c.cs#L2646);
it might be some vestigial part of SQLite that's not needed. Since //
seems OK I'm assuming that's the case.

Doug, can you open an issue? And in the meantime, is using // an
acceptable workaround?

- Jeff

>
>
> On Mon, Feb 10, 2014 at 3:14 PM, Doug Blank <doug.blank at gmail.com> wrote:
>>
>> That must be the issue, as "//home/dblank/name.db" works:
>>
>> $ ipy.exe
>> IronPython 2.9.9a0 DEBUG (2.9.0.0) on Mono 4.0.30319.1 (64-bit)
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import sqlite3
>> >>> sqlite3.connect("//home/dblank/test.db")
>> <Connection object at 0x000000000000002B>
>> >>>
>>
>> -Doug
>>
>>
>>
>> On Mon, Feb 10, 2014 at 5:12 PM, Doug Blank <doug.blank at gmail.com> wrote:
>>>
>>> After more testing, it appears that it only fails when the path begins
>>> with a slash "/".
>>>
>>> I see:
>>>
>>> ./c#sqlite/os_win_c.cs:      if ( zConverted.StartsWith( "/" ) &&
>>> !zConverted.StartsWith( "//"
>>>
>>> I'll try a couple of more things...
>>>
>>> -Doug
>>>
>>>
>>> On Mon, Feb 10, 2014 at 9:19 AM, Doug Blank <doug.blank at gmail.com> wrote:
>>>>
>>>>
>>>> On Mon, Feb 10, 2014 at 8:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>>>>>
>>>>> All the connect call does is this:
>>>>>
>>>>> int rc = Sqlite3.sqlite3_open(database, out this.db);
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> if(rc != Sqlite3.SQLITE_OK)
>>>>>     throw GetSqliteError(this.db, null);
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Which makes me thing that sqlite3_open is returning an error code. What
>>>>> happens if you do the same in CPython?
>>>>
>>>>
>>>> No error, creates a file of length 0, and works:
>>>>
>>>> $ python
>>>> Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
>>>> [GCC 4.8.1] on linux2
>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>> >>> import sqlite3
>>>> >>> sqlite3.connect("/home/dblank/test.db")
>>>> <sqlite3.Connection object at 0x7fc3c21b6118>
>>>> >>>
>>>>
>>>> $ ls -al /home/dblank/test.db
>>>> -rw-r--r-- 1 dblank dblank 0 Feb 10 09:15 /home/dblank/test.db
>>>>
>>>> -Doug
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 7, 2014 at 6:01 AM, Doug Blank <doug.blank at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 7, 2014 at 7:47 AM, Slide <slide.o.mix at gmail.com> wrote:
>>>>>>>
>>>>>>> Do you get a specific error message?
>>>>>>
>>>>>>
>>>>>> Sorry, yes:
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>   File "<string>", line 1, in <module>
>>>>>> _sqlite3.OperationalError: unable to open database file
>>>>>>
>>>>>> Does this work for you? I get this error running latest ipy.exe (Mono
>>>>>> 2.10, Ubuntu 13.10) and IronPython in Calico.
>>>>>>
>>>>>> -Doug
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Just recently discovered the sqlite3 port for IronPython, but having
>>>>>>>> some basic trouble:
>>>>>>>>
>>>>>>>> This works fine:
>>>>>>>>
>>>>>>>> import sqlite3
>>>>>>>> sqlite3.connect("test.db")
>>>>>>>>
>>>>>>>> but this doesn't:
>>>>>>>>
>>>>>>>> import sqlite3
>>>>>>>> sqlite3.connect("/home/dblank/test.db")
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> -Doug
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Ironpython-users mailing list
>>>>>>>> Ironpython-users at python.org
>>>>>>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Website: http://earl-of-code.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Website: http://earl-of-code.com
>>>>>
>>>>> _______________________________________________
>>>>> Ironpython-users mailing list
>>>>> Ironpython-users at python.org
>>>>> https://mail.python.org/mailman/listinfo/ironpython-users
>>>>>
>>>>
>>>
>>
>
>
>
> --
> Website: http://earl-of-code.com
>
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>

From doug.blank at gmail.com  Tue Feb 11 12:44:29 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Tue, 11 Feb 2014 06:44:29 -0500
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAF7AXFG1bBuS-=kxzzsA8XBx4SfiCfGSJY2iLita2Vc2-W3RQA@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
 <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
 <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>
 <CAPiUgVfHcm5E=8JhBMYh-uw=FnDbe0=XzR_XntX7tsgk1eB5aQ@mail.gmail.com>
 <CAAusYCgT66iVMRq_m9-z___WKnk9HCy568ebf8U4bA68HfS+Yg@mail.gmail.com>
 <CAAusYCj9xGJBXUK44PW1WdMo4s5PW5v9H_PdnUvBGKujs2y6nQ@mail.gmail.com>
 <CAAusYCg8zv=MoAMJth27zS0SwWz-coS449jkmJ68jKR0r+mpgw@mail.gmail.com>
 <CAPiUgVc04tQ6hB_0dyjasaE9b86DPhcpEDAYYQQxkDr==uMBMA@mail.gmail.com>
 <CAF7AXFG1bBuS-=kxzzsA8XBx4SfiCfGSJY2iLita2Vc2-W3RQA@mail.gmail.com>
Message-ID: <CAAusYCgYq6gWqDwaDmnhnJ1osfsvcEHQ8augCbEDebRVOA8pHA@mail.gmail.com>

Issue #173 with patch:

https://github.com/IronLanguages/main/issues/173

That works for me! I just removed the removal of the slash. I don't know
what that will do for Windows users that had a path that started with "/".

We distribute our own DLLs, so no rush for a new release.

Thanks!

-Doug


On Tue, Feb 11, 2014 at 4:39 AM, Jeff Hardy <jdhardy at gmail.com> wrote:

> On Mon, Feb 10, 2014 at 10:15 PM, Slide <slide.o.mix at gmail.com> wrote:
> > That seems a little odd. Perhaps Jeff can shed some light on it, I think
> he
> > wrote the SQLite adaptation layer.
>
> I did, but the funny code is part of the C#-sqlite library, which is a
> port of the C library to C#. (if you're feeling brave, dig through the
> code. It's ... interesting.) It looks like the author just assumed
> Windows-ish systems in all cases, and Doug has to go and break
> everything by trying to run it on Unix. :(
>
> I have no idea what that code is trying to do (context:
>
> https://github.com/IronLanguages/main/blob/master/Languages/IronPython/IronPython.SQLite/c%23sqlite/os_win_c.cs#L2646
> );
> it might be some vestigial part of SQLite that's not needed. Since //
> seems OK I'm assuming that's the case.
>
> Doug, can you open an issue? And in the meantime, is using // an
> acceptable workaround?
>
> - Jeff
>
> >
> >
> > On Mon, Feb 10, 2014 at 3:14 PM, Doug Blank <doug.blank at gmail.com>
> wrote:
> >>
> >> That must be the issue, as "//home/dblank/name.db" works:
> >>
> >> $ ipy.exe
> >> IronPython 2.9.9a0 DEBUG (2.9.0.0) on Mono 4.0.30319.1 (64-bit)
> >> Type "help", "copyright", "credits" or "license" for more information.
> >> >>> import sqlite3
> >> >>> sqlite3.connect("//home/dblank/test.db")
> >> <Connection object at 0x000000000000002B>
> >> >>>
> >>
> >> -Doug
> >>
> >>
> >>
> >> On Mon, Feb 10, 2014 at 5:12 PM, Doug Blank <doug.blank at gmail.com>
> wrote:
> >>>
> >>> After more testing, it appears that it only fails when the path begins
> >>> with a slash "/".
> >>>
> >>> I see:
> >>>
> >>> ./c#sqlite/os_win_c.cs:      if ( zConverted.StartsWith( "/" ) &&
> >>> !zConverted.StartsWith( "//"
> >>>
> >>> I'll try a couple of more things...
> >>>
> >>> -Doug
> >>>
> >>>
> >>> On Mon, Feb 10, 2014 at 9:19 AM, Doug Blank <doug.blank at gmail.com>
> wrote:
> >>>>
> >>>>
> >>>> On Mon, Feb 10, 2014 at 8:47 AM, Slide <slide.o.mix at gmail.com> wrote:
> >>>>>
> >>>>> All the connect call does is this:
> >>>>>
> >>>>> int rc = Sqlite3.sqlite3_open(database, out this.db);
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> if(rc != Sqlite3.SQLITE_OK)
> >>>>>     throw GetSqliteError(this.db, null);
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Which makes me thing that sqlite3_open is returning an error code.
> What
> >>>>> happens if you do the same in CPython?
> >>>>
> >>>>
> >>>> No error, creates a file of length 0, and works:
> >>>>
> >>>> $ python
> >>>> Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
> >>>> [GCC 4.8.1] on linux2
> >>>> Type "help", "copyright", "credits" or "license" for more information.
> >>>> >>> import sqlite3
> >>>> >>> sqlite3.connect("/home/dblank/test.db")
> >>>> <sqlite3.Connection object at 0x7fc3c21b6118>
> >>>> >>>
> >>>>
> >>>> $ ls -al /home/dblank/test.db
> >>>> -rw-r--r-- 1 dblank dblank 0 Feb 10 09:15 /home/dblank/test.db
> >>>>
> >>>> -Doug
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, Feb 7, 2014 at 6:01 AM, Doug Blank <doug.blank at gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Feb 7, 2014 at 7:47 AM, Slide <slide.o.mix at gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> Do you get a specific error message?
> >>>>>>
> >>>>>>
> >>>>>> Sorry, yes:
> >>>>>>
> >>>>>> Traceback (most recent call last):
> >>>>>>   File "<string>", line 1, in <module>
> >>>>>> _sqlite3.OperationalError: unable to open database file
> >>>>>>
> >>>>>> Does this work for you? I get this error running latest ipy.exe
> (Mono
> >>>>>> 2.10, Ubuntu 13.10) and IronPython in Calico.
> >>>>>>
> >>>>>> -Doug
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Just recently discovered the sqlite3 port for IronPython, but
> having
> >>>>>>>> some basic trouble:
> >>>>>>>>
> >>>>>>>> This works fine:
> >>>>>>>>
> >>>>>>>> import sqlite3
> >>>>>>>> sqlite3.connect("test.db")
> >>>>>>>>
> >>>>>>>> but this doesn't:
> >>>>>>>>
> >>>>>>>> import sqlite3
> >>>>>>>> sqlite3.connect("/home/dblank/test.db")
> >>>>>>>>
> >>>>>>>> Any ideas?
> >>>>>>>>
> >>>>>>>> -Doug
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> Ironpython-users mailing list
> >>>>>>>> Ironpython-users at python.org
> >>>>>>>> https://mail.python.org/mailman/listinfo/ironpython-users
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Website: http://earl-of-code.com
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Website: http://earl-of-code.com
> >>>>>
> >>>>> _______________________________________________
> >>>>> Ironpython-users mailing list
> >>>>> Ironpython-users at python.org
> >>>>> https://mail.python.org/mailman/listinfo/ironpython-users
> >>>>>
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > Website: http://earl-of-code.com
> >
> > _______________________________________________
> > Ironpython-users mailing list
> > Ironpython-users at python.org
> > https://mail.python.org/mailman/listinfo/ironpython-users
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140211/3ad8b69c/attachment.html>

From vano at mail.mipt.ru  Tue Feb 11 14:25:53 2014
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Tue, 11 Feb 2014 17:25:53 +0400
Subject: [Ironpython-users] IronPython SQLite interface
In-Reply-To: <CAAusYCgYq6gWqDwaDmnhnJ1osfsvcEHQ8augCbEDebRVOA8pHA@mail.gmail.com>
References: <CAAusYCgLrRFOhnCEmMxXsfXqdirRVjquM6N6F1T4FN=Wk1HHMg@mail.gmail.com>
 <CAPiUgVcyXxn0D21K0iS_Hg4rPU26AZUqz0Jqa400dP0_OFhd1w@mail.gmail.com>
 <CAAusYCi=h9t9aAsg_Dnu8zW7CFkYkd0PW-qP59rCZiYF73TqnQ@mail.gmail.com>
 <CAPiUgVfHcm5E=8JhBMYh-uw=FnDbe0=XzR_XntX7tsgk1eB5aQ@mail.gmail.com>
 <CAAusYCgT66iVMRq_m9-z___WKnk9HCy568ebf8U4bA68HfS+Yg@mail.gmail.com>
 <CAAusYCj9xGJBXUK44PW1WdMo4s5PW5v9H_PdnUvBGKujs2y6nQ@mail.gmail.com>
 <CAAusYCg8zv=MoAMJth27zS0SwWz-coS449jkmJ68jKR0r+mpgw@mail.gmail.com>
 <CAPiUgVc04tQ6hB_0dyjasaE9b86DPhcpEDAYYQQxkDr==uMBMA@mail.gmail.com>
 <CAF7AXFG1bBuS-=kxzzsA8XBx4SfiCfGSJY2iLita2Vc2-W3RQA@mail.gmail.com>
 <CAAusYCgYq6gWqDwaDmnhnJ1osfsvcEHQ8augCbEDebRVOA8pHA@mail.gmail.com>
Message-ID: <1413646938.20140211172553@mail.mipt.ru>

> Issue #173 with patch:

> https://github.com/IronLanguages/main/issues/173



> That works for me! I just removed the removal of the slash. I don't
> know what that will do for Windows users that had a path that started with "/".


> We distribute our own DLLs, so no rush for a new release.


> Thanks!


> -Doug


> On Tue, Feb 11, 2014 at 4:39 AM, Jeff Hardy <jdhardy at gmail.com> wrote:
>  
> On Mon, Feb 10, 2014 at 10:15 PM, Slide <slide.o.mix at gmail.com> wrote:
  >> That seems a little odd. Perhaps Jeff can shed some light on it, I think he
 >> wrote the SQLite adaptation layer.
>  
>  I did, but the funny code is part of the C#-sqlite library, which is a
>  port of the C library to C#. (if you're feeling brave, dig through the
>  code. It's ... interesting.) It looks like the author just assumed
>  Windows-ish systems in all cases, and Doug has to go and break
>  everything by trying to run it on Unix. :(
>
The file is indeed os_win_c.cs while the original library
(http://www.sqlite.org/cgi/src/tree?ci=trunk) has os_unix.c as well.

The relevant change in csharp-sqlite is
http://code.google.com/p/csharp-sqlite/source/detail?r=2fd39fa5400f6492b97617816c4f20a891b2e432&path=/Community.CsharpSqlite/src/os_win_c.cs
with a rather evasive message "Initial 3.7.7 changes".

It appears to be a bad port of the original sqlite diff 3.7.6.8->3.7.7
http://www.sqlite.org/cgi/src/fdiff?v1=24d72407a9055196&v2=eafcd6b91cf204a7&sbs=1

with related lines in it ultimately coming from changeset
http://www.sqlite.org/cgi/src/info/fe57a8f621
that clearly states "Have the xFullpath method in os_win.c discard the
initial "/" if a filename begins with "/X:", where X is any alphabetic character"
without explaining the cause.

The branch is called `uri' so I guess it's needed to handle URIs
like file:///C:/what/ya/mah/call/it.sqlite
(Windows kernel APIs treat (and always treated) '/' and '\' the same
(http://bytes.com/topic/python/answers/23123-when-did-windows-start-accepting-forward-slash-path-separator )).



So, the code was clearly intended for Windows only but ultimately,
it's csharp-sqlite's /bydlocode/ that broke things down in this
particular case.

  
>  I have no idea what that code is trying to do (context:
> 
> https://github.com/IronLanguages/main/blob/master/Languages/IronPython/IronPython.SQLite/c%23sqlite/os_win_c.cs#L2646);
>   it might be some vestigial part of SQLite that's not needed. Since //
>  seems OK I'm assuming that's the case.
>  
>  Doug, can you open an issue? And in the meantime, is using // an
>  acceptable workaround?
>  
>  - Jeff
>  

 >>
 >>
 >> On Mon, Feb 10, 2014 at 3:14 PM, Doug Blank <doug.blank at gmail.com> wrote:
 >>>
 >>> That must be the issue, as "//home/dblank/name.db" works:
 >>>
 >>> $ ipy.exe
 >>> IronPython 2.9.9a0 DEBUG (2.9.0.0) on Mono 4.0.30319.1 (64-bit)
 >>> Type "help", "copyright", "credits" or "license" for more information.
 >>> >>> import sqlite3
 >>> >>> sqlite3.connect("//home/dblank/test.db")
 >>> <Connection object at 0x000000000000002B>
 >>> >>>
 >>>
 >>> -Doug
 >>>
 >>>
 >>>
 >>> On Mon, Feb 10, 2014 at 5:12 PM, Doug Blank <doug.blank at gmail.com> wrote:
 >>>>
 >>>> After more testing, it appears that it only fails when the path begins
 >>>> with a slash "/".
 >>>>
 >>>> I see:
 >>>>
 >>>> ./c#sqlite/os_win_c.cs: ? ? ?if ( zConverted.StartsWith( "/" ) &&
 >>>> !zConverted.StartsWith( "//"
 >>>>
 >>>> I'll try a couple of more things...
 >>>>
 >>>> -Doug
 >>>>
 >>>>
 >>>> On Mon, Feb 10, 2014 at 9:19 AM, Doug Blank <doug.blank at gmail.com> wrote:
 >>>>>
 >>>>>
 >>>>> On Mon, Feb 10, 2014 at 8:47 AM, Slide <slide.o.mix at gmail.com> wrote:
 >>>>>>
 >>>>>> All the connect call does is this:
 >>>>>>
 >>>>>> int rc = Sqlite3.sqlite3_open(database, out this.db);
 >>>>>>
 >>>>>>
 >>>>>>
 >>>>>>
 >>>>>>
 >>>>>> if(rc != Sqlite3.SQLITE_OK)
 >>>>>> ? ? throw GetSqliteError(this.db, null);
 >>>>>>
 >>>>>>
 >>>>>>
 >>>>>>
 >>>>>>
 >>>>>>
 >>>>>> Which makes me thing that sqlite3_open is returning an error code. What
 >>>>>> happens if you do the same in CPython?
 >>>>>
 >>>>>
 >>>>> No error, creates a file of length 0, and works:
 >>>>>
 >>>>> $ python
 >>>>> Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
 >>>>> [GCC 4.8.1] on linux2
 >>>>> Type "help", "copyright", "credits" or "license" for more information.
 >>>>> >>> import sqlite3
 >>>>> >>> sqlite3.connect("/home/dblank/test.db")
 >>>>> <sqlite3.Connection object at 0x7fc3c21b6118>
 >>>>> >>>
 >>>>>
 >>>>> $ ls -al /home/dblank/test.db
 >>>>> -rw-r--r-- 1 dblank dblank 0 Feb 10 09:15 /home/dblank/test.db
 >>>>>
 >>>>> -Doug
 >>>>>
 >>>>>
 >>>>>>
 >>>>>>
 >>>>>>
 >>>>>> On Fri, Feb 7, 2014 at 6:01 AM, Doug Blank <doug.blank at gmail.com>
 >>>>>> wrote:
 >>>>>>>
 >>>>>>>
 >>>>>>> On Fri, Feb 7, 2014 at 7:47 AM, Slide <slide.o.mix at gmail.com> wrote:
 >>>>>>>>
 >>>>>>>> Do you get a specific error message?
 >>>>>>>
 >>>>>>>
 >>>>>>> Sorry, yes:
 >>>>>>>
 >>>>>>> Traceback (most recent call last):
 >>>>>>> ? File "<string>", line 1, in <module>
 >>>>>>> _sqlite3.OperationalError: unable to open database file
 >>>>>>>
 >>>>>>> Does this work for you? I get this error running latest ipy.exe (Mono
 >>>>>>> 2.10, Ubuntu 13.10) and IronPython in Calico.
 >>>>>>>
 >>>>>>> -Doug
 >>>>>>>
 >>>>>>>
 >>>>>>>>
 >>>>>>>>
 >>>>>>>>
 >>>>>>>> On Fri, Feb 7, 2014 at 5:37 AM, Doug Blank <doug.blank at gmail.com>
 >>>>>>>> wrote:
 >>>>>>>>>
 >>>>>>>>> Just recently discovered the sqlite3 port for IronPython, but having
 >>>>>>>>> some basic trouble:
 >>>>>>>>>
 >>>>>>>>> This works fine:
 >>>>>>>>>
 >>>>>>>>> import sqlite3
 >>>>>>>>> sqlite3.connect("test.db")
 >>>>>>>>>
 >>>>>>>>> but this doesn't:
 >>>>>>>>>
 >>>>>>>>> import sqlite3
 >>>>>>>>> sqlite3.connect("/home/dblank/test.db")
 >>>>>>>>>
 >>>>>>>>> Any ideas?
 >>>>>>>>>
 >>>>>>>>> -Doug
 >>>>>>>>>
 >>>>>>>>>
 >>>>>>>>> _______________________________________________
 >>>>>>>>> Ironpython-users mailing list
 >>>>>>>>> Ironpython-users at python.org
 >>>>>>>>> https://mail.python.org/mailman/listinfo/ironpython-users
 >>>>>>>>>
 >>>>>>>>
 >>>>>>>>
 >>>>>>>>
 >>>>>>>> --
 >>>>>>>> Website: http://earl-of-code.com
 >>>>>>>
 >>>>>>>
 >>>>>>
 >>>>>>
 >>>>>>
 >>>>>> --
 >>>>>> Website: http://earl-of-code.com
 >>>>>>
 >>>>>> _______________________________________________
 >>>>>> Ironpython-users mailing list
 >>>>>> Ironpython-users at python.org
 >>>>>> https://mail.python.org/mailman/listinfo/ironpython-users
 >>>>>>
 >>>>>
 >>>>
 >>>
 >>
 >>
 >>
 >> --
 >> Website: http://earl-of-code.com
 >>
 >> _______________________________________________
 >> Ironpython-users mailing list
 >> Ironpython-users at python.org
 >> https://mail.python.org/mailman/listinfo/ironpython-users
 >>
>  




-- 
Best regards,
 Ivan                            mailto:vano at mail.mipt.ru


From pablodalma93 at hotmail.com  Wed Feb 12 13:09:46 2014
From: pablodalma93 at hotmail.com (Pablo Dalmazzo)
Date: Wed, 12 Feb 2014 09:09:46 -0300
Subject: [Ironpython-users] =?cp1256?q?keeping_wrapping_of_blocks_after_cl?=
 =?cp1256?q?osing_for_Visual_Studio_2010/2012_with_asp=2Enet=FE?=
Message-ID: <BLU181-W84BBC5CC28382BE4A42CE5D0920@phx.gbl>

Hi there,
I was wondering if there is a way to keep the blocks of code wrapped after closing and reopen Visual Studio 2010/2012 when using IronPython with asp.net . This behavior of Visual Studio is kept after closing VS for the original languages (C#/VB.NET etc.) and since recently I managed to change the Windows registry to make aspx /aspx.py source code files nest like they do when the source code is C#/VB.NET , I thought may be there is something like that to solve this issue
Greetings 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140212/60c2bf6b/attachment.html>

From no_reply at codeplex.com  Sat Feb 15 09:23:33 2014
From: no_reply at codeplex.com (CodePlex)
Date: 15 Feb 2014 00:23:33 -0800
Subject: [Ironpython-users] IronPython, Daily Digest 2/14/2014
Message-ID: <CO1MSDNADMCDS2q6Dqa000283f0@discussions.codeplex.com>

Hi ironpython,

Here's your Daily Digest of new issues for project "IronPython".

In today's digest:ISSUES

1. [New comment] Importing .NET types causes ImportExceptions (zipimport handler)

----------------------------------------------

ISSUES

1. [New comment] Importing .NET types causes ImportExceptions (zipimport handler)
http://ironpython.codeplex.com/workitem/34602
User sopelt has commented on the issue:

"<p>Hi, sorry for the late reply. I see the issue with the constructor logic ... <br>For your question on &quot;only happens on the first import&quot;: I am no expert on the importer system, path_hooks ... and am not sure on some of the constraints and states of that system. For the same Engine/PythonContext it looks like only the first .NET import causes the exceptions (see attached source file) while subsequent imports seem to work without exceptions.</p><p>As far as I understand once instantiated the `zipimport` is kept around and does not need to be constructed again? Might it be possible to construct/warm up zipimport in a controlled fashion to avoid the construction using a path that cannot be handled (i.e. is not a zip file) to avoid having to break zip importing all together?</p><p>Could you give me a hint on how to remove only the zipimport from path_hooks?</p>"
----------------------------------------------



----------------------------------------------
You are receiving this email because you subscribed to notifications on CodePlex.

To report a bug, request a feature, or add a comment, visit IronPython Issue Tracker. You can unsubscribe or change your issue notification settings on CodePlex.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140215/49255378/attachment.html>

From m.schaber at codesys.com  Tue Feb 25 13:43:06 2014
From: m.schaber at codesys.com (Markus Schaber)
Date: Tue, 25 Feb 2014 12:43:06 +0000
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
	than in cPython
Message-ID: <727D8E16AE957149B447FE368139F2B53987CF85@SERVER10>

Hi,

I forgot to add the attachment...

Best regards

Markus Schaber

CODESYS(r) a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions

3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915

-------------- next part --------------
A non-text attachment was scrubbed...
Name: crc-fixed.py
Type: application/octet-stream
Size: 481 bytes
Desc: crc-fixed.py
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140225/ef7ce650/attachment.obj>

From m.schaber at codesys.com  Tue Feb 25 13:38:36 2014
From: m.schaber at codesys.com (Markus Schaber)
Date: Tue, 25 Feb 2014 12:38:36 +0000
Subject: [Ironpython-users] Hashing a directory is magnitudes slower than in
	cPython
Message-ID: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>

Hi,

A coworker just consulted me on a performance problem of IronPython vs. cPython.

The attached test script reproduces the problem. 

On cPython 2.7.6, it needs about 1.5 seconds on our test directory (once the OS disk cache is hot), and cPython 3.3 needs about 1.7 seconds, while IronPython needs more than 10 minutes(!).

C:\Users\m.schaber>c:\Python33\python.exe d:\crc-fixed.py "c:\Test Specifications\AutotestRepository"
Examining: c:\Test Specifications\AutotestRepository
Checksum: f7ff573eb219b0ce79bd204e3625b5e2
Seconds: 1.721932103156255

C:\Users\m.schaber>c:\Python33\python.exe d:\crc-fixed.py "c:\Test Specifications\AutotestRepository"
Examining: c:\Test Specifications\AutotestRepository
Checksum: f7ff573eb219b0ce79bd204e3625b5e2
Seconds: 1.7523154039322837

C:\Users\m.schaber>python d:\crc-fixed.py "c:\Test Specifications\AutotestRepository"
Examining: c:\Test Specifications\AutotestRepository
Checksum: f7ff573eb219b0ce79bd204e3625b5e2
Seconds: 1.44541429616

C:\Users\m.schaber>python d:\crc-fixed.py "c:\Test Specifications\AutotestRepository"
Examining: c:\Test Specifications\AutotestRepository
Checksum: f7ff573eb219b0ce79bd204e3625b5e2
Seconds: 1.40604227074

C:\Users\m.schaber>"c:\Program Files (x86)\IronPython 2.7\ipy.exe" d:\crc.py "c:\Test Specifications\AutotestRepository"
Examining: c:\Test Specifications\AutotestRepository
Checksum: f7ff573eb219b0ce79bd204e3625b5e2
Seconds: 602.745100044

C:\Users\m.schaber>"c:\Program Files (x86)\IronPython 2.7\ipy.exe" d:\crc.py "c:\Test Specifications\AutotestRepository"
Examining: c:\Test Specifications\AutotestRepository
Checksum: f7ff573eb219b0ce79bd204e3625b5e2
Seconds: 607.252915722


My first guess was that it's a problem of the cPython 8 Bit strings vs. .NET strings, which cause expensive conversions. (I also guess that a Python 3 based IronPython will fix this issue.)

One idea to fix this may be to add an overload to MD5Type.update() which directly accepts strings (and maybe one accepting byte arrays), to avoid the call to the conversion functions.

On a closer look, there's the additional (and IMHO much worse) problem that the update() method seems not to work incrementally:

private void update(IList<byte> newBytes) {
    byte[] updatedBytes = new byte[_bytes.Length + newBytes.Count];
    Array.Copy(_bytes, updatedBytes, _bytes.Length);
    newBytes.CopyTo(updatedBytes, _bytes.Length);
    _bytes = updatedBytes;
    _hash = GetHasher().ComputeHash(_bytes);
}

In our use-case, this means that every file which is read leads to a reallocation and copying and recalculation of the MD5 sum of all the data which was read until now. This is suboptimal from memory and performance perspective.

I'm not an expert on the .NET crypto APIs, but I guess there should be some incremental API available there which could be exploited.

If not, we could try to find a suitable pure .NET implementation like http://archive.msdn.microsoft.com/SilverlightMD5.

A less intrusive workaround may be to collect the bytes using a MemoryStream, and feeding it to ComputeHash() only on demand, when someone actually requests the hash result via digest() or hexdigest().


PS: Our use-case of MD5 is purely for technical data integrity, not against malicious users, cryptographic security is not required.

Best regards

Markus Schaber

CODESYS(r) a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions

3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915


From doug.blank at gmail.com  Tue Feb 25 14:03:00 2014
From: doug.blank at gmail.com (Doug Blank)
Date: Tue, 25 Feb 2014 08:03:00 -0500
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <727D8E16AE957149B447FE368139F2B53987CF85@SERVER10>
References: <727D8E16AE957149B447FE368139F2B53987CF85@SERVER10>
Message-ID: <CAAusYCjzGSi-qEb5wXfEMbRSw3AmnmAYAeT_OYZFt-NV+f2Xdw@mail.gmail.com>

I think I just wrote the relevant parts in C#, in a slightly different form:

public string sign(List<string> list) {
            hmac.Initialize();
            foreach (string item in list) {
byte [] sourcebytes = Encoding.UTF8.GetBytes(item);
                hmac.TransformBlock(sourcebytes, 0, sourcebytes.Length,
null, 0);
            }
            hmac.TransformFinalBlock(new byte [0], 0, 0);
    return BitConverter.ToString(hmac.Hash).Replace("-", "").ToLower();
}

Maybe that will be useful, or save someone a few minutes in researching.
Perhaps this can be sped up as well.

-Doug


On Tue, Feb 25, 2014 at 7:43 AM, Markus Schaber <m.schaber at codesys.com>wrote:

> Hi,
>
> I forgot to add the attachment...
>
> Best regards
>
> Markus Schaber
>
> CODESYS(r) a trademark of 3S-Smart Software Solutions GmbH
>
> Inspiring Automation Solutions
>
> 3S-Smart Software Solutions GmbH
> Dipl.-Inf. Markus Schaber | Product Development Core Technology
> Memminger Str. 151 | 87439 Kempten | Germany
> Tel. +49-831-54031-979 | Fax +49-831-54031-50
>
> E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS
> store: http://store.codesys.com
> CODESYS forum: http://forum.codesys.com
>
> Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner |
> Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
>
>
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140225/50af1f00/attachment.html>

From jdhardy at gmail.com  Tue Feb 25 17:53:33 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Tue, 25 Feb 2014 16:53:33 +0000
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
Message-ID: <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>

On Tue, Feb 25, 2014 at 12:38 PM, Markus Schaber <m.schaber at codesys.com> wrote:
> Hi,
>
> A coworker just consulted me on a performance problem of IronPython vs. cPython.
>
> ... snip ...
>
> On a closer look, there's the additional (and IMHO much worse) problem that the update() method seems not to work incrementally:
>
> private void update(IList<byte> newBytes) {
>     byte[] updatedBytes = new byte[_bytes.Length + newBytes.Count];
>     Array.Copy(_bytes, updatedBytes, _bytes.Length);
>     newBytes.CopyTo(updatedBytes, _bytes.Length);
>     _bytes = updatedBytes;
>     _hash = GetHasher().ComputeHash(_bytes);
> }
>
> In our use-case, this means that every file which is read leads to a reallocation and copying and recalculation of the MD5 sum of all the data which was read until now. This is suboptimal from memory and performance perspective.
>
> I'm not an expert on the .NET crypto APIs, but I guess there should be some incremental API available there which could be exploited.

http://ironpython.codeplex.com/workitem/34022

I've also CC'd Emmanuel Chomarat, who was investigating a fix for
this. Unfortunately I don't think there's an easy solution based on
how the .NET APIs are constructed. Quoting from Emmanuel's email to me
a while back:

"I am now using TransformBlock / TransformBlockFinal to compute the
current hash with a linear complexity ( whereas we had before n**2)
but I am still facing an issue.
First we need to have a copy operator, this is not possible because we
can not share the hash instance between two objects in .net, the only
way to make it consistent with what python is doing is by keeping a
copy of the full data in MEMORY in order to create a new instance with
these data when copy is called.
The second thing is that digest can be called several times in python
with some new data added/updated to the hash , in C# as soon as
TransformBlockFinal has been called once we can not anymore add more
data to the stream. Once again I have been able to use the same
previous technic but at a memory cost + computation cost if we call
serveral times digest/hexdigest.

I don't see any to escape this pb with MS api that does not expose
internal states as the underlying md5 lib in python does."

Basically, there's a mismatch between what .NET provides and what
Python needs for perfect compatibility. Keeping all data in memory is
not desirable, but neither is failing some operations. And I would
*really* prefer not to have to reimplement all of the cryptographic
hash functions Python has.

One option is to default to not buffering and failing on certain
operations, and offer a constructor flag that enables buffering to
allow the otherwise-impossible operations. Not my favourite idea, but
workable.

- Jeff

From curt at hagenlocher.org  Tue Feb 25 17:58:52 2014
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Tue, 25 Feb 2014 08:58:52 -0800
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
 <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
Message-ID: <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>

"Basically, there's a mismatch between what .NET provides and what Python
needs for perfect compatibility."

Yes. I think I remember implementing this and that's exactly the problem I
ran into. I think we looked into incorporating a modified version of the
BCL code directly into IronPython, but at least in those days, that was a
pretty hard thing to get done. We ran into a similar issue when
implementing the compression API.

You could get around the problem in the client code with an "if
sys.platform == 'cli'" and then use the .NET classes directly.



On Tue, Feb 25, 2014 at 8:53 AM, Jeff Hardy <jdhardy at gmail.com> wrote:

> On Tue, Feb 25, 2014 at 12:38 PM, Markus Schaber <m.schaber at codesys.com>
> wrote:
> > Hi,
> >
> > A coworker just consulted me on a performance problem of IronPython vs.
> cPython.
> >
> > ... snip ...
> >
> > On a closer look, there's the additional (and IMHO much worse) problem
> that the update() method seems not to work incrementally:
> >
> > private void update(IList<byte> newBytes) {
> >     byte[] updatedBytes = new byte[_bytes.Length + newBytes.Count];
> >     Array.Copy(_bytes, updatedBytes, _bytes.Length);
> >     newBytes.CopyTo(updatedBytes, _bytes.Length);
> >     _bytes = updatedBytes;
> >     _hash = GetHasher().ComputeHash(_bytes);
> > }
> >
> > In our use-case, this means that every file which is read leads to a
> reallocation and copying and recalculation of the MD5 sum of all the data
> which was read until now. This is suboptimal from memory and performance
> perspective.
> >
> > I'm not an expert on the .NET crypto APIs, but I guess there should be
> some incremental API available there which could be exploited.
>
> http://ironpython.codeplex.com/workitem/34022
>
> I've also CC'd Emmanuel Chomarat, who was investigating a fix for
> this. Unfortunately I don't think there's an easy solution based on
> how the .NET APIs are constructed. Quoting from Emmanuel's email to me
> a while back:
>
> "I am now using TransformBlock / TransformBlockFinal to compute the
> current hash with a linear complexity ( whereas we had before n**2)
> but I am still facing an issue.
> First we need to have a copy operator, this is not possible because we
> can not share the hash instance between two objects in .net, the only
> way to make it consistent with what python is doing is by keeping a
> copy of the full data in MEMORY in order to create a new instance with
> these data when copy is called.
> The second thing is that digest can be called several times in python
> with some new data added/updated to the hash , in C# as soon as
> TransformBlockFinal has been called once we can not anymore add more
> data to the stream. Once again I have been able to use the same
> previous technic but at a memory cost + computation cost if we call
> serveral times digest/hexdigest.
>
> I don't see any to escape this pb with MS api that does not expose
> internal states as the underlying md5 lib in python does."
>
> Basically, there's a mismatch between what .NET provides and what
> Python needs for perfect compatibility. Keeping all data in memory is
> not desirable, but neither is failing some operations. And I would
> *really* prefer not to have to reimplement all of the cryptographic
> hash functions Python has.
>
> One option is to default to not buffering and failing on certain
> operations, and offer a constructor flag that enables buffering to
> allow the otherwise-impossible operations. Not my favourite idea, but
> workable.
>
> - Jeff
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140225/fe5dafe2/attachment.html>

From emmanuel.chomarat at gmail.com  Tue Feb 25 18:08:43 2014
From: emmanuel.chomarat at gmail.com (Emmanuel Chomarat)
Date: Tue, 25 Feb 2014 18:08:43 +0100
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
 <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
 <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
Message-ID: <CAM4Ce1u1v_nqwtL5Yo2yFjyMJV+eUeUwEQdCX=XZe17JiwBWbw@mail.gmail.com>

The main issue from what I remember is related to the fact that we can
interleave update and digest calls in CPython. This is impossible in .Net
(at least with the provided API). The workaround to this issue is to store
a local buffer of everything that has already been hashed, in order to
rehash it if someone ask for digest in the middle of the operation. The
exec time was very good and there were now degradation on the execution
time, the problem was that the caching was growing linearly with the amount
of data. If we are not caching data there is a way to make it quick and
easy on the memory but it does not then follow python API (
http://docs.python.org/2/library/hashlib.html)


On Tue, Feb 25, 2014 at 5:58 PM, Curt Hagenlocher <curt at hagenlocher.org>wrote:

> "Basically, there's a mismatch between what .NET provides and what Python
> needs for perfect compatibility."
>
> Yes. I think I remember implementing this and that's exactly the problem I
> ran into. I think we looked into incorporating a modified version of the
> BCL code directly into IronPython, but at least in those days, that was a
> pretty hard thing to get done. We ran into a similar issue when
> implementing the compression API.
>
> You could get around the problem in the client code with an "if
> sys.platform == 'cli'" and then use the .NET classes directly.
>
>
>
> On Tue, Feb 25, 2014 at 8:53 AM, Jeff Hardy <jdhardy at gmail.com> wrote:
>
>> On Tue, Feb 25, 2014 at 12:38 PM, Markus Schaber <m.schaber at codesys.com>
>> wrote:
>> > Hi,
>> >
>> > A coworker just consulted me on a performance problem of IronPython vs.
>> cPython.
>> >
>> > ... snip ...
>> >
>> > On a closer look, there's the additional (and IMHO much worse) problem
>> that the update() method seems not to work incrementally:
>> >
>> > private void update(IList<byte> newBytes) {
>> >     byte[] updatedBytes = new byte[_bytes.Length + newBytes.Count];
>> >     Array.Copy(_bytes, updatedBytes, _bytes.Length);
>> >     newBytes.CopyTo(updatedBytes, _bytes.Length);
>> >     _bytes = updatedBytes;
>> >     _hash = GetHasher().ComputeHash(_bytes);
>> > }
>> >
>> > In our use-case, this means that every file which is read leads to a
>> reallocation and copying and recalculation of the MD5 sum of all the data
>> which was read until now. This is suboptimal from memory and performance
>> perspective.
>> >
>> > I'm not an expert on the .NET crypto APIs, but I guess there should be
>> some incremental API available there which could be exploited.
>>
>> http://ironpython.codeplex.com/workitem/34022
>>
>> I've also CC'd Emmanuel Chomarat, who was investigating a fix for
>> this. Unfortunately I don't think there's an easy solution based on
>> how the .NET APIs are constructed. Quoting from Emmanuel's email to me
>> a while back:
>>
>> "I am now using TransformBlock / TransformBlockFinal to compute the
>> current hash with a linear complexity ( whereas we had before n**2)
>> but I am still facing an issue.
>> First we need to have a copy operator, this is not possible because we
>> can not share the hash instance between two objects in .net, the only
>> way to make it consistent with what python is doing is by keeping a
>> copy of the full data in MEMORY in order to create a new instance with
>> these data when copy is called.
>> The second thing is that digest can be called several times in python
>> with some new data added/updated to the hash , in C# as soon as
>> TransformBlockFinal has been called once we can not anymore add more
>> data to the stream. Once again I have been able to use the same
>> previous technic but at a memory cost + computation cost if we call
>> serveral times digest/hexdigest.
>>
>> I don't see any to escape this pb with MS api that does not expose
>> internal states as the underlying md5 lib in python does."
>>
>> Basically, there's a mismatch between what .NET provides and what
>> Python needs for perfect compatibility. Keeping all data in memory is
>> not desirable, but neither is failing some operations. And I would
>> *really* prefer not to have to reimplement all of the cryptographic
>> hash functions Python has.
>>
>> One option is to default to not buffering and failing on certain
>> operations, and offer a constructor flag that enables buffering to
>> allow the otherwise-impossible operations. Not my favourite idea, but
>> workable.
>>
>> - Jeff
>> _______________________________________________
>> Ironpython-users mailing list
>> Ironpython-users at python.org
>> https://mail.python.org/mailman/listinfo/ironpython-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140225/a5b966a0/attachment.html>

From no_reply at codeplex.com  Wed Feb 26 09:22:31 2014
From: no_reply at codeplex.com (CodePlex)
Date: 26 Feb 2014 00:22:31 -0800
Subject: [Ironpython-users] IronPython, Daily Digest 2/25/2014
Message-ID: <CO1MSDNADMCDS2C8wSA00007ee6@discussions.codeplex.com>

Hi ironpython,

Here's your Daily Digest of new issues for project "IronPython".

In today's digest:ISSUES

1. [New comment] Include some way to discover IronPython path
2. [New issue] ast.parse raises SystemError on "yield"

----------------------------------------------

ISSUES

1. [New comment] Include some way to discover IronPython path
http://ironpython.codeplex.com/workitem/34692
User javierpr19 has commented on the issue:

"<p>Hi,</p><p>Now, I have IronPython 2.7.4 installed over Windows 8.1 and I don't have this entry.<br>I have reinstalled IronPython with the same resultt.</p><p>Any idea?</p><p>Thanks</p>"-----------------

2. [New issue] ast.parse raises SystemError on "yield"
http://ironpython.codeplex.com/workitem/35001
User Demon has proposed the issue:

"ast.parse raises SystemError on "yield"

Reproduction code:

import ast
ast.parse("def f(): yield")

Result:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files (x86)\IronPython 2.7\Lib\ast.py", line 37, in parse
SystemError: Object reference not set to an instance of an object."
----------------------------------------------



----------------------------------------------
You are receiving this email because you subscribed to notifications on CodePlex.

To report a bug, request a feature, or add a comment, visit IronPython Issue Tracker. You can unsubscribe or change your issue notification settings on CodePlex.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140226/3778a4be/attachment.html>

From no_reply at codeplex.com  Thu Feb 27 09:23:42 2014
From: no_reply at codeplex.com (CodePlex)
Date: 27 Feb 2014 00:23:42 -0800
Subject: [Ironpython-users] IronPython, Daily Digest 2/26/2014
Message-ID: <CO1MSDNADMCDS2oIIzQ0000a583@discussions.codeplex.com>

Hi ironpython,

Here's your Daily Digest of new issues for project "IronPython".

In today's digest:ISSUES

1. [New comment] Include some way to discover IronPython path
2. [New comment] Include some way to discover IronPython path
3. [New comment] Include some way to discover IronPython path

----------------------------------------------

ISSUES

1. [New comment] Include some way to discover IronPython path
http://ironpython.codeplex.com/workitem/34692
User jdhardy has commented on the issue:

"<p>Not sure. I have 8.1 installed at home; I'll check to see if it's there when I can.</p>"-----------------

2. [New comment] Include some way to discover IronPython path
http://ironpython.codeplex.com/workitem/34692
User jdhardy has commented on the issue:

"<p>You're most likely a victim of [registry virtualization](http://msdn.microsoft.com/en-us/library/windows/desktop/aa965884(v=vs.85).aspx). The key is probably under `HKLM\SOFTWARE\Wow6432Node\IronPython\2.7\InstallPath`, because IronPython uses a 32-bit installer. If you're in a 32-bit process that gets mapped to `HKLM\SOFTWARE\IronPython\2.7\InstallPath`, but in a 64-bit process you have to go into the `Wow6432Node` instead.</p>"-----------------

3. [New comment] Include some way to discover IronPython path
http://ironpython.codeplex.com/workitem/34692
User jdhardy has commented on the issue:

"<p>Oops, wrong registry virtualization link. Try http://msdn.microsoft.com/en-us/library/windows/desktop/ms724072(v=vs.85).aspx instead.</p>"
----------------------------------------------



----------------------------------------------
You are receiving this email because you subscribed to notifications on CodePlex.

To report a bug, request a feature, or add a comment, visit IronPython Issue Tracker. You can unsubscribe or change your issue notification settings on CodePlex.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140227/f88b9564/attachment.html>

From m.schaber at codesys.com  Thu Feb 27 12:11:45 2014
From: m.schaber at codesys.com (Markus Schaber)
Date: Thu, 27 Feb 2014 11:11:45 +0000
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
 <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
 <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
Message-ID: <727D8E16AE957149B447FE368139F2B539B53862@SERVER10>

Hi,

I'm just trying to sum it up:

1) The current code:
   - High memory usage. 
   - High load on the large object heap.
   - Limited by the available amount of memory (which might be considered a violation of the Python API). 
   - High CPU usage when used incrementally (quadratic to the number of blocks added).

2) Optimizing with MemoryStream and lazy calculation:
   - High memory usage.
   - High load on the large object heap.
   - Limited by the available amount of memory (which might be considered a violation of the Python API).
   + Optimal CPU usage when the hash is only fetched once.
   ? Better than current code, but still not optimal when hash is incrementally fetched several times.
 
3) Optimizing with jagged arrays and lazy calculation:
   - High memory usage.
   + Improved or no impact on the large object heap (depending on the exact implementation)
   - Limited by the available amount of memory (which might be considered a violation of the Python API).
   + Optimal CPU usage when the hash is only fetched once.
   ? Better than current code, but still not optimal when hash is incrementally fetched several times.

4) Using the existing .NET incremental APIs
   + Low, constant memory usage.
   + No impact on the large object heap.
   + No limit of data length by the amount of memory.
   + Optimal CPU usage when the hash is only fetched once.
   - Breaks when hash is incrementally fetched several times (which likely is a violation of the Python API).

5) Finding or porting a different Hash implementation in C#:
   + Low, constant memory usage.
   + No impact on the large object heap.
   + No limit of data length by the amount of memory.
   + Optimal CPU usage when the hash is only fetched once.
   + Optimal CPU usage when the hash is incrementally fetched several times.

I've a local prototype implemented for 2), but I'm not sure whether that's the best way to go...

Maybe we should google for purely managed implementations of the hash codes with a sensible license...

Best regards

Markus Schaber

CODESYS? a trademark of 3S-Smart Software Solutions GmbH 

Inspiring Automation Solutions 
________________________________________
3S-Smart Software Solutions GmbH 
Dipl.-Inf. Markus Schaber | Product Development Core Technology 
Memminger Str. 151 | 87439 Kempten | Germany 
Tel. +49-831-54031-979 | Fax +49-831-54031-50 

E-Mail: m.schaber at codesys.com | Web: codesys.com | CODESYS store: store.codesys.com 
CODESYS forum: forum.codesys.com 

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915 
Von: Curt Hagenlocher [mailto:curt at hagenlocher.org] 
Gesendet: Dienstag, 25. Februar 2014 17:59
An: Jeff Hardy
Cc: Markus Schaber; Discussion of IronPython; Emmanuel Chomarat
Betreff: Re: [Ironpython-users] Hashing a directory is magnitudes slower than in cPython

"Basically, there's a mismatch between what .NET provides and what Python needs for perfect compatibility."

Yes. I think I remember implementing this and that's exactly the problem I ran into. I think we looked into incorporating a modified version of the BCL code directly into IronPython, but at least in those days, that was a pretty hard thing to get done. We ran into a similar issue when implementing the compression API.

You could get around the problem in the client code with an "if sys.platform == 'cli'" and then use the .NET classes directly.


On Tue, Feb 25, 2014 at 8:53 AM, Jeff Hardy <jdhardy at gmail.com> wrote:
On Tue, Feb 25, 2014 at 12:38 PM, Markus Schaber <m.schaber at codesys.com> wrote:
> Hi,
>
> A coworker just consulted me on a performance problem of IronPython vs. cPython.
>
> ... snip ...
>
> On a closer look, there's the additional (and IMHO much worse) problem that the update() method seems not to work incrementally:
>
> private void update(IList<byte> newBytes) {
> ? ? byte[] updatedBytes = new byte[_bytes.Length + newBytes.Count];
> ? ? Array.Copy(_bytes, updatedBytes, _bytes.Length);
> ? ? newBytes.CopyTo(updatedBytes, _bytes.Length);
> ? ? _bytes = updatedBytes;
> ? ? _hash = GetHasher().ComputeHash(_bytes);
> }
>
> In our use-case, this means that every file which is read leads to a reallocation and copying and recalculation of the MD5 sum of all the data which was read until now. This is suboptimal from memory and performance perspective.
>
> I'm not an expert on the .NET crypto APIs, but I guess there should be some incremental API available there which could be exploited.
http://ironpython.codeplex.com/workitem/34022

I've also CC'd Emmanuel Chomarat, who was investigating a fix for
this. Unfortunately I don't think there's an easy solution based on
how the .NET APIs are constructed. Quoting from Emmanuel's email to me
a while back:

"I am now using TransformBlock / TransformBlockFinal to compute the
current hash with a linear complexity ( whereas we had before n**2)
but I am still facing an issue.
First we need to have a copy operator, this is not possible because we
can not share the hash instance between two objects in .net, the only
way to make it consistent with what python is doing is by keeping a
copy of the full data in MEMORY in order to create a new instance with
these data when copy is called.
The second thing is that digest can be called several times in python
with some new data added/updated to the hash , in C# as soon as
TransformBlockFinal has been called once we can not anymore add more
data to the stream. Once again I have been able to use the same
previous technic but at a memory cost + computation cost if we call
serveral times digest/hexdigest.

I don't see any to escape this pb with MS api that does not expose
internal states as the underlying md5 lib in python does."

Basically, there's a mismatch between what .NET provides and what
Python needs for perfect compatibility. Keeping all data in memory is
not desirable, but neither is failing some operations. And I would
*really* prefer not to have to reimplement all of the cryptographic
hash functions Python has.

One option is to default to not buffering and failing on certain
operations, and offer a constructor flag that enables buffering to
allow the otherwise-impossible operations. Not my favourite idea, but
workable.

- Jeff
_______________________________________________
Ironpython-users mailing list
Ironpython-users at python.org
https://mail.python.org/mailman/listinfo/ironpython-users


From jdhardy at gmail.com  Thu Feb 27 13:27:48 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Thu, 27 Feb 2014 12:27:48 +0000
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <727D8E16AE957149B447FE368139F2B539B53862@SERVER10>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
 <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
 <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B53862@SERVER10>
Message-ID: <CAF7AXFGRsN3_dEogmt_NPzMrK_FDaRx_9Bq-T5+gu8J-+3ioMw@mail.gmail.com>

On Thu, Feb 27, 2014 at 11:11 AM, Markus Schaber <m.schaber at codesys.com> wrote:
> Hi,
>
> I'm just trying to sum it up:
>
> 1) The current code:
>    - High memory usage.
>    - High load on the large object heap.
>    - Limited by the available amount of memory (which might be considered a violation of the Python API).
>    - High CPU usage when used incrementally (quadratic to the number of blocks added).
>
> 2) Optimizing with MemoryStream and lazy calculation:
>    - High memory usage.
>    - High load on the large object heap.
>    - Limited by the available amount of memory (which might be considered a violation of the Python API).
>    + Optimal CPU usage when the hash is only fetched once.
>    ? Better than current code, but still not optimal when hash is incrementally fetched several times.
>
> 3) Optimizing with jagged arrays and lazy calculation:
>    - High memory usage.
>    + Improved or no impact on the large object heap (depending on the exact implementation)
>    - Limited by the available amount of memory (which might be considered a violation of the Python API).
>    + Optimal CPU usage when the hash is only fetched once.
>    ? Better than current code, but still not optimal when hash is incrementally fetched several times.
>
> 4) Using the existing .NET incremental APIs
>    + Low, constant memory usage.
>    + No impact on the large object heap.
>    + No limit of data length by the amount of memory.
>    + Optimal CPU usage when the hash is only fetched once.
>    - Breaks when hash is incrementally fetched several times (which likely is a violation of the Python API).
>
> 5) Finding or porting a different Hash implementation in C#:
>    + Low, constant memory usage.
>    + No impact on the large object heap.
>    + No limit of data length by the amount of memory.
>    + Optimal CPU usage when the hash is only fetched once.
>    + Optimal CPU usage when the hash is incrementally fetched several times.
>
> I've a local prototype implemented for 2), but I'm not sure whether that's the best way to go...

Good analysis!

My preference would be for (4), raising an exception if .update() is
called after .digest(), or .copy() is called at all. As a fallback, an
extra parameter to hashlib.new (&c) that triggers (2), for cases where
its needed - I can't say for sure, but I would think calling .update()
after .digest() would be rare, and so would .copy() (damn you Google
for shutting down code search). At least then the common case is fast
and edge cases are (usually) possible.

>
> Maybe we should google for purely managed implementations of the hash codes with a sensible license...
>

There seems to be for MD5 and SHA1 but not SHA2 or RIPEMD160. They
could be ported from the public domain Crypto++ library, but that
seems like a lot of work for an edge case.

- Jeff

From m.schaber at codesys.com  Thu Feb 27 14:10:17 2014
From: m.schaber at codesys.com (Markus Schaber)
Date: Thu, 27 Feb 2014 13:10:17 +0000
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <CAF7AXFGRsN3_dEogmt_NPzMrK_FDaRx_9Bq-T5+gu8J-+3ioMw@mail.gmail.com>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
 <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
 <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B53862@SERVER10>
 <CAF7AXFGRsN3_dEogmt_NPzMrK_FDaRx_9Bq-T5+gu8J-+3ioMw@mail.gmail.com>
Message-ID: <727D8E16AE957149B447FE368139F2B539B55A04@SERVER10>

Hi,

Von: Jeff Hardy [mailto:jdhardy at gmail.com]
> On Thu, Feb 27, 2014 at 11:11 AM, Markus Schaber <m.schaber at codesys.com>
> wrote:
> > Hi,
> >
> > I'm just trying to sum it up:
> >
> > 1) The current code:
> >    - High memory usage.
> >    - High load on the large object heap.
> >    - Limited by the available amount of memory (which might be considered a
> violation of the Python API).
> >    - High CPU usage when used incrementally (quadratic to the number of
> blocks added).
> >
> > 2) Optimizing with MemoryStream and lazy calculation:
> >    - High memory usage.
> >    - High load on the large object heap.
> >    - Limited by the available amount of memory (which might be considered a
> violation of the Python API).
> >    + Optimal CPU usage when the hash is only fetched once.
> >    ? Better than current code, but still not optimal when hash is
> incrementally fetched several times.
> >
> > 3) Optimizing with jagged arrays and lazy calculation:
> >    - High memory usage.
> >    + Improved or no impact on the large object heap (depending on the exact
> implementation)
> >    - Limited by the available amount of memory (which might be considered a
> violation of the Python API).
> >    + Optimal CPU usage when the hash is only fetched once.
> >    ? Better than current code, but still not optimal when hash is
> incrementally fetched several times.
> >
> > 4) Using the existing .NET incremental APIs
> >    + Low, constant memory usage.
> >    + No impact on the large object heap.
> >    + No limit of data length by the amount of memory.
> >    + Optimal CPU usage when the hash is only fetched once.
> >    - Breaks when hash is incrementally fetched several times (which likely
> is a violation of the Python API).
> >
> > 5) Finding or porting a different Hash implementation in C#:
> >    + Low, constant memory usage.
> >    + No impact on the large object heap.
> >    + No limit of data length by the amount of memory.
> >    + Optimal CPU usage when the hash is only fetched once.
> >    + Optimal CPU usage when the hash is incrementally fetched several times.
> >
> > I've a local prototype implemented for 2), but I'm not sure whether that's
> > the best way to go...
> 
> Good analysis!
> 
> My preference would be for (4), raising an exception if .update() is called
> after .digest(), or .copy() is called at all. As a fallback, an extra
> parameter to hashlib.new (&c) that triggers (2), for cases where its needed -
> I can't say for sure, but I would think calling .update() after .digest()
> would be rare, and so would .copy() (damn you Google for shutting down code
> search). At least then the common case is fast and edge cases are (usually)
> possible.

Do you think asking on some cPython lists could give usable feedback how
common it is to call copy() or to continue feeding data after calling
digest()?

> > Maybe we should google for purely managed implementations of the hash codes
> > with a sensible license...
> 
> There seems to be for MD5 and SHA1 but not SHA2 or RIPEMD160. They could be
> ported from the public domain Crypto++ library, but that seems like a lot of
> work for an edge case.

Yes, that seems to be a lot of work.

On the other hand, it's the 100% solution. :-)

Best regards

Markus Schaber

CODESYS? a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions

3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
CODESYS forum: http://forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915


From pawel.jasinski at gmail.com  Thu Feb 27 20:16:58 2014
From: pawel.jasinski at gmail.com (Pawel Jasinski)
Date: Thu, 27 Feb 2014 20:16:58 +0100
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <727D8E16AE957149B447FE368139F2B539B55A04@SERVER10>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
 <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
 <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B53862@SERVER10>
 <CAF7AXFGRsN3_dEogmt_NPzMrK_FDaRx_9Bq-T5+gu8J-+3ioMw@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B55A04@SERVER10>
Message-ID: <CA+7con=_JubQPy8XtMPSW+_6xM65BREOEJBOcprtq6wWf2mhcQ@mail.gmail.com>

Is there any reason not to use code out of mono?
It looks like it supports SHA2 and RIPEMD160.
https://bugzilla.xamarin.com/show_bug.cgi?id=11703

On Thu, Feb 27, 2014 at 2:10 PM, Markus Schaber <m.schaber at codesys.com> wrote:
> Hi,
>
> Von: Jeff Hardy [mailto:jdhardy at gmail.com]
>> On Thu, Feb 27, 2014 at 11:11 AM, Markus Schaber <m.schaber at codesys.com>
>> wrote:
>> > Hi,
>> >
>> > I'm just trying to sum it up:
>> >
>> > 1) The current code:
>> >    - High memory usage.
>> >    - High load on the large object heap.
>> >    - Limited by the available amount of memory (which might be considered a
>> violation of the Python API).
>> >    - High CPU usage when used incrementally (quadratic to the number of
>> blocks added).
>> >
>> > 2) Optimizing with MemoryStream and lazy calculation:
>> >    - High memory usage.
>> >    - High load on the large object heap.
>> >    - Limited by the available amount of memory (which might be considered a
>> violation of the Python API).
>> >    + Optimal CPU usage when the hash is only fetched once.
>> >    ? Better than current code, but still not optimal when hash is
>> incrementally fetched several times.
>> >
>> > 3) Optimizing with jagged arrays and lazy calculation:
>> >    - High memory usage.
>> >    + Improved or no impact on the large object heap (depending on the exact
>> implementation)
>> >    - Limited by the available amount of memory (which might be considered a
>> violation of the Python API).
>> >    + Optimal CPU usage when the hash is only fetched once.
>> >    ? Better than current code, but still not optimal when hash is
>> incrementally fetched several times.
>> >
>> > 4) Using the existing .NET incremental APIs
>> >    + Low, constant memory usage.
>> >    + No impact on the large object heap.
>> >    + No limit of data length by the amount of memory.
>> >    + Optimal CPU usage when the hash is only fetched once.
>> >    - Breaks when hash is incrementally fetched several times (which likely
>> is a violation of the Python API).
>> >
>> > 5) Finding or porting a different Hash implementation in C#:
>> >    + Low, constant memory usage.
>> >    + No impact on the large object heap.
>> >    + No limit of data length by the amount of memory.
>> >    + Optimal CPU usage when the hash is only fetched once.
>> >    + Optimal CPU usage when the hash is incrementally fetched several times.
>> >
>> > I've a local prototype implemented for 2), but I'm not sure whether that's
>> > the best way to go...
>>
>> Good analysis!
>>
>> My preference would be for (4), raising an exception if .update() is called
>> after .digest(), or .copy() is called at all. As a fallback, an extra
>> parameter to hashlib.new (&c) that triggers (2), for cases where its needed -
>> I can't say for sure, but I would think calling .update() after .digest()
>> would be rare, and so would .copy() (damn you Google for shutting down code
>> search). At least then the common case is fast and edge cases are (usually)
>> possible.
>
> Do you think asking on some cPython lists could give usable feedback how
> common it is to call copy() or to continue feeding data after calling
> digest()?
>
>> > Maybe we should google for purely managed implementations of the hash codes
>> > with a sensible license...
>>
>> There seems to be for MD5 and SHA1 but not SHA2 or RIPEMD160. They could be
>> ported from the public domain Crypto++ library, but that seems like a lot of
>> work for an edge case.
>
> Yes, that seems to be a lot of work.
>
> On the other hand, it's the 100% solution. :-)
>
> Best regards
>
> Markus Schaber
>
> CODESYS? a trademark of 3S-Smart Software Solutions GmbH
>
> Inspiring Automation Solutions
>
> 3S-Smart Software Solutions GmbH
> Dipl.-Inf. Markus Schaber | Product Development Core Technology
> Memminger Str. 151 | 87439 Kempten | Germany
> Tel. +49-831-54031-979 | Fax +49-831-54031-50
>
> E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS store: http://store.codesys.com
> CODESYS forum: http://forum.codesys.com
>
> Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
>
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users

From slide.o.mix at gmail.com  Thu Feb 27 20:48:17 2014
From: slide.o.mix at gmail.com (Slide)
Date: Thu, 27 Feb 2014 12:48:17 -0700
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <CA+7con=_JubQPy8XtMPSW+_6xM65BREOEJBOcprtq6wWf2mhcQ@mail.gmail.com>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
 <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
 <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B53862@SERVER10>
 <CAF7AXFGRsN3_dEogmt_NPzMrK_FDaRx_9Bq-T5+gu8J-+3ioMw@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B55A04@SERVER10>
 <CA+7con=_JubQPy8XtMPSW+_6xM65BREOEJBOcprtq6wWf2mhcQ@mail.gmail.com>
Message-ID: <CAPiUgVc6fEsLiJerEQhtaGCB+mrhcepYY5=1uD5afyUB-rmgsw@mail.gmail.com>

I had asked this question before, but there was some hesitation based on
the licensing of Mono. I'm not sure if that is an issue anymore.


On Thu, Feb 27, 2014 at 12:16 PM, Pawel Jasinski
<pawel.jasinski at gmail.com>wrote:

> Is there any reason not to use code out of mono?
> It looks like it supports SHA2 and RIPEMD160.
> https://bugzilla.xamarin.com/show_bug.cgi?id=11703
>
> On Thu, Feb 27, 2014 at 2:10 PM, Markus Schaber <m.schaber at codesys.com>
> wrote:
> > Hi,
> >
> > Von: Jeff Hardy [mailto:jdhardy at gmail.com]
> >> On Thu, Feb 27, 2014 at 11:11 AM, Markus Schaber <m.schaber at codesys.com
> >
> >> wrote:
> >> > Hi,
> >> >
> >> > I'm just trying to sum it up:
> >> >
> >> > 1) The current code:
> >> >    - High memory usage.
> >> >    - High load on the large object heap.
> >> >    - Limited by the available amount of memory (which might be
> considered a
> >> violation of the Python API).
> >> >    - High CPU usage when used incrementally (quadratic to the number
> of
> >> blocks added).
> >> >
> >> > 2) Optimizing with MemoryStream and lazy calculation:
> >> >    - High memory usage.
> >> >    - High load on the large object heap.
> >> >    - Limited by the available amount of memory (which might be
> considered a
> >> violation of the Python API).
> >> >    + Optimal CPU usage when the hash is only fetched once.
> >> >    ? Better than current code, but still not optimal when hash is
> >> incrementally fetched several times.
> >> >
> >> > 3) Optimizing with jagged arrays and lazy calculation:
> >> >    - High memory usage.
> >> >    + Improved or no impact on the large object heap (depending on the
> exact
> >> implementation)
> >> >    - Limited by the available amount of memory (which might be
> considered a
> >> violation of the Python API).
> >> >    + Optimal CPU usage when the hash is only fetched once.
> >> >    ? Better than current code, but still not optimal when hash is
> >> incrementally fetched several times.
> >> >
> >> > 4) Using the existing .NET incremental APIs
> >> >    + Low, constant memory usage.
> >> >    + No impact on the large object heap.
> >> >    + No limit of data length by the amount of memory.
> >> >    + Optimal CPU usage when the hash is only fetched once.
> >> >    - Breaks when hash is incrementally fetched several times (which
> likely
> >> is a violation of the Python API).
> >> >
> >> > 5) Finding or porting a different Hash implementation in C#:
> >> >    + Low, constant memory usage.
> >> >    + No impact on the large object heap.
> >> >    + No limit of data length by the amount of memory.
> >> >    + Optimal CPU usage when the hash is only fetched once.
> >> >    + Optimal CPU usage when the hash is incrementally fetched several
> times.
> >> >
> >> > I've a local prototype implemented for 2), but I'm not sure whether
> that's
> >> > the best way to go...
> >>
> >> Good analysis!
> >>
> >> My preference would be for (4), raising an exception if .update() is
> called
> >> after .digest(), or .copy() is called at all. As a fallback, an extra
> >> parameter to hashlib.new (&c) that triggers (2), for cases where its
> needed -
> >> I can't say for sure, but I would think calling .update() after
> .digest()
> >> would be rare, and so would .copy() (damn you Google for shutting down
> code
> >> search). At least then the common case is fast and edge cases are
> (usually)
> >> possible.
> >
> > Do you think asking on some cPython lists could give usable feedback how
> > common it is to call copy() or to continue feeding data after calling
> > digest()?
> >
> >> > Maybe we should google for purely managed implementations of the hash
> codes
> >> > with a sensible license...
> >>
> >> There seems to be for MD5 and SHA1 but not SHA2 or RIPEMD160. They
> could be
> >> ported from the public domain Crypto++ library, but that seems like a
> lot of
> >> work for an edge case.
> >
> > Yes, that seems to be a lot of work.
> >
> > On the other hand, it's the 100% solution. :-)
> >
> > Best regards
> >
> > Markus Schaber
> >
> > CODESYS? a trademark of 3S-Smart Software Solutions GmbH
> >
> > Inspiring Automation Solutions
> >
> > 3S-Smart Software Solutions GmbH
> > Dipl.-Inf. Markus Schaber | Product Development Core Technology
> > Memminger Str. 151 | 87439 Kempten | Germany
> > Tel. +49-831-54031-979 | Fax +49-831-54031-50
> >
> > E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS
> store: http://store.codesys.com
> > CODESYS forum: http://forum.codesys.com
> >
> > Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner |
> Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
> >
> > _______________________________________________
> > Ironpython-users mailing list
> > Ironpython-users at python.org
> > https://mail.python.org/mailman/listinfo/ironpython-users
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>



-- 
Website: http://earl-of-code.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140227/fa871fb6/attachment.html>

From pawel.jasinski at gmail.com  Thu Feb 27 21:10:07 2014
From: pawel.jasinski at gmail.com (Pawel Jasinski)
Date: Thu, 27 Feb 2014 21:10:07 +0100
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <CAPiUgVc6fEsLiJerEQhtaGCB+mrhcepYY5=1uD5afyUB-rmgsw@mail.gmail.com>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
 <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
 <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B53862@SERVER10>
 <CAF7AXFGRsN3_dEogmt_NPzMrK_FDaRx_9Bq-T5+gu8J-+3ioMw@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B55A04@SERVER10>
 <CA+7con=_JubQPy8XtMPSW+_6xM65BREOEJBOcprtq6wWf2mhcQ@mail.gmail.com>
 <CAPiUgVc6fEsLiJerEQhtaGCB+mrhcepYY5=1uD5afyUB-rmgsw@mail.gmail.com>
Message-ID: <CA+7conns1o9Bj5HARd_3rZQiADW6u_Y33RM5Nid7NbwWS9HSbw@mail.gmail.com>

I just checked, the part in question is covered by MIT.

On Thu, Feb 27, 2014 at 8:48 PM, Slide <slide.o.mix at gmail.com> wrote:
> I had asked this question before, but there was some hesitation based on the
> licensing of Mono. I'm not sure if that is an issue anymore.
>
>
> On Thu, Feb 27, 2014 at 12:16 PM, Pawel Jasinski <pawel.jasinski at gmail.com>
> wrote:
>>
>> Is there any reason not to use code out of mono?
>> It looks like it supports SHA2 and RIPEMD160.
>> https://bugzilla.xamarin.com/show_bug.cgi?id=11703
>>
>> On Thu, Feb 27, 2014 at 2:10 PM, Markus Schaber <m.schaber at codesys.com>
>> wrote:
>> > Hi,
>> >
>> > Von: Jeff Hardy [mailto:jdhardy at gmail.com]
>> >> On Thu, Feb 27, 2014 at 11:11 AM, Markus Schaber
>> >> <m.schaber at codesys.com>
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > I'm just trying to sum it up:
>> >> >
>> >> > 1) The current code:
>> >> >    - High memory usage.
>> >> >    - High load on the large object heap.
>> >> >    - Limited by the available amount of memory (which might be
>> >> > considered a
>> >> violation of the Python API).
>> >> >    - High CPU usage when used incrementally (quadratic to the number
>> >> > of
>> >> blocks added).
>> >> >
>> >> > 2) Optimizing with MemoryStream and lazy calculation:
>> >> >    - High memory usage.
>> >> >    - High load on the large object heap.
>> >> >    - Limited by the available amount of memory (which might be
>> >> > considered a
>> >> violation of the Python API).
>> >> >    + Optimal CPU usage when the hash is only fetched once.
>> >> >    ? Better than current code, but still not optimal when hash is
>> >> incrementally fetched several times.
>> >> >
>> >> > 3) Optimizing with jagged arrays and lazy calculation:
>> >> >    - High memory usage.
>> >> >    + Improved or no impact on the large object heap (depending on the
>> >> > exact
>> >> implementation)
>> >> >    - Limited by the available amount of memory (which might be
>> >> > considered a
>> >> violation of the Python API).
>> >> >    + Optimal CPU usage when the hash is only fetched once.
>> >> >    ? Better than current code, but still not optimal when hash is
>> >> incrementally fetched several times.
>> >> >
>> >> > 4) Using the existing .NET incremental APIs
>> >> >    + Low, constant memory usage.
>> >> >    + No impact on the large object heap.
>> >> >    + No limit of data length by the amount of memory.
>> >> >    + Optimal CPU usage when the hash is only fetched once.
>> >> >    - Breaks when hash is incrementally fetched several times (which
>> >> > likely
>> >> is a violation of the Python API).
>> >> >
>> >> > 5) Finding or porting a different Hash implementation in C#:
>> >> >    + Low, constant memory usage.
>> >> >    + No impact on the large object heap.
>> >> >    + No limit of data length by the amount of memory.
>> >> >    + Optimal CPU usage when the hash is only fetched once.
>> >> >    + Optimal CPU usage when the hash is incrementally fetched several
>> >> > times.
>> >> >
>> >> > I've a local prototype implemented for 2), but I'm not sure whether
>> >> > that's
>> >> > the best way to go...
>> >>
>> >> Good analysis!
>> >>
>> >> My preference would be for (4), raising an exception if .update() is
>> >> called
>> >> after .digest(), or .copy() is called at all. As a fallback, an extra
>> >> parameter to hashlib.new (&c) that triggers (2), for cases where its
>> >> needed -
>> >> I can't say for sure, but I would think calling .update() after
>> >> .digest()
>> >> would be rare, and so would .copy() (damn you Google for shutting down
>> >> code
>> >> search). At least then the common case is fast and edge cases are
>> >> (usually)
>> >> possible.
>> >
>> > Do you think asking on some cPython lists could give usable feedback how
>> > common it is to call copy() or to continue feeding data after calling
>> > digest()?
>> >
>> >> > Maybe we should google for purely managed implementations of the hash
>> >> > codes
>> >> > with a sensible license...
>> >>
>> >> There seems to be for MD5 and SHA1 but not SHA2 or RIPEMD160. They
>> >> could be
>> >> ported from the public domain Crypto++ library, but that seems like a
>> >> lot of
>> >> work for an edge case.
>> >
>> > Yes, that seems to be a lot of work.
>> >
>> > On the other hand, it's the 100% solution. :-)
>> >
>> > Best regards
>> >
>> > Markus Schaber
>> >
>> > CODESYS? a trademark of 3S-Smart Software Solutions GmbH
>> >
>> > Inspiring Automation Solutions
>> >
>> > 3S-Smart Software Solutions GmbH
>> > Dipl.-Inf. Markus Schaber | Product Development Core Technology
>> > Memminger Str. 151 | 87439 Kempten | Germany
>> > Tel. +49-831-54031-979 | Fax +49-831-54031-50
>> >
>> > E-Mail: m.schaber at codesys.com | Web: http://www.codesys.com | CODESYS
>> > store: http://store.codesys.com
>> > CODESYS forum: http://forum.codesys.com
>> >
>> > Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner |
>> > Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915
>> >
>> > _______________________________________________
>> > Ironpython-users mailing list
>> > Ironpython-users at python.org
>> > https://mail.python.org/mailman/listinfo/ironpython-users
>> _______________________________________________
>> Ironpython-users mailing list
>> Ironpython-users at python.org
>> https://mail.python.org/mailman/listinfo/ironpython-users
>
>
>
>
> --
> Website: http://earl-of-code.com

From jdhardy at gmail.com  Thu Feb 27 21:59:03 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Thu, 27 Feb 2014 20:59:03 +0000
Subject: [Ironpython-users] Hashing a directory is magnitudes slower
 than in cPython
In-Reply-To: <CA+7conns1o9Bj5HARd_3rZQiADW6u_Y33RM5Nid7NbwWS9HSbw@mail.gmail.com>
References: <727D8E16AE957149B447FE368139F2B53987CEFA@SERVER10>
 <CAF7AXFFjEo6xkXzZ63TU6e81kapYs9ufvFYWqgeisXvCaKNMLA@mail.gmail.com>
 <CAO-Cae6HH6Gev-uurkLPyHqSazJFVsSt=6TyH4ptYiE13K+A+w@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B53862@SERVER10>
 <CAF7AXFGRsN3_dEogmt_NPzMrK_FDaRx_9Bq-T5+gu8J-+3ioMw@mail.gmail.com>
 <727D8E16AE957149B447FE368139F2B539B55A04@SERVER10>
 <CA+7con=_JubQPy8XtMPSW+_6xM65BREOEJBOcprtq6wWf2mhcQ@mail.gmail.com>
 <CAPiUgVc6fEsLiJerEQhtaGCB+mrhcepYY5=1uD5afyUB-rmgsw@mail.gmail.com>
 <CA+7conns1o9Bj5HARd_3rZQiADW6u_Y33RM5Nid7NbwWS9HSbw@mail.gmail.com>
Message-ID: <CAF7AXFHFufC03L6VLt51DBbxoh0k-Tjf3pLVtJ6QYtwUg=LOSQ@mail.gmail.com>

On Thu, Feb 27, 2014 at 8:10 PM, Pawel Jasinski
<pawel.jasinski at gmail.com> wrote:
> I just checked, the part in question is covered by MIT.

Oh, that's better. MIT is no problem (once upon a time parts of Mono
were LGPL, which is right out).

Also, I had assumed that Mono would use native platform libraries for
crypto. You know what happens when you assume... :|

If anyone wants to copy the Mono versions (licence intact!) and modify
them to work standalone (i.e. you don't have to keep the base classes,
necessarily, although it might help), *and* update the Python module
implementations, I'll make sure it gets in the next release.

- Jeff

>
> On Thu, Feb 27, 2014 at 8:48 PM, Slide <slide.o.mix at gmail.com> wrote:
>> I had asked this question before, but there was some hesitation based on the
>> licensing of Mono. I'm not sure if that is an issue anymore.
>>
>>
>> On Thu, Feb 27, 2014 at 12:16 PM, Pawel Jasinski <pawel.jasinski at gmail.com>
>> wrote:
>>>
>>> Is there any reason not to use code out of mono?
>>> It looks like it supports SHA2 and RIPEMD160.
>>> https://bugzilla.xamarin.com/show_bug.cgi?id=11703
>>>

From jdhardy at gmail.com  Thu Feb 27 22:07:59 2014
From: jdhardy at gmail.com (Jeff Hardy)
Date: Thu, 27 Feb 2014 21:07:59 +0000
Subject: [Ironpython-users] IronPython 2.7.5 Schedule
Message-ID: <CAF7AXFEe7_97myUjnZYPMhT30sHAnNK-L1DLp5nJFnr0uUBBtg@mail.gmail.com>

I know everyone wants to hear about ItonPython 3, but it's time to
talk about IronPython 2.7 for a bit. Enough changes have accumulated
(thanks Pawel!) that it's probably time for another release. I'm
thinking Beta 1 around March 17, Beta 2 around April 1, and an
RC/final around April 15. Does anybody have anything they want to get
in but needs more time than that? The next release will probably be
around October.

There's a bunch of low-hanging fruit in the issue tracker that could
conceivably make it in. I just created the 2.7.5 release on CodePlex,
so if you have edit access and you want something in 2.7.5, move it;
if you don't, leave a comment or an email here and I'll add it to
2.7.5.

If you have something you want fixed, the best way is to do it
yourself, and if you need any help, feel free to ask!

- Jeff

From pawel.jasinski at gmail.com  Thu Feb 27 23:37:35 2014
From: pawel.jasinski at gmail.com (Pawel Jasinski)
Date: Thu, 27 Feb 2014 23:37:35 +0100
Subject: [Ironpython-users] IronPython 2.7.5 Schedule
In-Reply-To: <CAF7AXFEe7_97myUjnZYPMhT30sHAnNK-L1DLp5nJFnr0uUBBtg@mail.gmail.com>
References: <CAF7AXFEe7_97myUjnZYPMhT30sHAnNK-L1DLp5nJFnr0uUBBtg@mail.gmail.com>
Message-ID: <CA+7conn0jUbV7iE481fw_UfPafgK+bhwx6Veq_S5unfYfEjLeA@mail.gmail.com>

I have cp34837, cp34892, cp34910 fixed. I will make a PR.

I also have some workarounds where I am not clear if they should get
into release:

1. cp32331 - better unicode success rate in real application, but
decreased number of unit test passed
2. update to installer to modify PATH and register .py extension. This
could make users of both cpython and ironpython under windows unhappy.
3. importable version of multiprocessing - just to avoid import
errors, this sounds a bit like cp34959

any comments on last 3 items are welcome.


--pawel

On Thu, Feb 27, 2014 at 10:07 PM, Jeff Hardy <jdhardy at gmail.com> wrote:
> I know everyone wants to hear about ItonPython 3, but it's time to
> talk about IronPython 2.7 for a bit. Enough changes have accumulated
> (thanks Pawel!) that it's probably time for another release. I'm
> thinking Beta 1 around March 17, Beta 2 around April 1, and an
> RC/final around April 15. Does anybody have anything they want to get
> in but needs more time than that? The next release will probably be
> around October.
>
> There's a bunch of low-hanging fruit in the issue tracker that could
> conceivably make it in. I just created the 2.7.5 release on CodePlex,
> so if you have edit access and you want something in 2.7.5, move it;
> if you don't, leave a comment or an email here and I'll add it to
> 2.7.5.
>
> If you have something you want fixed, the best way is to do it
> yourself, and if you need any help, feel free to ask!
>
> - Jeff
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users

From vernondcole at gmail.com  Fri Feb 28 09:15:50 2014
From: vernondcole at gmail.com (Vernon D. Cole)
Date: Fri, 28 Feb 2014 09:15:50 +0100
Subject: [Ironpython-users] IronPython 2.7.5 Schedule
In-Reply-To: <CA+7conn0jUbV7iE481fw_UfPafgK+bhwx6Veq_S5unfYfEjLeA@mail.gmail.com>
References: <CAF7AXFEe7_97myUjnZYPMhT30sHAnNK-L1DLp5nJFnr0uUBBtg@mail.gmail.com>
 <CA+7conn0jUbV7iE481fw_UfPafgK+bhwx6Veq_S5unfYfEjLeA@mail.gmail.com>
Message-ID: <CAH-ZgAd+nFdkSTiT3=R+44EReyUthLJZRzG25CRMG_L+gugARg@mail.gmail.com>

Under no circumstances should the .py extension ever be taken away from *Python
Launcher for Windows* <http://legacy.python.org/dev/peps/pep-0397/>.  I
have not looked at what happens when PLW is run on a computer which does
not have a version of CPython installed. If it does the right thing (launch
the first program mentioned in py.ini) then perhaps we should consider
shipping PLW with the installer, like CPython 3.3 does.
  At any rate, we should make sure that py.ini contains a proper
description of IronPython, so that shebang processing will work.  On my
Windows development machine, py.exe launches all my versions of Python. My
py.ini looks like this (note: the "jython" line is so long it wraps):

[commands]
> ipy="c:\program files (x86)\IronPython 2.7\ipy.exe"
> ipy64="c:\program files (x86)\IronPython 2.7\ipy64.exe"
> jython="C:\Program Files (x86)\Java\jre7\bin\java"  -Xmx512m -Xss1152k
> -Dpython.home="C:\jython2.7b1"
> -Dpython.executable="C:\jython2.7b1\jython.bat"  -classpath
> "C:\jython2.7b1\jython.jar" org.python.util.jython
> kivy="c:\utils\kivy\kivy.bat"
>



On Thu, Feb 27, 2014 at 11:37 PM, Pawel Jasinski
<pawel.jasinski at gmail.com>wrote:

> I have cp34837, cp34892, cp34910 fixed. I will make a PR.
>
> I also have some workarounds where I am not clear if they should get
> into release:
>
> 1. cp32331 - better unicode success rate in real application, but
> decreased number of unit test passed
> 2. update to installer to modify PATH and register .py extension. This
> could make users of both cpython and ironpython under windows unhappy.
> 3. importable version of multiprocessing - just to avoid import
> errors, this sounds a bit like cp34959
>
> any comments on last 3 items are welcome.
>
>
> --pawel
>
> On Thu, Feb 27, 2014 at 10:07 PM, Jeff Hardy <jdhardy at gmail.com> wrote:
> > I know everyone wants to hear about ItonPython 3, but it's time to
> > talk about IronPython 2.7 for a bit. Enough changes have accumulated
> > (thanks Pawel!) that it's probably time for another release. I'm
> > thinking Beta 1 around March 17, Beta 2 around April 1, and an
> > RC/final around April 15. Does anybody have anything they want to get
> > in but needs more time than that? The next release will probably be
> > around October.
> >
> > There's a bunch of low-hanging fruit in the issue tracker that could
> > conceivably make it in. I just created the 2.7.5 release on CodePlex,
> > so if you have edit access and you want something in 2.7.5, move it;
> > if you don't, leave a comment or an email here and I'll add it to
> > 2.7.5.
> >
> > If you have something you want fixed, the best way is to do it
> > yourself, and if you need any help, feel free to ask!
> >
> > - Jeff
> > _______________________________________________
> > Ironpython-users mailing list
> > Ironpython-users at python.org
> > https://mail.python.org/mailman/listinfo/ironpython-users
> _______________________________________________
> Ironpython-users mailing list
> Ironpython-users at python.org
> https://mail.python.org/mailman/listinfo/ironpython-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140228/f09de2c9/attachment.html>

From vernondcole at gmail.com  Fri Feb 28 10:26:17 2014
From: vernondcole at gmail.com (Vernon D. Cole)
Date: Fri, 28 Feb 2014 10:26:17 +0100
Subject: [Ironpython-users] IronPython 2.7.5 Schedule
In-Reply-To: <CAH-ZgAd+nFdkSTiT3=R+44EReyUthLJZRzG25CRMG_L+gugARg@mail.gmail.com>
References: <CAF7AXFEe7_97myUjnZYPMhT30sHAnNK-L1DLp5nJFnr0uUBBtg@mail.gmail.com>
 <CA+7conn0jUbV7iE481fw_UfPafgK+bhwx6Veq_S5unfYfEjLeA@mail.gmail.com>
 <CAH-ZgAd+nFdkSTiT3=R+44EReyUthLJZRzG25CRMG_L+gugARg@mail.gmail.com>
Message-ID: <CAH-ZgAcRXFXLHFOfxRjK7hqGzabAQBPcziz9nBERFpEce9FAag@mail.gmail.com>

(Answering myself -- sorry)
I have re-read the PEP and skimmed the .c code for the launcher.  There is
provision in the PLW for setting a default version of Python, but it
appears that the only choices allowed are versions of CPython selected (by
number) from pythons found in the registry.  One could perhaps hack this by
adding IronPython to an assumed place in the registry (2.0.0?) and then set
the default to that hack entry.



On Fri, Feb 28, 2014 at 9:15 AM, Vernon D. Cole <vernondcole at gmail.com>wrote:

> Under no circumstances should the .py extension ever be taken away from *Python
> Launcher for Windows* <http://legacy.python.org/dev/peps/pep-0397/>.  I
> have not looked at what happens when PLW is run on a computer which does
> not have a version of CPython installed. If it does the right thing (launch
> the first program mentioned in py.ini) then perhaps we should consider
> shipping PLW with the installer, like CPython 3.3 does.
>   At any rate, we should make sure that py.ini contains a proper
> description of IronPython, so that shebang processing will work.  On my
> Windows development machine, py.exe launches all my versions of Python. My
> py.ini looks like this (note: the "jython" line is so long it wraps):
>
> [commands]
>> ipy="c:\program files (x86)\IronPython 2.7\ipy.exe"
>> ipy64="c:\program files (x86)\IronPython 2.7\ipy64.exe"
>> jython="C:\Program Files (x86)\Java\jre7\bin\java"  -Xmx512m -Xss1152k
>> -Dpython.home="C:\jython2.7b1"
>> -Dpython.executable="C:\jython2.7b1\jython.bat"  -classpath
>> "C:\jython2.7b1\jython.jar" org.python.util.jython
>> kivy="c:\utils\kivy\kivy.bat"
>>
>
>
>
> On Thu, Feb 27, 2014 at 11:37 PM, Pawel Jasinski <pawel.jasinski at gmail.com
> > wrote:
>
>> I have cp34837, cp34892, cp34910 fixed. I will make a PR.
>>
>> I also have some workarounds where I am not clear if they should get
>> into release:
>>
>> 1. cp32331 - better unicode success rate in real application, but
>> decreased number of unit test passed
>> 2. update to installer to modify PATH and register .py extension. This
>> could make users of both cpython and ironpython under windows unhappy.
>> 3. importable version of multiprocessing - just to avoid import
>> errors, this sounds a bit like cp34959
>>
>> any comments on last 3 items are welcome.
>>
>>
>> --pawel
>>
>> On Thu, Feb 27, 2014 at 10:07 PM, Jeff Hardy <jdhardy at gmail.com> wrote:
>> > I know everyone wants to hear about ItonPython 3, but it's time to
>> > talk about IronPython 2.7 for a bit. Enough changes have accumulated
>> > (thanks Pawel!) that it's probably time for another release. I'm
>> > thinking Beta 1 around March 17, Beta 2 around April 1, and an
>> > RC/final around April 15. Does anybody have anything they want to get
>> > in but needs more time than that? The next release will probably be
>> > around October.
>> >
>> > There's a bunch of low-hanging fruit in the issue tracker that could
>> > conceivably make it in. I just created the 2.7.5 release on CodePlex,
>> > so if you have edit access and you want something in 2.7.5, move it;
>> > if you don't, leave a comment or an email here and I'll add it to
>> > 2.7.5.
>> >
>> > If you have something you want fixed, the best way is to do it
>> > yourself, and if you need any help, feel free to ask!
>> >
>> > - Jeff
>> > _______________________________________________
>> > Ironpython-users mailing list
>> > Ironpython-users at python.org
>> > https://mail.python.org/mailman/listinfo/ironpython-users
>> _______________________________________________
>> Ironpython-users mailing list
>> Ironpython-users at python.org
>> https://mail.python.org/mailman/listinfo/ironpython-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20140228/56fb3423/attachment.html>