From dalcinl at gmail.com  Wed Aug  1 03:09:50 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 31 Jul 2007 22:09:50 -0300
Subject: [Python-Dev] Python Package Index hostname change
In-Reply-To: <46AFA27A.4090700@v.loewis.de>
References: <46AFA27A.4090700@v.loewis.de>
Message-ID: <e7ba66e40707311809m75f4cefwd7b1cebc02240ce3@mail.gmail.com>

On 7/31/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> The Python Packaging Index (the software formerly known
> as Cheeseshop) is now available at
>
> http://pypi.python.org/pypi

Please, update 'DEFAULT_REPOSITORY' in Lib/distutils/command/upload.py
(py-2.6 and py3k-struni branches)


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From steve at holdenweb.com  Wed Aug  1 03:19:19 2007
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 31 Jul 2007 21:19:19 -0400
Subject: [Python-Dev] Cygwin: Problem detecting subprocess termination
 after _spawn_posix in distutils?
In-Reply-To: <46AFAD41.9060203@v.loewis.de>
References: <f8lcd3$9ou$1@sea.gmane.org> <46AFAD41.9060203@v.loewis.de>
Message-ID: <46AFDF97.3090909@holdenweb.com>

Martin v. L?wis wrote:
>> It would be really nice if test_distutils showed any failures, but it 
>> doesn't so any assistance would be welcome. At this point I can't even 
>> replicate the failure in a simpler test :-(
> 
> My guess is that it's the environment; if not that, the working
> directory. Assuming you have already instrumented
> ccompiler.CCompiler.spawn, I suggest to dump os.environ and
> print os.getcwd(). Assuming you really meant that you run under
> Cygwin Python (instead of just using --compiler), you might
> want to instrument spawn._spawn_posix instead.
> 
> When you say you extracted _spawn_all from distutils/spawn.py:
> what version of Python are you talking about? I can't find
> _spawn_all in the sources of 2.5.x, or 2.6.
> 
Thanks for taking the time to have a look at this.

Sorry, it *was* _spawn_posix I extracted (and have instrumented in the 
live version) - I have no idea where "_spawn_all" came from. I am indeed 
running under Cygwin Python.

Here-s a diff -u output against the original spawn.py so you can see 
what I have changed.

$ diff -u /lib/python2.5/distutils/{spawn.py.org,spawn.py}
--- /lib/python2.5/distutils/spawn.py.org       2007-07-14 
09:09:24.114921600 -0400
+++ /lib/python2.5/distutils/spawn.py   2007-07-31 20:53:33.325945600 -0400
@@ -118,7 +118,9 @@
                    search_path=1,
                    verbose=0,
                    dry_run=0):
-
+    for _k in sorted(os.environ.keys()):
+        print "%s=%s" % (_k, os.environ[_k])
+    print "SPAWN:", cmd, "PATH?", search_path, "V:", verbose, "D:", dry_run
      log.info(string.join(cmd, ' '))
      if dry_run:
          return
@@ -144,20 +146,25 @@
          # Loop until the child either exits or is terminated by a signal
          # (ie. keep waiting if it's merely stopped)
          while 1:
+            print "Are we done yet? Waiting on pid", pid
              try:
                  (pid, status) = os.waitpid(pid, 0)
+                print "Got pid, status", pid, status
              except OSError, exc:
                  import errno
+                print "Got OSError", exc.errno
                  if exc.errno == errno.EINTR:
                      continue
                  raise DistutilsExecError, \
                        "command '%s' failed: %s" % (cmd[0], exc[-1])
              if os.WIFSIGNALED(status):
+                print "Got WIFSIGNALED", status
                  raise DistutilsExecError, \
                        "command '%s' terminated by signal %d" % \
                        (cmd[0], os.WTERMSIG(status))

              elif os.WIFEXITED(status):
+                print "Got WIFEXITED", status
                  exit_status = os.WEXITSTATUS(status)
                  if exit_status == 0:
                      return              # hey, it succeeded!
@@ -167,9 +174,11 @@
                            (cmd[0], exit_status)

              elif os.WIFSTOPPED(status):
+                print "Got WIFSTOPPED", status
                  continue

              else:
+                print "Got unknown exception", status
                  raise DistutilsExecError, \
                        "unknown error executing '%s': termination 
status %d" % \

                        (cmd[0], status)

The output now includes the environment:

$ python setup.py install
running install
running build
running build_py
running build_ext
building '_imaging' extension
!::=::\
!C:=C:\cygwin\bin
ALLUSERSPROFILE=C:\Documents and Settings\All Users
APPDATA=C:\Documents and Settings\sholden\Application Data
APR_ICONV_PATH=C:\Program Files\Subversion\iconv
CDPATH=.:/c/Steve:/c/Steve/Projects:/usr/local
CLIENTNAME=Console
COMMONPROGRAMFILES=C:\Program Files\Common Files
COMPUTERNAME=BIGBOY
COMSPEC=C:\WINDOWS\system32\cmd.exe
CVSROOT=/usr/local/repository/
CVS_RSH=/bin/ssh
FP_NO_HOST_CHECK=NO
HOME=/c/Steve
HOMEDRIVE=C:
HOMEPATH=\Documents and Settings\sholden
HOSTNAME=bigboy
INFOPATH=/usr/local/info:/usr/share/info:/usr/info:
LANG=C
LOGONSERVER=\\BIGBOY
MAKE_MODE=unix
MANPATH=/usr/local/man:/usr/share/man:/usr/man::/usr/ssl/man
NUMBER_OF_PROCESSORS=1
OLDPWD=/c/Steve
OS=Windows_NT
PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/c/WINDOWS/system32:/c/WINDOWS:
/c/WINDOWS/System32/Wbem:/c/Program Files/ATI Technologies/ATI Control 
Panel:/c/
Program Files/Common Files/GTK/2.0/bin:/c/Program 
Files/Subversion/bin:/c/Python
25:/c/Steve/bin
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
PLAT=cygwin-1.5.24-i686
PRINTER=HP Photosmart C6100 series
PROCESSOR_ARCHITECTURE=x86
PROCESSOR_IDENTIFIER=x86 Family 6 Model 13 Stepping 6, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=0d06
PROGRAMFILES=C:\Program Files
PROMPT=$P$G
PS1=\[\e]0;\w\a\]\n\[\e[32m\]\u@\h \[\e[33m\]\w\[\e[0m\]\n\$
PWD=/c/Steve/Imaging-1.1.6
PYSVN=svn+ssh://pythondev at svn.python.org/
PYTHONSTARTUP=/c/Steve/.pythonrc
SESSIONNAME=Console
SHLVL=1
SYSTEMDRIVE=C:
SYSTEMROOT=C:\WINDOWS
TEMP=/c/DOCUME~1/sholden/LOCALS~1/Temp
TERM=cygwin
TMP=/c/DOCUME~1/sholden/LOCALS~1/Temp
USER=sholden
USER1=u35582809 at s90820416.onlinehome.us
USERDOMAIN=BIGBOY
USERNAME=sholden
USERPROFILE=C:\Documents and Settings\sholden
VISUAL=vi
VS80COMNTOOLS=C:\Program Files\Microsoft Visual Studio 8\Common7\Tools\
WINDIR=C:\WINDOWS
_=/usr/bin/python
SPAWN: ['gcc', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O3', '-Wall', 
'-Wstri
ct-prototypes', '-DHAVE_LIBZ', '-IlibImaging', '-I/usr/include', 
'-I/usr/include
/python2.5', '-c', 'libImaging/Dib.c', '-o', 
'build/temp.cygwin-1.5.24-i686-2.5/
libImaging/Dib.o'] PATH? 1 V: 0 D: 0
gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes 
-DHAVE_LIBZ -
IlibImaging -I/usr/include -I/usr/include/python2.5 -c libImaging/Dib.c 
-o build
/temp.cygwin-1.5.24-i686-2.5/libImaging/Dib.o
Are we done yet? Waiting on pid 416

The only environment variables that don't appear in the shell output 
from the env command are INFOPATH, MAKE_MODE and PLAT. I am still flummoxed.

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

From steve at holdenweb.com  Wed Aug  1 03:19:19 2007
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 31 Jul 2007 21:19:19 -0400
Subject: [Python-Dev] Cygwin: Problem detecting subprocess termination
 after _spawn_posix in distutils?
In-Reply-To: <46AFAD41.9060203@v.loewis.de>
References: <f8lcd3$9ou$1@sea.gmane.org> <46AFAD41.9060203@v.loewis.de>
Message-ID: <46AFDF97.3090909@holdenweb.com>

Martin v. L?wis wrote:
>> It would be really nice if test_distutils showed any failures, but it 
>> doesn't so any assistance would be welcome. At this point I can't even 
>> replicate the failure in a simpler test :-(
> 
> My guess is that it's the environment; if not that, the working
> directory. Assuming you have already instrumented
> ccompiler.CCompiler.spawn, I suggest to dump os.environ and
> print os.getcwd(). Assuming you really meant that you run under
> Cygwin Python (instead of just using --compiler), you might
> want to instrument spawn._spawn_posix instead.
> 
> When you say you extracted _spawn_all from distutils/spawn.py:
> what version of Python are you talking about? I can't find
> _spawn_all in the sources of 2.5.x, or 2.6.
> 
Thanks for taking the time to have a look at this.

Sorry, it *was* _spawn_posix I extracted (and have instrumented in the 
live version) - I have no idea where "_spawn_all" came from. I am indeed 
running under Cygwin Python.

Here-s a diff -u output against the original spawn.py so you can see 
what I have changed.

$ diff -u /lib/python2.5/distutils/{spawn.py.org,spawn.py}
--- /lib/python2.5/distutils/spawn.py.org       2007-07-14 
09:09:24.114921600 -0400
+++ /lib/python2.5/distutils/spawn.py   2007-07-31 20:53:33.325945600 -0400
@@ -118,7 +118,9 @@
                    search_path=1,
                    verbose=0,
                    dry_run=0):
-
+    for _k in sorted(os.environ.keys()):
+        print "%s=%s" % (_k, os.environ[_k])
+    print "SPAWN:", cmd, "PATH?", search_path, "V:", verbose, "D:", dry_run
      log.info(string.join(cmd, ' '))
      if dry_run:
          return
@@ -144,20 +146,25 @@
          # Loop until the child either exits or is terminated by a signal
          # (ie. keep waiting if it's merely stopped)
          while 1:
+            print "Are we done yet? Waiting on pid", pid
              try:
                  (pid, status) = os.waitpid(pid, 0)
+                print "Got pid, status", pid, status
              except OSError, exc:
                  import errno
+                print "Got OSError", exc.errno
                  if exc.errno == errno.EINTR:
                      continue
                  raise DistutilsExecError, \
                        "command '%s' failed: %s" % (cmd[0], exc[-1])
              if os.WIFSIGNALED(status):
+                print "Got WIFSIGNALED", status
                  raise DistutilsExecError, \
                        "command '%s' terminated by signal %d" % \
                        (cmd[0], os.WTERMSIG(status))

              elif os.WIFEXITED(status):
+                print "Got WIFEXITED", status
                  exit_status = os.WEXITSTATUS(status)
                  if exit_status == 0:
                      return              # hey, it succeeded!
@@ -167,9 +174,11 @@
                            (cmd[0], exit_status)

              elif os.WIFSTOPPED(status):
+                print "Got WIFSTOPPED", status
                  continue

              else:
+                print "Got unknown exception", status
                  raise DistutilsExecError, \
                        "unknown error executing '%s': termination 
status %d" % \

                        (cmd[0], status)

The output now includes the environment:

$ python setup.py install
running install
running build
running build_py
running build_ext
building '_imaging' extension
!::=::\
!C:=C:\cygwin\bin
ALLUSERSPROFILE=C:\Documents and Settings\All Users
APPDATA=C:\Documents and Settings\sholden\Application Data
APR_ICONV_PATH=C:\Program Files\Subversion\iconv
CDPATH=.:/c/Steve:/c/Steve/Projects:/usr/local
CLIENTNAME=Console
COMMONPROGRAMFILES=C:\Program Files\Common Files
COMPUTERNAME=BIGBOY
COMSPEC=C:\WINDOWS\system32\cmd.exe
CVSROOT=/usr/local/repository/
CVS_RSH=/bin/ssh
FP_NO_HOST_CHECK=NO
HOME=/c/Steve
HOMEDRIVE=C:
HOMEPATH=\Documents and Settings\sholden
HOSTNAME=bigboy
INFOPATH=/usr/local/info:/usr/share/info:/usr/info:
LANG=C
LOGONSERVER=\\BIGBOY
MAKE_MODE=unix
MANPATH=/usr/local/man:/usr/share/man:/usr/man::/usr/ssl/man
NUMBER_OF_PROCESSORS=1
OLDPWD=/c/Steve
OS=Windows_NT
PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/c/WINDOWS/system32:/c/WINDOWS:
/c/WINDOWS/System32/Wbem:/c/Program Files/ATI Technologies/ATI Control 
Panel:/c/
Program Files/Common Files/GTK/2.0/bin:/c/Program 
Files/Subversion/bin:/c/Python
25:/c/Steve/bin
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
PLAT=cygwin-1.5.24-i686
PRINTER=HP Photosmart C6100 series
PROCESSOR_ARCHITECTURE=x86
PROCESSOR_IDENTIFIER=x86 Family 6 Model 13 Stepping 6, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=0d06
PROGRAMFILES=C:\Program Files
PROMPT=$P$G
PS1=\[\e]0;\w\a\]\n\[\e[32m\]\u@\h \[\e[33m\]\w\[\e[0m\]\n\$
PWD=/c/Steve/Imaging-1.1.6
PYSVN=svn+ssh://pythondev at svn.python.org/
PYTHONSTARTUP=/c/Steve/.pythonrc
SESSIONNAME=Console
SHLVL=1
SYSTEMDRIVE=C:
SYSTEMROOT=C:\WINDOWS
TEMP=/c/DOCUME~1/sholden/LOCALS~1/Temp
TERM=cygwin
TMP=/c/DOCUME~1/sholden/LOCALS~1/Temp
USER=sholden
USER1=u35582809 at s90820416.onlinehome.us
USERDOMAIN=BIGBOY
USERNAME=sholden
USERPROFILE=C:\Documents and Settings\sholden
VISUAL=vi
VS80COMNTOOLS=C:\Program Files\Microsoft Visual Studio 8\Common7\Tools\
WINDIR=C:\WINDOWS
_=/usr/bin/python
SPAWN: ['gcc', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O3', '-Wall', 
'-Wstri
ct-prototypes', '-DHAVE_LIBZ', '-IlibImaging', '-I/usr/include', 
'-I/usr/include
/python2.5', '-c', 'libImaging/Dib.c', '-o', 
'build/temp.cygwin-1.5.24-i686-2.5/
libImaging/Dib.o'] PATH? 1 V: 0 D: 0
gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes 
-DHAVE_LIBZ -
IlibImaging -I/usr/include -I/usr/include/python2.5 -c libImaging/Dib.c 
-o build
/temp.cygwin-1.5.24-i686-2.5/libImaging/Dib.o
Are we done yet? Waiting on pid 416

The only environment variables that don't appear in the shell output 
from the env command are INFOPATH, MAKE_MODE and PLAT. I am still flummoxed.

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------


From guido at python.org  Wed Aug  1 03:48:50 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 31 Jul 2007 18:48:50 -0700
Subject: [Python-Dev] Python Package Index hostname change
In-Reply-To: <e7ba66e40707311809m75f4cefwd7b1cebc02240ce3@mail.gmail.com>
References: <46AFA27A.4090700@v.loewis.de>
	<e7ba66e40707311809m75f4cefwd7b1cebc02240ce3@mail.gmail.com>
Message-ID: <ca471dc20707311848o4b2616d6x4a57efe4be7ad926@mail.gmail.com>

And why not in the upcoming 2.5 release as well?

On 7/31/07, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 7/31/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > The Python Packaging Index (the software formerly known
> > as Cheeseshop) is now available at
> >
> > http://pypi.python.org/pypi
>
> Please, update 'DEFAULT_REPOSITORY' in Lib/distutils/command/upload.py
> (py-2.6 and py3k-struni branches)
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Wed Aug  1 07:15:26 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 01 Aug 2007 07:15:26 +0200
Subject: [Python-Dev] Python Package Index hostname change
In-Reply-To: <e7ba66e40707311809m75f4cefwd7b1cebc02240ce3@mail.gmail.com>
References: <46AFA27A.4090700@v.loewis.de>
	<e7ba66e40707311809m75f4cefwd7b1cebc02240ce3@mail.gmail.com>
Message-ID: <46B016EE.9030309@v.loewis.de>

> Please, update 'DEFAULT_REPOSITORY' in Lib/distutils/command/upload.py
> (py-2.6 and py3k-struni branches)

I did already, for 2.6, in r56543. For the other branches, this change
will propagate through merging.

Regards,
Martin

From martin at v.loewis.de  Wed Aug  1 07:16:39 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 01 Aug 2007 07:16:39 +0200
Subject: [Python-Dev] Python Package Index hostname change
In-Reply-To: <ca471dc20707311848o4b2616d6x4a57efe4be7ad926@mail.gmail.com>
References: <46AFA27A.4090700@v.loewis.de>	
	<e7ba66e40707311809m75f4cefwd7b1cebc02240ce3@mail.gmail.com>
	<ca471dc20707311848o4b2616d6x4a57efe4be7ad926@mail.gmail.com>
Message-ID: <46B01737.60608@v.loewis.de>

Guido van Rossum schrieb:
> And why not in the upcoming 2.5 release as well?

It's changed there as well.

Regards,
Martin

From martin at v.loewis.de  Wed Aug  1 07:48:00 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 01 Aug 2007 07:48:00 +0200
Subject: [Python-Dev] Cygwin: Problem detecting subprocess termination
 after _spawn_posix in distutils?
In-Reply-To: <46AFDF97.3090909@holdenweb.com>
References: <f8lcd3$9ou$1@sea.gmane.org> <46AFAD41.9060203@v.loewis.de>
	<46AFDF97.3090909@holdenweb.com>
Message-ID: <46B01E90.8090402@v.loewis.de>

> The only environment variables that don't appear in the shell output 
> from the env command are INFOPATH, MAKE_MODE and PLAT. I am still flummoxed.

At this point, I'd recommend to perform a cygwin update; with Cygwin,
these problems often go away with an update.

If that doesn't help, you can ask on the Cygwin list also; to analyse
this further, ISTM one will need to debug the internals of cygwin.

One thing you could try is to add -v to the list of gcc options;
you can then see whether gcc is progressing correctly.

Regards,
Martin

From steve at holdenweb.com  Wed Aug  1 12:49:56 2007
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 01 Aug 2007 06:49:56 -0400
Subject: [Python-Dev] Cygwin: Problem detecting subprocess termination
 after _spawn_posix in distutils?
In-Reply-To: <46B01E90.8090402@v.loewis.de>
References: <f8lcd3$9ou$1@sea.gmane.org>
	<46AFAD41.9060203@v.loewis.de>	<46AFDF97.3090909@holdenweb.com>
	<46B01E90.8090402@v.loewis.de>
Message-ID: <46B06554.6020501@holdenweb.com>

Martin v. L?wis wrote:
>> The only environment variables that don't appear in the shell output 
>> from the env command are INFOPATH, MAKE_MODE and PLAT. I am still flummoxed.
> 
> At this point, I'd recommend to perform a cygwin update; with Cygwin,
> these problems often go away with an update.
> 
I updated Cygwin and did a rebaseall before posting.

> If that doesn't help, you can ask on the Cygwin list also; to analyse
> this further, ISTM one will need to debug the internals of cygwin.
> 
I posted on Cygwin before asking here.

> One thing you could try is to add -v to the list of gcc options;
> you can then see whether gcc is progressing correctly.
> 
I'll do that, though I have reason to believe the gcc *is* terminating 
and _spawn_posix isn't detecting the end of the process. At the very 
least we should get another test out of this dreadfully irritating bug.

Thanks again for looking at this.

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------


From dalcinl at gmail.com  Wed Aug  1 17:17:46 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Aug 2007 12:17:46 -0300
Subject: [Python-Dev] Python Package Index hostname change
In-Reply-To: <46B01737.60608@v.loewis.de>
References: <46AFA27A.4090700@v.loewis.de>
	<e7ba66e40707311809m75f4cefwd7b1cebc02240ce3@mail.gmail.com>
	<ca471dc20707311848o4b2616d6x4a57efe4be7ad926@mail.gmail.com>
	<46B01737.60608@v.loewis.de>
Message-ID: <e7ba66e40708010817gbd05bb0tcfaeea6b715041a2@mail.gmail.com>

Martin, could you please verify if this change did not break the
download counter?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From martin at v.loewis.de  Wed Aug  1 21:45:21 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 01 Aug 2007 21:45:21 +0200
Subject: [Python-Dev] Python Package Index hostname change
In-Reply-To: <e7ba66e40708010817gbd05bb0tcfaeea6b715041a2@mail.gmail.com>
References: <46AFA27A.4090700@v.loewis.de>	
	<e7ba66e40707311809m75f4cefwd7b1cebc02240ce3@mail.gmail.com>	
	<ca471dc20707311848o4b2616d6x4a57efe4be7ad926@mail.gmail.com>	
	<46B01737.60608@v.loewis.de>
	<e7ba66e40708010817gbd05bb0tcfaeea6b715041a2@mail.gmail.com>
Message-ID: <46B0E2D1.5010301@v.loewis.de>

> Martin, could you please verify if this change did not break the
> download counter?

Do you have reason to believe that it did? It should not have.

Regards,
Martin

From facundobatista at gmail.com  Thu Aug  2 20:11:47 2007
From: facundobatista at gmail.com (Facundo Batista)
Date: Thu, 2 Aug 2007 15:11:47 -0300
Subject: [Python-Dev] NotImplemented comparisons
Message-ID: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>

People:

Pablo Hoffman opened this bug: "[1764761] Decimal comparison with None
fails in Windows".

It's not a Decimal problem, see the differente behaviour of this basic
test in Linux and Windows:

Python 2.5.1 (r251:54863, May  2 2007, 16:56:35) [GCC 4.1.2 (Ubuntu
4.1.2-0ubuntu4)] on linux2
>>> class C(object):
...     def __cmp__(self, other):
...         return NotImplemented
...
>>> c = C()
>>> print c < None
False
>>> print NotImplemented < None
False


Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)] on win32
>>> class C(object):
    def __cmp__(self, other):
        return NotImplemented

>>> c = C()
>>> print c < None
True
>>> print NotImplemented < None
False


Here's where I stop: don't know where to keep looking... Does somebody
know why is a difference here?

Furthermore, we can check that is a problem regarding __cmp__:

>>> class C(object):
    def __cmp__(self, other):
        return NotImplemented
    def m(self):
	return NotImplemented

>>> c = C()
>>> print c < None
True
>>> print c.m() < None
False


This is not the first time I find an issue through Decimal regarding
NotImplemented, there was this thread:

    http://mail.python.org/pipermail/python-dev/2005-December/059046.html

, but I don't know if that's a separate issue or not.

Thanks for your help!

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From guido at python.org  Thu Aug  2 20:35:44 2007
From: guido at python.org (Guido van Rossum)
Date: Thu, 2 Aug 2007 11:35:44 -0700
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
Message-ID: <ca471dc20708021135g183fc2b9i8008ff3aa1bcf4c0@mail.gmail.com>

NonImplemented isn't treated as special when returned by __cmp__();
__cmp__ is not considered a binary operator like __add__. (__lt__ and
friends *do* get treated as such -- but instead of __rlt__ we use
__gt__, etc.)

--Guido

On 8/2/07, Facundo Batista <facundobatista at gmail.com> wrote:
> People:
>
> Pablo Hoffman opened this bug: "[1764761] Decimal comparison with None
> fails in Windows".
>
> It's not a Decimal problem, see the differente behaviour of this basic
> test in Linux and Windows:
>
> Python 2.5.1 (r251:54863, May  2 2007, 16:56:35) [GCC 4.1.2 (Ubuntu
> 4.1.2-0ubuntu4)] on linux2
> >>> class C(object):
> ...     def __cmp__(self, other):
> ...         return NotImplemented
> ...
> >>> c = C()
> >>> print c < None
> False
> >>> print NotImplemented < None
> False
>
>
> Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
> (Intel)] on win32
> >>> class C(object):
>     def __cmp__(self, other):
>         return NotImplemented
>
> >>> c = C()
> >>> print c < None
> True
> >>> print NotImplemented < None
> False
>
>
> Here's where I stop: don't know where to keep looking... Does somebody
> know why is a difference here?
>
> Furthermore, we can check that is a problem regarding __cmp__:
>
> >>> class C(object):
>     def __cmp__(self, other):
>         return NotImplemented
>     def m(self):
>         return NotImplemented
>
> >>> c = C()
> >>> print c < None
> True
> >>> print c.m() < None
> False
>
>
> This is not the first time I find an issue through Decimal regarding
> NotImplemented, there was this thread:
>
>     http://mail.python.org/pipermail/python-dev/2005-December/059046.html
>
> , but I don't know if that's a separate issue or not.
>
> Thanks for your help!
>
> --
> .    Facundo
>
> Blog: http://www.taniquetil.com.ar/plog/
> PyAr: http://www.python.org/ar/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From facundobatista at gmail.com  Thu Aug  2 21:14:37 2007
From: facundobatista at gmail.com (Facundo Batista)
Date: Thu, 2 Aug 2007 16:14:37 -0300
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <e04bdf310708021213g421250e6u7edcfcfb8038e68d@mail.gmail.com>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
	<ca471dc20708021135g183fc2b9i8008ff3aa1bcf4c0@mail.gmail.com>
	<e04bdf310708021213g421250e6u7edcfcfb8038e68d@mail.gmail.com>
Message-ID: <e04bdf310708021214u3ae177b5w2b76a783fcfe695c@mail.gmail.com>

2007/8/2, Guido van Rossum <guido at python.org>:

> NonImplemented isn't treated as special when returned by __cmp__();
> __cmp__ is not considered a binary operator like __add__. (__lt__ and
> friends *do* get treated as such -- but instead of __rlt__ we use
> __gt__, etc.)

I understand that is tricky how NotImplemented and comparisons interact.

But how do you explain the difference in behaviour between Linux and Windows?

Thanks!

--
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From p.f.moore at gmail.com  Thu Aug  2 21:45:00 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 2 Aug 2007 20:45:00 +0100
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <e04bdf310708021214u3ae177b5w2b76a783fcfe695c@mail.gmail.com>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
	<ca471dc20708021135g183fc2b9i8008ff3aa1bcf4c0@mail.gmail.com>
	<e04bdf310708021213g421250e6u7edcfcfb8038e68d@mail.gmail.com>
	<e04bdf310708021214u3ae177b5w2b76a783fcfe695c@mail.gmail.com>
Message-ID: <79990c6b0708021245t3ed790fdwf12f5bca227a0b5d@mail.gmail.com>

On 02/08/07, Facundo Batista <facundobatista at gmail.com> wrote:
> I understand that is tricky how NotImplemented and comparisons interact.
>
> But how do you explain the difference in behaviour between Linux and Windows?

A wild guess: c < None falls back to checking c.__cmp__(None) < 0.
This translates to NotImplemented < 0, and as the ordering of built in
types is implementation dependent, maybe that explains the difference
between Windows and Linux?

Paul.

From g.brandl at gmx.net  Thu Aug  2 22:06:14 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Aug 2007 22:06:14 +0200
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <ca471dc20708021135g183fc2b9i8008ff3aa1bcf4c0@mail.gmail.com>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
	<ca471dc20708021135g183fc2b9i8008ff3aa1bcf4c0@mail.gmail.com>
Message-ID: <f8tdff$41d$1@sea.gmane.org>

Guido van Rossum schrieb:
> NonImplemented isn't treated as special when returned by __cmp__();
> __cmp__ is not considered a binary operator like __add__. (__lt__ and
> friends *do* get treated as such -- but instead of __rlt__ we use
> __gt__, etc.)

But if it's not treated as special, why doesn't the comparison raise an
exception, like when __cmp__ returns "foo", for example?

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From tjreedy at udel.edu  Thu Aug  2 22:11:35 2007
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 2 Aug 2007 16:11:35 -0400
Subject: [Python-Dev] NotImplemented comparisons
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
Message-ID: <f8tdpl$553$1@sea.gmane.org>


"Facundo Batista" <facundobatista at gmail.com> wrote in message 
news:e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2 at mail.gmail.com...
| >>> class C(object):
| ...     def __cmp__(self, other):
| ...         return NotImplemented
| ...

Given that you 'should' return an int, doing elsewise has undefined 
results.

| >>> c = C()
| >>> print c < None

I presume that this translates into c.__compare(None) <  0 which becomes 
NotImplemented < 0.  The result of that is undefined and interpreter 
dependent.

| >>> print NotImplemented < None

As is this.  There is no reason to expect the two comparisons 
(NotImplemented to 0 and None) to give the same or different results.

| Does somebody know why is a difference here?

Different interpreters, different arbitrary results.  I believe checking 
the ids of the right objects (the type objects, I have read) would explain.

| Furthermore, we can check that is a problem regarding __cmp__:
|
| >>> class C(object):
|    def __cmp__(self, other):
|        return NotImplemented
|    def m(self):
| return NotImplemented
|
| >>> c = C()
| >>> print c < None
| True
| >>> print c.m() < None
| False

This is still NotImplemented < 0 versus NotImplemented < None.  As I 
understand, such nonsense comparisions will raise exceptions in 3.0.

tjr




From g.brandl at gmx.net  Thu Aug  2 22:15:46 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Aug 2007 22:15:46 +0200
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <e04bdf310708021214u3ae177b5w2b76a783fcfe695c@mail.gmail.com>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>	<ca471dc20708021135g183fc2b9i8008ff3aa1bcf4c0@mail.gmail.com>	<e04bdf310708021213g421250e6u7edcfcfb8038e68d@mail.gmail.com>
	<e04bdf310708021214u3ae177b5w2b76a783fcfe695c@mail.gmail.com>
Message-ID: <f8te1b$61a$1@sea.gmane.org>

Facundo Batista schrieb:
> 2007/8/2, Guido van Rossum <guido at python.org>:
> 
>> NonImplemented isn't treated as special when returned by __cmp__();
>> __cmp__ is not considered a binary operator like __add__. (__lt__ and
>> friends *do* get treated as such -- but instead of __rlt__ we use
>> __gt__, etc.)
> 
> I understand that is tricky how NotImplemented and comparisons interact.
> 
> But how do you explain the difference in behaviour between Linux and Windows?

I now investigated that, and it seems that if you return NotImplemented from
a __cmp__() function, and the other's __cmp__() isn't helpful either, you end
up comparing the addresses of the objects (in your case c and None) -- the
outcome of which is not consistent across machines or sessions.


Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From facundobatista at gmail.com  Thu Aug  2 22:17:18 2007
From: facundobatista at gmail.com (Facundo Batista)
Date: Thu, 2 Aug 2007 17:17:18 -0300
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <ca471dc20708021245p1061d3a5t9e068053b5fd8dd@mail.gmail.com>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
	<ca471dc20708021135g183fc2b9i8008ff3aa1bcf4c0@mail.gmail.com>
	<e04bdf310708021213g421250e6u7edcfcfb8038e68d@mail.gmail.com>
	<ca471dc20708021245p1061d3a5t9e068053b5fd8dd@mail.gmail.com>
Message-ID: <e04bdf310708021317t2fc091f9p8652f59efa417eeb@mail.gmail.com>

2007/8/2, Paul Moore <p.f.moore at gmail.com>:

> A wild guess: c < None falls back to checking c.__cmp__(None) < 0.
> This translates to NotImplemented < 0, and as the ordering of built in
> types is implementation dependent, maybe that explains the difference
> between Windows and Linux?

"NotImplemented < 0" returns False, which is ok, but different from "c < None"


2007/8/2, Guido van Rossum <guido at python.org>:

> > But how do you explain the difference in behaviour between Linux and Windows?
>
> Perhaps the comparison compares the objects' address.

No, because NotImplemented and None are always the same: if this is
the problem Linux and Windows could be different but they would be
consistent with themselves (and Windows is not coherent with itself).


Bottom line: I can easily fix Decimal to handle this special case, the
point is that maybe we have a lower level bug here...

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From g.brandl at gmx.net  Thu Aug  2 23:25:27 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Aug 2007 23:25:27 +0200
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <f8tdpl$553$1@sea.gmane.org>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
	<f8tdpl$553$1@sea.gmane.org>
Message-ID: <f8ti3v$ip0$1@sea.gmane.org>

Terry Reedy schrieb:
> "Facundo Batista" <facundobatista at gmail.com> wrote in message 
> news:e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2 at mail.gmail.com...
> | >>> class C(object):
> | ...     def __cmp__(self, other):
> | ...         return NotImplemented
> | ...
> 
> Given that you 'should' return an int, doing elsewise has undefined 
> results.

Returning anything other than an int or NotImplemented raises an exception.
NotImplemented seems to be special cased so that the other object's
__cmp__ can be tried too.

> | >>> c = C()
> | >>> print c < None
> 
> I presume that this translates into c.__compare(None) <  0 which becomes 
> NotImplemented < 0.  The result of that is undefined and interpreter 
> dependent.

No, it becomes id(c) < id(None). See half_compare in Objects/typeobject.c.


> This is still NotImplemented < 0 versus NotImplemented < None.  As I 
> understand, such nonsense comparisions will raise exceptions in 3.0.

Yes, fortunately.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From facundobatista at gmail.com  Fri Aug  3 00:15:13 2007
From: facundobatista at gmail.com (Facundo Batista)
Date: Thu, 2 Aug 2007 19:15:13 -0300
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <f8tdpl$553$1@sea.gmane.org>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
	<f8tdpl$553$1@sea.gmane.org>
Message-ID: <e04bdf310708021515s587eb5er1eb07efcf7f2bfc7@mail.gmail.com>

2007/8/2, Terry Reedy <tjreedy at udel.edu>:

> Given that you 'should' return an int, doing elsewise has undefined
> results.

I'll fix decimal to always return sane values from __cmp__, :)

Thank you all!

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From guido at python.org  Fri Aug  3 00:27:18 2007
From: guido at python.org (Guido van Rossum)
Date: Thu, 2 Aug 2007 15:27:18 -0700
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <f8ti3v$ip0$1@sea.gmane.org>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
	<f8tdpl$553$1@sea.gmane.org> <f8ti3v$ip0$1@sea.gmane.org>
Message-ID: <ca471dc20708021527t6d01b0bcyec7d203fab55cd7a@mail.gmail.com>

On 8/2/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Returning anything other than an int or NotImplemented raises an exception.
> NotImplemented seems to be special cased so that the other object's
> __cmp__ can be tried too.

Oops, sorry for the misinformation. :-(

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From nnorwitz at gmail.com  Fri Aug  3 08:07:29 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 2 Aug 2007 23:07:29 -0700
Subject: [Python-Dev] T_PYSSIZET in Include/structmember.h can be hidden
Message-ID: <ee2a432c0708022307n37bb89f8m31fe59b9e48429e@mail.gmail.com>

Martin,

Do you know why T_PYSSIZET is inside a #ifdef HAVE_LONG_LONG?  That
seems like a mistake.  Here's the code:

#ifdef HAVE_LONG_LONG
#define T_LONGLONG      17
#define T_ULONGLONG     18
#define T_PYSSIZET       19 /* Py_ssize_t */
#endif /* HAVE_LONG_LONG */

ISTM, that T_PYSSIZET should be after the #endif.  Was this a mistake
or intentional?

n

From walter at livinglogic.de  Fri Aug  3 08:34:46 2007
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Fri, 03 Aug 2007 08:34:46 +0200
Subject: [Python-Dev] T_PYSSIZET in Include/structmember.h can be hidden
In-Reply-To: <ee2a432c0708022307n37bb89f8m31fe59b9e48429e@mail.gmail.com>
References: <ee2a432c0708022307n37bb89f8m31fe59b9e48429e@mail.gmail.com>
Message-ID: <46B2CC86.1020802@livinglogic.de>

Neal Norwitz wrote:
> Martin,
> 
> Do you know why T_PYSSIZET is inside a #ifdef HAVE_LONG_LONG?  That
> seems like a mistake.  Here's the code:
> 
> #ifdef HAVE_LONG_LONG
> #define T_LONGLONG      17
> #define T_ULONGLONG     18
> #define T_PYSSIZET       19 /* Py_ssize_t */
> #endif /* HAVE_LONG_LONG */
> 
> ISTM, that T_PYSSIZET should be after the #endif.  Was this a mistake
> or intentional?

That was my mistake. Iy should be outside of the #ifdef.

Servus,
    Walter

From g.brandl at gmx.net  Fri Aug  3 10:23:55 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 03 Aug 2007 10:23:55 +0200
Subject: [Python-Dev] make iter() return an empty iterator?
Message-ID: <f8uomj$gm0$1@sea.gmane.org>

Sure, you could use ``iter(())`` or ``iter([])``, but for consistency's sake
wouldn't it make sense for ``iter()`` to return an empty iterator, as ``str()``
returns an empty string etc.?

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From andrew-pythondev at puzzling.org  Fri Aug  3 11:10:20 2007
From: andrew-pythondev at puzzling.org (Andrew Bennetts)
Date: Fri, 3 Aug 2007 19:10:20 +1000
Subject: [Python-Dev] make iter() return an empty iterator?
In-Reply-To: <f8uomj$gm0$1@sea.gmane.org>
References: <f8uomj$gm0$1@sea.gmane.org>
Message-ID: <20070803091020.GA25389@steerpike.home.puzzling.org>

Georg Brandl wrote:
> Sure, you could use ``iter(())`` or ``iter([])``, but for consistency's sake
> wouldn't it make sense for ``iter()`` to return an empty iterator, as ``str()``
> returns an empty string etc.?

I had no idea that "str()" or "int()" would do that.  "file()" certainly
doesn't! :)

I don't really think there's much reason to make "iter()" work.  As you say,
"iter([])" works just fine.  For those rare times you want an empty iterator, I
don't think the two extra characters is much of a price to pay.

-Andrew.


From rrr at ronadam.com  Fri Aug  3 12:11:25 2007
From: rrr at ronadam.com (Ron Adam)
Date: Fri, 03 Aug 2007 05:11:25 -0500
Subject: [Python-Dev] make iter() return an empty iterator?
In-Reply-To: <f8uomj$gm0$1@sea.gmane.org>
References: <f8uomj$gm0$1@sea.gmane.org>
Message-ID: <46B2FF4D.8060207@ronadam.com>


Georg Brandl wrote:
> Sure, you could use ``iter(())`` or ``iter([])``, but for consistency's sake
> wouldn't it make sense for ``iter()`` to return an empty iterator, as ``str()``
> returns an empty string etc.?
> 
> Georg

There is a difference.

 >>> type(iter)
<type 'builtin_function_or_method'>
 >>> type(str)
<type 'type'>
 >>> type(int)
<type 'type'>
 >>> type(list)
<type 'type'>

Cheers,
    Ron


From jon+python-dev at unequivocal.co.uk  Fri Aug  3 14:29:47 2007
From: jon+python-dev at unequivocal.co.uk (Jon Ribbens)
Date: Fri, 3 Aug 2007 13:29:47 +0100
Subject: [Python-Dev] Pythreads and BSD descendants
In-Reply-To: <20070726160837.GA24583@lairds.us>
References: <20070726160837.GA24583@lairds.us>
Message-ID: <20070803122947.GM11696@snowy.squish.net>

On Thu, Jul 26, 2007 at 04:08:37PM +0000, Cameron Laird wrote:
> Folklore that I remember so unreliably I avoid trying to repeat it here
> held that Python threading had problems on BSD and allied Unixes.  What's
> the status of this?  I suspect the answer is, "Everything works, and the
> only real problem ever was that *signals* have different semantics under
> Linux and *BSD."  Anyone who can answer explicitly, though, would repre-
> sent a help to me.

This is just my personal opinion, but I suspect that this is perhaps
because people have *tried* threading more in Python than in many
other languages, because Python makes it particularly easy.

I've certainly had the experience that multithreaded stuff I have
tried has sometimes had problems under various OSes (Linux, Solaris,
OpenBSD, etc) due to operating system bugs with threading in general
rather than Python problems per se.

From facundobatista at gmail.com  Fri Aug  3 15:01:47 2007
From: facundobatista at gmail.com (Facundo Batista)
Date: Fri, 3 Aug 2007 10:01:47 -0300
Subject: [Python-Dev]  make iter() return an empty iterator?
In-Reply-To: <e04bdf310708030601i4005ef1dh3fafc1c88ebad893@mail.gmail.com>
References: <f8uomj$gm0$1@sea.gmane.org>
	<20070803091020.GA25389@steerpike.home.puzzling.org>
	<e04bdf310708030601i4005ef1dh3fafc1c88ebad893@mail.gmail.com>
Message-ID: <e04bdf310708030601n570ad3a3y723626b5efcf2f36@mail.gmail.com>

2007/8/3, Andrew Bennetts <andrew-pythondev at puzzling.org>:

> I don't really think there's much reason to make "iter()" work.  As you say,

What bad thing could happen if we make iter() work? If nothing, we
should ask ourselves: which is the more intuitive behaviour to expect
of iter()? To raise an exception or to return an empty iterator?

I'm +0 for the latter.

--
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From facundobatista at gmail.com  Fri Aug  3 15:04:24 2007
From: facundobatista at gmail.com (Facundo Batista)
Date: Fri, 3 Aug 2007 10:04:24 -0300
Subject: [Python-Dev] NotImplemented comparisons
In-Reply-To: <e04bdf310708021515s587eb5er1eb07efcf7f2bfc7@mail.gmail.com>
References: <e04bdf310708021111g2870662bo5c6fdb3c1c68a9c2@mail.gmail.com>
	<f8tdpl$553$1@sea.gmane.org>
	<e04bdf310708021515s587eb5er1eb07efcf7f2bfc7@mail.gmail.com>
Message-ID: <e04bdf310708030604g68d867c7yf8bcb212798e361d@mail.gmail.com>

2007/8/2, Facundo Batista <facundobatista at gmail.com>:

> > Given that you 'should' return an int, doing elsewise has undefined
> > results.
>
> I'll fix decimal to always return sane values from __cmp__, :)

Done, thanks again everybody!

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From bioinformed at gmail.com  Fri Aug  3 15:14:20 2007
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Fri, 3 Aug 2007 09:14:20 -0400
Subject: [Python-Dev] make iter() return an empty iterator?
In-Reply-To: <e04bdf310708030601n570ad3a3y723626b5efcf2f36@mail.gmail.com>
References: <f8uomj$gm0$1@sea.gmane.org>
	<20070803091020.GA25389@steerpike.home.puzzling.org>
	<e04bdf310708030601i4005ef1dh3fafc1c88ebad893@mail.gmail.com>
	<e04bdf310708030601n570ad3a3y723626b5efcf2f36@mail.gmail.com>
Message-ID: <2e1434c10708030614t5a188a81w7f338483f151ae96@mail.gmail.com>

On 8/3/07, Facundo Batista <facundobatista at gmail.com> wrote:
>
> 2007/8/3, Andrew Bennetts <andrew-pythondev at puzzling.org>:
>
> > I don't really think there's much reason to make "iter()" work.  As you
> say,
>
> What bad thing could happen if we make iter() work? If nothing, we
> should ask ourselves: which is the more intuitive behaviour to expect
> of iter()? To raise an exception or to return an empty iterator?
>
> I'm +0 for the latter.
>

-1.  I'm a heavy user of iterators on finite and infinite streams and, for
me, iter() is an error that I do not want to paper over.  The alternate
logic implies, e.g., len() should return 0.

-Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070803/40c517a6/attachment.htm 

From steve at holdenweb.com  Fri Aug  3 15:33:25 2007
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 03 Aug 2007 09:33:25 -0400
Subject: [Python-Dev] make iter() return an empty iterator?
In-Reply-To: <2e1434c10708030614t5a188a81w7f338483f151ae96@mail.gmail.com>
References: <f8uomj$gm0$1@sea.gmane.org>	<20070803091020.GA25389@steerpike.home.puzzling.org>	<e04bdf310708030601i4005ef1dh3fafc1c88ebad893@mail.gmail.com>	<e04bdf310708030601n570ad3a3y723626b5efcf2f36@mail.gmail.com>
	<2e1434c10708030614t5a188a81w7f338483f151ae96@mail.gmail.com>
Message-ID: <f8var6$b81$1@sea.gmane.org>

Kevin Jacobs <jacobs at bioinformed.com> wrote:
> On 8/3/07, *Facundo Batista* <facundobatista at gmail.com 
> <mailto:facundobatista at gmail.com>> wrote:
> 
>     2007/8/3, Andrew Bennetts <andrew-pythondev at puzzling.org
>     <mailto:andrew-pythondev at puzzling.org>>:
> 
>      > I don't really think there's much reason to make "iter()"
>     work.  As you say,
> 
>     What bad thing could happen if we make iter() work? If nothing, we
>     should ask ourselves: which is the more intuitive behaviour to expect
>     of iter()? To raise an exception or to return an empty iterator?
> 
>     I'm +0 for the latter.
> 
> 
> -1.  I'm a heavy user of iterators on finite and infinite streams and, 
> for me, iter() is an error that I do not want to paper over.  The 
> alternate logic implies, e.g ., len() should return 0.
> 
-1 here too. iter() should have an argument just like sum() and len().

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------


From pc at gafol.net  Fri Aug  3 17:21:43 2007
From: pc at gafol.net (Paul Colomiets)
Date: Fri, 03 Aug 2007 18:21:43 +0300
Subject: [Python-Dev] Pythreads and BSD descendants
In-Reply-To: <20070726160837.GA24583@lairds.us>
References: <20070726160837.GA24583@lairds.us>
Message-ID: <46B34807.9020307@gafol.net>

Cameron Laird wrote:
> Folklore that I remember so unreliably I avoid trying to repeat it here
> held that Python threading had problems on BSD and allied Unixes.  What's
> the status of this?  I suspect the answer is, "Everything works, and the
> only real problem ever was that *signals* have different semantics under
> Linux and *BSD."  Anyone who can answer explicitly, though, would repre-
> sent a help to me.
>   
I use Python with multithreading applications on FreeBSD
for several years, and really single problem I've discovered
is that default stack size for new threads is small for the
default recursion limit. It can be easily fixed in Python 2.5.

Apart from that everything works OK for me.


From guido at python.org  Fri Aug  3 19:07:24 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 3 Aug 2007 10:07:24 -0700
Subject: [Python-Dev] make iter() return an empty iterator?
In-Reply-To: <f8var6$b81$1@sea.gmane.org>
References: <f8uomj$gm0$1@sea.gmane.org>
	<20070803091020.GA25389@steerpike.home.puzzling.org>
	<e04bdf310708030601i4005ef1dh3fafc1c88ebad893@mail.gmail.com>
	<e04bdf310708030601n570ad3a3y723626b5efcf2f36@mail.gmail.com>
	<2e1434c10708030614t5a188a81w7f338483f151ae96@mail.gmail.com>
	<f8var6$b81$1@sea.gmane.org>
Message-ID: <ca471dc20708031007v26de959ckf755db687de08252@mail.gmail.com>

On 8/3/07, Steve Holden <steve at holdenweb.com> wrote:
> Kevin Jacobs <jacobs at bioinformed.com> wrote:
> > On 8/3/07, *Facundo Batista* <facundobatista at gmail.com
> > <mailto:facundobatista at gmail.com>> wrote:
> >
> >     2007/8/3, Andrew Bennetts <andrew-pythondev at puzzling.org
> >     <mailto:andrew-pythondev at puzzling.org>>:
> >
> >      > I don't really think there's much reason to make "iter()"
> >     work.  As you say,
> >
> >     What bad thing could happen if we make iter() work? If nothing, we
> >     should ask ourselves: which is the more intuitive behaviour to expect
> >     of iter()? To raise an exception or to return an empty iterator?
> >
> >     I'm +0 for the latter.
> >
> >
> > -1.  I'm a heavy user of iterators on finite and infinite streams and,
> > for me, iter() is an error that I do not want to paper over.  The
> > alternate logic implies, e.g ., len() should return 0.
> >
> -1 here too. iter() should have an argument just like sum() and len().

Amen.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From alex.neundorf at kitware.com  Fri Aug  3 19:34:42 2007
From: alex.neundorf at kitware.com (Alexander Neundorf)
Date: Fri, 3 Aug 2007 13:34:42 -0400
Subject: [Python-Dev] Building Python with CMake
In-Reply-To: <f78mad$876$1@sea.gmane.org>
References: <200707131359.17030.alex.neundorf@kitware.com>
	<e04bdf310707131153x780293b3p9561b8e5d9952069@mail.gmail.com>
	<f78mad$876$1@sea.gmane.org>
Message-ID: <200708031334.42813.alex.neundorf@kitware.com>

On Friday 13 July 2007 16:11, Giovanni Bajo wrote:
...
> Because it would be a single unified build system instead of having two
> build systems like we have one (UNIX and Windows).
>
> Also, it would be much easier to maintain because Visual Studio projects
> are generated from a simple description, while right now if you want to
> change something you need to go through the hassle of defining it within
> the Visual Studio GUI.
>
> Consider for instance if you want to change the Windows build so that a
> builtin module is compiled as an external .pyd instead. Right now, you need
> to go through the hassle of manually defining a new project, setting all
> the include/libraries dependencies correctly, ecc. ecc. With CMake or a
> similar tool, it would be a matter of a couple of textual line changes.
>
> [ I'll also remember that "ease of maintanance for developers" is the #1
> reason for having a 2.1Mb python25.dll under Windows, which I would really
> love to reduce. ]

I thought I'll keep you updated, so:

attached you can find the current cmake files I use for Python 2.5.1.
They work for eCos, Linux, BlueGene and Windows (which doesn't mean everything 
is supported or installed, but they create working python interpreters and 
libs and install the most required files).

Compared to the first version they now contain more install rules, the 
platform path is not hardcoded anymore and it contains a basic setup for 
creating source and binary packages (tgz, Debian, Nullsoft installer, 
PackageMaker depending on your cmake version).

Bye
Alex
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Python-2.5.1-cmakefiles.tar.gz
Type: application/x-tgz
Size: 16101 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20070803/3e04c813/attachment.bin 

From martin at v.loewis.de  Sat Aug  4 00:46:39 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Aug 2007 00:46:39 +0200
Subject: [Python-Dev] Pythreads and BSD descendants
In-Reply-To: <20070726160837.GA24583@lairds.us>
References: <20070726160837.GA24583@lairds.us>
Message-ID: <46B3B04F.7060100@v.loewis.de>

Cameron Laird schrieb:
> Folklore that I remember so unreliably I avoid trying to repeat it here
> held that Python threading had problems on BSD and allied Unixes.  What's
> the status of this?

The problem that people run into again and again is the stack size. The
BSDs allow for so little stack so that even the quite conservative
estimates of Python as to how many recursions you can have are
incorrect, and you get an interpreter crash rather than a RuntimeError
(as you should).

Furthermore, every time we decrease the that number, the next system
release somehow manages to make the limit even smaller. This was
never properly analyzed; I suspect that the stack usage of Python
increases, either due to compiler changes or due to change to Python
itself.

Another annoyance is the ongoing battle with Posix; the BSDs have
not been very accepting towards Posix for many years. This resulted
in an interpretation of Posix where defining _XOPEN_SOURCE hides
many system interfaces, resulting in these system interfaces either
not being present, or compilation to fail. I consider this a bug
in the system: compilation should *never* fail if you define
_XOPEN_SOURCE, and additional interfaces should be available if
requested (that requires a way to request them). The work-around
was to not define _XOPEN_SOURCE for those buggy system releases,
hoping that the next release would fix the bug. Over the years,
the maintainers of these systems seem to have come to a better
understanding, so they offer various custom _SOURCE macros
(_NETBSD_SOURCE, __BSD_VISIBLE). The latest addition here was
OpenBSD, which now supports _BSD_SOURCE (apparently following a
tradition set by GNU libc, and perhaps others). So I hope this is
fixed for good now (except that FreeBSD may decide to break
__BSD_VISIBLE the same way it got "broken" in OpenBSD, so we
need to add their "official" feature selection macro once we
find out what that is). The same problem exists of course on many
other systems, but those solved the problem long ago (e.g.
_GNU_SOURCE - glibc, _BSD_TYPES - Irix)

Regards,
Martin

From dalcinl at gmail.com  Sat Aug  4 01:44:23 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 3 Aug 2007 20:44:23 -0300
Subject: [Python-Dev] [Python-3000] optimizing [x]range
In-Reply-To: <f8t06h$efu$1@sea.gmane.org>
References: <1d85506f0707280806n1764151cx4961a0573dda435e@mail.gmail.com>
	<f8srod$5m7$1@sea.gmane.org> <f8t06h$efu$1@sea.gmane.org>
Message-ID: <e7ba66e40708031644j664ed699ge3db977eccc985d7@mail.gmail.com>

On 8/2/07, Stargaming <stargaming at gmail.com> wrote:
> >> made into an O(1) operation. here's a demo code (it should be trivial
> >> to implement this in CPython)
> [snipped algorithm]

Did you taked into account that your patch is not backward compatible
with py2.5?? Just try to do this with your patch,

$ python
Python 2.5.1 (r251:54863, Jun  1 2007, 12:15:26)
>>> class A:
...   def __eq__(self, other):
...     return other == 3
...
>>> A() in xrange(3)
False
>>> A() in xrange(4)
True
>>>

I know, my example is biased, but I have to insist. With this patch,
'a in xrange' will in general not be the same as 'a in range(...)'. I
am fine with this for py3k, but not sure if all people will agree on
this for python 2.6.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From kbk at shore.net  Sat Aug  4 07:26:32 2007
From: kbk at shore.net (Kurt B. Kaiser)
Date: Sat, 4 Aug 2007 01:26:32 -0400 (EDT)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200708040526.l745QWEW014541@hampton.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  404 open ( +5) /  3847 closed (+11) /  4251 total (+16)
Bugs    : 1059 open ( +3) /  6784 closed ( +8) /  7843 total (+11)
RFE     :  263 open ( +0) /   295 closed ( +1) /   558 total ( +1)

New / Reopened Patches
______________________

struni: Fix test_aepack by converting 4cc's to bytes  (2007-07-26)
CLOSED http://python.org/sf/1761465  reopened by  gvanrossum

distutils.util.get_platform() return value on 64bit Windows  (2007-07-27)
       http://python.org/sf/1761786  opened by  Mark Hammond

Some fix abount _WIN32_WINNT  (2007-07-27)
CLOSED http://python.org/sf/1761803  opened by  Hirokazu Yamamoto

struni: test_xml_etree.py  (2007-07-27)
CLOSED http://python.org/sf/1762412  opened by  Joe Gregorio

unable to serialize Infinity or NaN on ARM using marshal  (2007-07-28)
       http://python.org/sf/1762561  opened by  Matthias Klose

struni: test_urllib2, test_cookielib  (2007-07-29)
CLOSED http://python.org/sf/1762940  opened by  Joe Gregorio

socket close fixed  (2007-07-29)
CLOSED http://python.org/sf/1763387  opened by  Hasan Diwan

tiny addition to peephole optimizer  (2007-07-31)
       http://python.org/sf/1764087  opened by  Paul Pogonyshev

Fix for test_socketserver for Py3k  (2007-07-31)
CLOSED http://python.org/sf/1764815  opened by  ??PC??

generic and more efficient removal of unreachable code  (2007-08-01)
       http://python.org/sf/1764986  opened by  Paul Pogonyshev

logging: delay_fh option and configuration kwargs  (2007-08-01)
       http://python.org/sf/1765140  opened by  Chris Leary

small improvement for peephole conditional jump optimizer  (2007-08-01)
       http://python.org/sf/1765558  opened by  Paul Pogonyshev

urllib2-howto - correction  (2007-08-02)
       http://python.org/sf/1765839  opened by  O.R.Senthil Kumaran

improve xrange.__contains__  (2007-08-02)
       http://python.org/sf/1766304  opened by  Stargaming

Fix for test_zipimport  (2007-08-03)
CLOSED http://python.org/sf/1766592  opened by  ??PC??

Make xmlrpc use HTTP/1.1 and keepalive  (2007-08-04)
       http://python.org/sf/1767370  opened by  Donovan Baarda

test_csv struni fixes + unicode support in _csv  (2007-08-03)
       http://python.org/sf/1767398  opened by  Adam Hupp

Patches Closed
______________

struni: Fix test_aepack by converting 4cc's to bytes  (2007-07-26)
       http://python.org/sf/1761465  closed by  gvanrossum

struni: Fix test_aepack by converting 4cc's to bytes  (2007-07-26)
       http://python.org/sf/1761465  closed by  gvanrossum

Some fix abount _WIN32_WINNT  (2007-07-27)
       http://python.org/sf/1761803  closed by  mhammond

struni pulldom: Don't use 'types' to check strings  (2007-07-24)
       http://python.org/sf/1759922  closed by  gvanrossum

struni: test_xml_etree.py  (2007-07-27)
       http://python.org/sf/1762412  closed by  loewis

struni: test_urllib2, test_cookielib  (2007-07-28)
       http://python.org/sf/1762940  closed by  gvanrossum

socket close fixed  (2007-07-30)
       http://python.org/sf/1763387  closed by  facundobatista

itertools.getitem()  (2007-07-08)
       http://python.org/sf/1749857  closed by  rhettinger

Fix for test_socketserver for Py3k  (2007-07-31)
       http://python.org/sf/1764815  closed by  gvanrossum

Fix Decimal.sqrt bugs described in #1725899  (2007-06-22)
       http://python.org/sf/1741308  closed by  facundobatista

Fix for test_zipimport  (2007-08-03)
       http://python.org/sf/1766592  closed by  gvanrossum

urllib2 tests pass  (2007-07-16)
       http://python.org/sf/1755133  closed by  gvanrossum

New / Reopened Bugs
___________________

'exec' does not accept what 'open' returns  (2007-07-28)
       http://python.org/sf/1762972  opened by  Brett Cannon

S.find documentation uses s[start, end] vs. s[start:end]  (2007-07-29)
CLOSED http://python.org/sf/1763149  opened by  Rob

copy 2  (2007-07-30)
       http://python.org/sf/1764044  opened by  robs pythonid

_RLock.__repr__ throws exception  (2007-07-30)
CLOSED http://python.org/sf/1764059  opened by  Greg Kochanski

The -m switch does not use the builtin __main__ module  (2007-07-31)
       http://python.org/sf/1764407  opened by  Nick Coghlan

Decimal comparison with None fails in Windows  (2007-07-31)
CLOSED http://python.org/sf/1764761  opened by  pablohoffman.com

setup.py trashes LDFLAGS  (2007-08-01)
       http://python.org/sf/1765375  opened by  Harald Koenig

poll() returns "status code", not "return code"  (2007-08-02)
       http://python.org/sf/1766421  opened by  sjbrown

os.chmod failure  (2007-08-03)
       http://python.org/sf/1767242  reopened by  rgheck

os.chmod failure  (2007-08-03)
       http://python.org/sf/1767242  opened by  Richard Heck

String.capwords() does not capitalize first word  (2007-08-03)
       http://python.org/sf/1767363  opened by  Saatvik Agarwal

Bugs Closed
___________

No docs for list comparison  (2007-07-25)
       http://python.org/sf/1760423  closed by  gbrandl

SSL-ed sockets don't close correct?  (2004-06-24)
       http://python.org/sf/978833  closed by  loewis

incorrect return value of unicodedata.lookup() - beoynd BMP  (2007-04-21)
       http://python.org/sf/1704793  closed by  loewis

Thread.join() fails to release Lock on KeyboardInterrupt  (2005-03-26)
       http://python.org/sf/1171023  closed by  phansen

S.find documentation uses s[start, end] vs. s[start:end]  (2007-07-29)
       http://python.org/sf/1763149  closed by  gbrandl

_RLock.__repr__ throws exception  (2007-07-31)
       http://python.org/sf/1764059  closed by  ncoghlan

Decimal comparison with None fails in Windows  (2007-07-31)
       http://python.org/sf/1764761  closed by  facundobatista

decimal sqrt method doesn't use round-half-even  (2007-05-25)
       http://python.org/sf/1725899  closed by  facundobatista

New / Reopened RFE
__________________

add new bytecodes: JUMP_IF_{FALSE|TRUE}_AND_POP  (2007-07-31)
       http://python.org/sf/1764638  opened by  Paul Pogonyshev

RFE Closed
__________

add operator.fst and snd functions  (2007-05-27)
       http://python.org/sf/1726697  closed by  rhettinger


From andymac at bullseye.apana.org.au  Sat Aug  4 11:20:45 2007
From: andymac at bullseye.apana.org.au (Andrew MacIntyre)
Date: Sat, 04 Aug 2007 20:20:45 +1100
Subject: [Python-Dev] Pythreads and BSD descendants
In-Reply-To: <46B3B04F.7060100@v.loewis.de>
References: <20070726160837.GA24583@lairds.us> <46B3B04F.7060100@v.loewis.de>
Message-ID: <46B444ED.8090802@bullseye.andymac.org>

Martin v. L?wis wrote:
> Cameron Laird schrieb:
>> Folklore that I remember so unreliably I avoid trying to repeat it here
>> held that Python threading had problems on BSD and allied Unixes.  What's
>> the status of this?
> 
> The problem that people run into again and again is the stack size. The
> BSDs allow for so little stack so that even the quite conservative
> estimates of Python as to how many recursions you can have are
> incorrect, and you get an interpreter crash rather than a RuntimeError
> (as you should).

There are 2 aspects to the thread stack size issue:
- the stack size for the primary thread;
- the stack size for created threads.

I haven't done any investigating for FreeBSD 6.x and later, but I know
that FreeBSD 4.x had a hard coded stack size of 1MB for the primary 
thread in a thread enabled application, which is what Martin's comment 
above particularly applies to.  This affects code that doesn't use 
threads at all, and was particularly painful with REs prior to SRE being 
made non-recursive.

If you build the interpreter without thread support, stack space is 
instead controlled by session limits which are usually generous 
(typically 64MB).

I don't recall exactly FreeBSD's default stack size for threads created 
via pthread_create() but it is fairly small (32kB or 64kB comes to 
mind).    Zope is one application known to be affected by this lack of 
stack size in created threads.  At least the stack size for new threads 
can be adjusted at runtime, and a mechanism for doing this was added to 
Python 2.5.

> Furthermore, every time we decrease the that number, the next system
> release somehow manages to make the limit even smaller. This was
> never properly analyzed; I suspect that the stack usage of Python
> increases, either due to compiler changes or due to change to Python
> itself.

I have seen examples of stack consumption increasing with increasing gcc 
version number, sometimes depending on optimisation choices.

Regards,
Andrew.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia

From jerker.back at telia.com  Sat Aug  4 16:54:33 2007
From: jerker.back at telia.com (=?iso-8859-1?Q?Jerker_B=E4ck?=)
Date: Sat, 4 Aug 2007 16:54:33 +0200
Subject: [Python-Dev] x86_64 Interix - Advise needed on size of long
Message-ID: <000801c7d6a7$5ec0e1d0$1c42a570$@back@telia.com>

Hello all,
I'm in need of an advise how to handle sizeof long in python. I wanted a
x86_64 compile of python for Interix (that is NT POSIX subsystem with x86_64
Interix 6 SDK).

My first attempt to build failed due to the makefile insisted on linking as
shared libraries (works only in x86 with GNU ld). Tried autoreconf to get
rid of libtool - no luck.
Q1: Is the static build broken?
Q2: Anyone have a useable Makefile.am?

My second attempt was based on the VS2005 project and the previous Makefile.
Not to tire you with details, but for this to work I need to explicit assign
the sizeof long (replace all long types with explicit sized ones, int32_t,
ssize_t etc).

There are 2 choices: All longs to 64bit (LP64 model) or all to 32bit (LLP64
model). Since Interix use LP64 the first alternative would be logic, but
considering compatibility with the Windows DLL, performance(?) and whatever,
I choosed the latter. A choice which later would turn me into trouble.

Here's how I am reasoning:

x64 Windows DLL = LLP64 model => sizeof(long) = 4
x86_64 Interix  = LP64 model  => sizeof(long) = 8

So, since the Windows build works, basically all long types in the code are
32bit (or at least works if they are 32bit). 64bit dependent variables like
pointers have already been taken care of. Right? While it sounds reasonable
as long as one are consistent, it's actually quite difficult to get it right
(and a lot of work).

To be precise, would this be OK?
long PyInt_AsLong(PyObject *);
change to:
int32_t PyInt_AsLong(PyObject *);
or
unsigned long PyOS_strtoul(char*, char**, int);
to:
uint32_t PyOS_strtoul(char*, char**, int);

Thanks, 
Erik





From martin at v.loewis.de  Sat Aug  4 22:33:14 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Aug 2007 22:33:14 +0200
Subject: [Python-Dev] x86_64 Interix - Advise needed on size of long
In-Reply-To: <000801c7d6a7$5ec0e1d0$1c42a570$@back@telia.com>
References: <000801c7d6a7$5ec0e1d0$1c42a570$@back@telia.com>
Message-ID: <46B4E28A.6090201@v.loewis.de>

> My first attempt to build failed due to the makefile insisted on linking as
> shared libraries (works only in x86 with GNU ld). Tried autoreconf to get
> rid of libtool - no luck.
> Q1: Is the static build broken?
> Q2: Anyone have a useable Makefile.am?

Are you sure you are talking about Python as released? It uses neither
automake nor libtool (IMO, fortunately so).

As for the static vs. shared libpython: On Unix, Python is typically
built as a single executable (only linked shared with the system
libraries). The challenge is then with extension modules, which are
shared libraries. In particular, it is a challenge that those want
to find symbols defined in the executable, without being linked with
it. So you have three options:

1. If you use a sane binary format (such as ELF), symbol resolution
   considers symbols defined by the executable for use in shared
   libraries. This is necessary to support standard C, as you want
   to be able to redefined malloc(3) in the executable, and then all
   libraries should use your malloc implementation; it comes handy
   for Python's extensions. By this definition, Portable Executable (PE)
   is insane.
2. Don't use extension modules. Edit Modules/Setup to statically link
   all extension modules into the interpreter binary.
3. Arrange to make the interpreter a shared library (libpythonxy.so),
   then link all extension modules against it.

> There are 2 choices: All longs to 64bit (LP64 model) or all to 32bit (LLP64
> model). Since Interix use LP64 the first alternative would be logic, but
> considering compatibility with the Windows DLL, performance(?) and whatever,
> I choosed the latter. A choice which later would turn me into trouble.

I don't see the compatibility issue. You aren't going to use Win32
extensions in the Interix interpreter, are you? So why care about Win32?

> Here's how I am reasoning:
> 
> x64 Windows DLL = LLP64 model => sizeof(long) = 4
> x86_64 Interix  = LP64 model  => sizeof(long) = 8

I think we agree that the Windows model is insane, also. A good 64-bit
platform has sizeof(long)==8.

> So, since the Windows build works, basically all long types in the code are
> 32bit (or at least works if they are 32bit).

Right. However, LP64 also works with Python, and has been for many more
years.

> 64bit dependent variables like
> pointers have already been taken care of. Right? While it sounds reasonable
> as long as one are consistent, it's actually quite difficult to get it right
> (and a lot of work).
> 
> To be precise, would this be OK?
> long PyInt_AsLong(PyObject *);
> change to:
> int32_t PyInt_AsLong(PyObject *);
> or
> unsigned long PyOS_strtoul(char*, char**, int);
> to:
> uint32_t PyOS_strtoul(char*, char**, int);

OK in what sense? You making these changes locally? You can make
whatever changes you please; this is free software. I can't
see *why* you want to make all these changes, but if you so
desire...

This becoming part of Python? No way. It is intentional that
PyInt_AsLong returns long (why else would the function be called
this way?), and it is also intentional that the int type has
its internal representation as a long.

Likewise for strtoul: this function is defined to return long,
for whatever definition long has on a platform.

Regards,
Martin

From status at bugs.python.org  Sun Aug  5 02:01:00 2007
From: status at bugs.python.org (Tracker)
Date: Sun,  5 Aug 2007 00:01:00 +0000 (UTC)
Subject: [Python-Dev] Summary of Tracker Issues
Message-ID: <20070805000100.3E765781B4@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (07/29/07 - 08/05/07)
Tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 1276 open ( +0) / 11101 closed ( +0) / 12377 total ( +0)

Average duration of open issues: 695 days.
Median duration of open issues: 561 days.

Open Issues Breakdown
   open  1274 ( +0)
pending     2 ( +0)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070805/94ba0f84/attachment.html 

From jerker.back at telia.com  Sun Aug  5 14:41:09 2007
From: jerker.back at telia.com (=?iso-8859-1?Q?Jerker_B=E4ck?=)
Date: Sun, 5 Aug 2007 14:41:09 +0200
Subject: [Python-Dev] x86_64 Interix - Advise needed on size of long
In-Reply-To: <46B4E28A.6090201@v.loewis.de>
References: <000801c7d6a7$5ec0e1d0$1c42a570$@back@telia.com>
	<46B4E28A.6090201@v.loewis.de>
Message-ID: <000901c7d75d$e68d78b0$b3a86a10$@back@telia.com>

Hello Martin,
Thanks very much for answering.

> As for the static vs. shared libpython: On Unix, Python is typically
> built as a single executable (only linked shared with the system
> libraries). The challenge is then with extension modules, which are
> shared libraries. In particular, it is a challenge that those want
> to find symbols defined in the executable, without being linked with
> it. So you have three options:
Aha, now it lightens a bit. As I understand, I will need a x86_64 PE GNU ld
to get this to work as intended - however, there is no such thing at this
moment.
> 2. Don't use extension modules. Edit Modules/Setup to statically link
>    all extension modules into the interpreter binary.
This is the way. But how to do that?

Shell output:
../configure --disable-shared
...
ar cr libpython2.5.a Objects/<all objects>
ar cr libpython2.5.a Python/<all objects>
ar cr libpython2.5.a Modules/<some objects - main, config...>
ar cr libpython2.5.a Modules/<some more objects - threadmodule.o ...>
cc -o python \
    Modules/python.o \
    libpython2.5.a -lsocket -lm
CC='cc' LDSHARED='ld' OPT='-DNDEBUG -O' ./python -E ../setup.py build;;
Memory fault (core dumped)
make: *** [sharedmods] Error 139

I assume the "Modules/<some more objects>" are the extension modules. To get
them statically linked, the functions must be called somewhere. Statically
linked = "Builtin modules"? You mean I should list all of these in
"Modules/Setup"? FYI I got the "dynload_stub.c" compiled in. BTW, shouldn't
"--disable-shared" take care of this?

> OK in what sense? You making these changes locally? You can make
> whatever changes you please; this is free software. I can't
> see *why* you want to make all these changes, but if you so
> desire...
It's really very simple - I got LP64 libraries (Interix SDK). To get them to
work with a LLP64 compiler I need explicit sized types in case of long.

FYI: cc is a shell script wrapper of a x64 PE compiler, which in this case
is the MS x64 compiler v 14.00.50727.762 - POSIX mode. It automatically
translates all longs to long long in case of a 64bit compile. Thus, cc
cannot easily be used in e.g. Visual Studio.

> This becoming part of Python? No way. It is intentional that
> PyInt_AsLong returns long (why else would the function be called
> this way?), and it is also intentional that the int type has
> its internal representation as a long.
Oh, it was never my intention to propose a change to the LLP64 model. And
your right: All exports should be according to the LP64 model in case of a
POSIX compile. One must follow some rules! But you must admit it's tempting
with all these:
#if SIZEOF_LONG > 4
< get rid of the 64bit crap >
#endif
In my case the different paradigms are a real pain. I must take it into
account all the time when porting. I can only hope people stop using <long>
in favour of explicit sized types or types like size_t, intptr_t etc. I
would love to see the damn thing obsolete.

Cheers,
Erik


From martin at v.loewis.de  Sun Aug  5 15:33:55 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 05 Aug 2007 15:33:55 +0200
Subject: [Python-Dev] x86_64 Interix - Advise needed on size of long
In-Reply-To: <000901c7d75d$e68d78b0$b3a86a10$@back@telia.com>
References: <000801c7d6a7$5ec0e1d0$1c42a570$@back@telia.com>
	<46B4E28A.6090201@v.loewis.de>
	<000901c7d75d$e68d78b0$b3a86a10$@back@telia.com>
Message-ID: <46B5D1C3.2010304@v.loewis.de>

>> 2. Don't use extension modules. Edit Modules/Setup to statically link
>>    all extension modules into the interpreter binary.
> This is the way. But how to do that?
> 
> Shell output:
> ../configure --disable-shared

--disable-shared should be the default, so you don't need to specify
it explicitly. It works fine for me on Linux, so if it crashes for
you, it is likely an Interix problem. Can you debug this?

Dynamic loading of extension modules is automatically detected;
it is active if the system is AIX, BeOS, HPUX, Darwin, atheos,
or if dlopen(3) has been found. If you explicitly want to disable
it, you can set DYNLOADFILE to dynload_stub.o in configure.
Not sure whether Interix has dlopen, but even if does, it should
do no harm to use it provided there aren't any modules to load.

> I assume the "Modules/<some more objects>" are the extension modules. To get
> them statically linked, the functions must be called somewhere. Statically
> linked = "Builtin modules"? You mean I should list all of these in
> "Modules/Setup"?

Exactly so. They are already listed - just uncomment them all
(with proper command line flags and libraries where necessary).

> FYI I got the "dynload_stub.c" compiled in.

Ok, so you don't have dynamic loading.

> BTW, shouldn't "--disable-shared" take care of this?

No, this talks only about libpythonxy.so.

> It's really very simple - I got LP64 libraries (Interix SDK). To get them to
> work with a LLP64 compiler I need explicit sized types in case of long.

I still don't understand. Are you *certain* that these are LP64
libraries? Can you kindly refer to some official document that says
Interix uses LP64 on AMD64?

And if so, how did Microsoft manage to build them, if their compiler
does not support LP64? (I see you kind of answer that below - although
I'm unsure what "translate all longs to long long means - you mean
literal text replacement?)

Methinks you should just activate the LP64 mode of VC 2005, and be
done (and no, I don't know how to do that :-)

> FYI: cc is a shell script wrapper of a x64 PE compiler, which in this case
> is the MS x64 compiler v 14.00.50727.762 - POSIX mode. It automatically
> translates all longs to long long in case of a 64bit compile. Thus, cc
> cannot easily be used in e.g. Visual Studio.

So don't use Visual Studio, then. What's wrong with the Makefile?

> Oh, it was never my intention to propose a change to the LLP64 model. And
> your right: All exports should be according to the LP64 model in case of a
> POSIX compile. One must follow some rules! But you must admit it's tempting
> with all these:
> #if SIZEOF_LONG > 4
> < get rid of the 64bit crap >
> #endif

My view would be different - I find it tempting not to use Interix,
let alone on AMD64...

> In my case the different paradigms are a real pain. I must take it into
> account all the time when porting. I can only hope people stop using <long>
> in favour of explicit sized types or types like size_t, intptr_t etc. I
> would love to see the damn thing obsolete.

You mean the long type? I see nothing wrong with it. The real fault here
is with Microsoft, who managed to provide a system for which they don't
provide a C compiler, but just a hack that looks like a C compiler from
remote.

Regards,
Martin

From alan.mcintyre at gmail.com  Sun Aug  5 16:02:58 2007
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Sun, 5 Aug 2007 10:02:58 -0400
Subject: [Python-Dev] [Python-3000] test_asyncore fails intermittently
	on Darwin
In-Reply-To: <5d44f72f0708041853m1bb0d005h9f1ff77103b9ebbe@mail.gmail.com>
References: <2cda2fc90707261505tdd9a0f1t861b5801c37ad11e@mail.gmail.com>
	<1d36917a0707261618oac94f20l98f464a2ab1edc4e@mail.gmail.com>
	<2cda2fc90707292338pff060c1i810737dcf6d5df54@mail.gmail.com>
	<2cda2fc90707292340k7eb11f2w82003e6f705438c3@mail.gmail.com>
	<46AE943D.1040105@canterbury.ac.nz>
	<5d44f72f0708041853m1bb0d005h9f1ff77103b9ebbe@mail.gmail.com>
Message-ID: <1d36917a0708050702n6b48594bn824bd97ea6622421@mail.gmail.com>

On 8/4/07, Jeffrey Yasskin <jyasskin at gmail.com> wrote:
> Well, regardless of the brokenness of the patch, I do get two
> different failures from this test on OSX. The first is caused by
> trying to socket.bind() a port that's already been bound recently:
<snip>
> That looks pretty easy to fix.

It was fixed in the trunk on July 28 as part of rev 56604, by letting
the OS assign the port (binding to port 0). I apologize if everybody
was expecting me to fix this in Python 3000; I thought the initial
complaint was in reference to 2.6. I'm working on test improvements
for 2.6, so I'm sort of fixated on the trunk at the moment. :)  I
wouldn't mind trying to roll my changes forward into Py3k after GSoC
is done if I have the time, though.

Alan

From g.brandl at gmx.net  Sun Aug  5 16:51:25 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 05 Aug 2007 16:51:25 +0200
Subject: [Python-Dev] cStringIO.StringIO() buffer behavior
Message-ID: <f94o54$i2o$1@sea.gmane.org>

See bugs #1548891 and #1730114.

In the former, it was reported that cStringIO works differently from StringIO
when handling unicode strings; it used GetReadBuffer which returned the raw
internal UCS-2 or UCS-4 encoded string.

I changed it to use GetCharBuffer, which converts to a string using the
default encoding first. This fix was also in 2.5.1.

The latter bug now complains that this excludes things like array.array()s
from being used as an argument to cStringIO.StringIO(), which worked before
with GetReadBuffer.

What's the preferred solution here?

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From guido at python.org  Sun Aug  5 17:00:27 2007
From: guido at python.org (Guido van Rossum)
Date: Sun, 5 Aug 2007 08:00:27 -0700
Subject: [Python-Dev] cStringIO.StringIO() buffer behavior
In-Reply-To: <f94o54$i2o$1@sea.gmane.org>
References: <f94o54$i2o$1@sea.gmane.org>
Message-ID: <ca471dc20708050800n6b19de58qc34410bf7bf0dca0@mail.gmail.com>

Methinks that this was a fundamental limitation of cStringIO, not  a
bug. Certainly not something to be "fixed" in a bugfix release.

On 8/5/07, Georg Brandl <g.brandl at gmx.net> wrote:
> See bugs #1548891 and #1730114.
>
> In the former, it was reported that cStringIO works differently from StringIO
> when handling unicode strings; it used GetReadBuffer which returned the raw
> internal UCS-2 or UCS-4 encoded string.
>
> I changed it to use GetCharBuffer, which converts to a string using the
> default encoding first. This fix was also in 2.5.1.
>
> The latter bug now complains that this excludes things like array.array()s
> from being used as an argument to cStringIO.StringIO(), which worked before
> with GetReadBuffer.
>
> What's the preferred solution here?
>
> Georg
>
> --
> Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
> Four shall be the number of spaces thou shalt indent, and the number of thy
> indenting shall be four. Eight shalt thou not indent, nor either indent thou
> two, excepting that thou then proceed to four. Tabs are right out.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From jerker.back at telia.com  Sun Aug  5 17:45:52 2007
From: jerker.back at telia.com (=?iso-8859-1?Q?Jerker_B=E4ck?=)
Date: Sun, 5 Aug 2007 17:45:52 +0200
Subject: [Python-Dev] x86_64 Interix - Advise needed on size of long
Message-ID: <000b01c7d777$b42fcca0$1c8f65e0$@back@telia.com>

Hello Martin,

> > You mean I should list all of these in "Modules/Setup"?
> Exactly so. They are already listed - just uncomment them all
> (with proper command line flags and libraries where necessary).
OK, I will try to get it compiled and tested.

Meanwhile, you asked so:
> I still don't understand. Are you *certain* that these are LP64
> libraries? Can you kindly refer to some official document that says
> Interix uses LP64 on AMD64?
MS is surprisingly very quiet of the POSIX subsystem and the Interix BSD
implementation, so it's hard to find any official info on the net. But here
is one the developers:
<http://blogs.msdn.com/shan/archive/2006/12/12/hi-there.aspx>
In Interix SDK releasenotes.htm (SDK download):
"64-bit compilation supports the LP64 data model."
Interix general:
<http://www4.osnews.com/story/5751/Review_of_Windows_Services_for_UNIX_3.5>
<http://www.interopcommunity.com>

To find details on how it all really works, one will have to look in the
headers and try different features oneself. (Which actually is pretty fun
because it's really fast and usually works well). The SDK comes with support
for x86, x86_64 (EM64T or AMD64) and IA64.

> And if so, how did Microsoft manage to build them, if their compiler
> does not support LP64? (I see you kind of answer that below - although
> I'm unsure what "translate all longs to long long means - you mean
> literal text replacement?)
Sure, cc precompiles the source file to a temporary file, flip it, runs a
conversion tool - "l2ll" => all <long> are converted to <long long> and
finally compiles the converted file. The compile is done via a call from
POSIX to the Windows subsystem and the compiler found in the POSIX path
environment. To understand the details one has to know that the POSIX
environment runs directly on top of the NT kernel and know nothing of
Windows, Windows paths etc. This is kind of a compile on the fly. The
libraries are also of two kinds: 
1 The core POSIX libraries - part of the OS, uses DDK tools
2 Interix SDK - BSD libc and utils, uses cc and Interix gcc (x86 only)

The DDK tools is turned to LP64 support via special defines in the headers.
But here is some unclear issues with functions directly exported from the OS
native LLP64 libraries (ntdll.dll) - don't know how this is solved.
 
Somewhere here lies the reason why cc is hard to use with Visual Studio and
why the long type is such a nuisance.

I also tried the Intel x64 PE compiler (for better C99 support), but it
produces applications which relies on Windows API functions (e.g.
VirtualAlloc, LoadLibrary) and thus cannot be used in POSIX.
 
Cheers,
Erik


From martin at v.loewis.de  Sun Aug  5 18:37:48 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 05 Aug 2007 18:37:48 +0200
Subject: [Python-Dev] cStringIO.StringIO() buffer behavior
In-Reply-To: <f94o54$i2o$1@sea.gmane.org>
References: <f94o54$i2o$1@sea.gmane.org>
Message-ID: <46B5FCDC.6050807@v.loewis.de>

> See bugs #1548891 and #1730114.
> 
> In the former, it was reported that cStringIO works differently from StringIO
> when handling unicode strings; it used GetReadBuffer which returned the raw
> internal UCS-2 or UCS-4 encoded string.
> 
> I changed it to use GetCharBuffer, which converts to a string using the
> default encoding first. This fix was also in 2.5.1.
> 
> The latter bug now complains that this excludes things like array.array()s
> from being used as an argument to cStringIO.StringIO(), which worked before
> with GetReadBuffer.
> 
> What's the preferred solution here?

I think the 2.5.0 behavior to accept array.array should be restored (and
a test case be added). What to do about Unicode strings, I don't know.
I agree with Guido that they are officially not supported in cStringIO,
so it would be best to reject them. OTOH, since 2.5.1 already supports
them, another choice would be continue supporting them, in the same way
as they are supported in 2.5.1. Either solution would special-case
Unicode strings.

Regards,
Martin


From alexandre at peadrop.com  Sun Aug  5 20:48:11 2007
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Sun, 5 Aug 2007 14:48:11 -0400
Subject: [Python-Dev] cStringIO.StringIO() buffer behavior
In-Reply-To: <f94o54$i2o$1@sea.gmane.org>
References: <f94o54$i2o$1@sea.gmane.org>
Message-ID: <acd65fa20708051148o597e12fsf41ad612de0c125f@mail.gmail.com>

On 8/5/07, Georg Brandl <g.brandl at gmx.net> wrote:
> See bugs #1548891 and #1730114.
>
> In the former, it was reported that cStringIO works differently from StringIO
> when handling unicode strings; it used GetReadBuffer which returned the raw
> internal UCS-2 or UCS-4 encoded string.
>
> I changed it to use GetCharBuffer, which converts to a string using the
> default encoding first. This fix was also in 2.5.1.
>
> The latter bug now complains that this excludes things like array.array()s
> from being used as an argument to cStringIO.StringIO(), which worked before
> with GetReadBuffer.
>
> What's the preferred solution here?
>

The best thing would be add a special case for ascii-only unicode
objects, and keep the old behavior. However, I believe this will be
ugly, especially in O_write. So, it would perhaps be better to simply
stop supporting unicode objects.

-- Alexandre

From g.brandl at gmx.net  Mon Aug  6 09:48:25 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 06 Aug 2007 09:48:25 +0200
Subject: [Python-Dev] cStringIO.StringIO() buffer behavior
In-Reply-To: <46B5FCDC.6050807@v.loewis.de>
References: <f94o54$i2o$1@sea.gmane.org> <46B5FCDC.6050807@v.loewis.de>
Message-ID: <f96jo0$vqb$1@sea.gmane.org>

Guido van Rossum schrieb:
> Methinks that this was a fundamental limitation of cStringIO, not  a
> bug. Certainly not something to be "fixed" in a bugfix release.

I'm sorry.

Martin v. L?wis schrieb:
>> See bugs #1548891 and #1730114.
>> 
>> In the former, it was reported that cStringIO works differently from StringIO
>> when handling unicode strings; it used GetReadBuffer which returned the raw
>> internal UCS-2 or UCS-4 encoded string.
>> 
>> I changed it to use GetCharBuffer, which converts to a string using the
>> default encoding first. This fix was also in 2.5.1.
>> 
>> The latter bug now complains that this excludes things like array.array()s
>> from being used as an argument to cStringIO.StringIO(), which worked before
>> with GetReadBuffer.
>> 
>> What's the preferred solution here?
> 
> I think the 2.5.0 behavior to accept array.array should be restored (and
> a test case be added). What to do about Unicode strings, I don't know.
> I agree with Guido that they are officially not supported in cStringIO,
> so it would be best to reject them. OTOH, since 2.5.1 already supports
> them, another choice would be continue supporting them, in the same way
> as they are supported in 2.5.1. Either solution would special-case
> Unicode strings.

Okay, I propose the following patch:

Index: Modules/cStringIO.c
===================================================================
--- Modules/cStringIO.c (Revision 56763)
+++ Modules/cStringIO.c (Arbeitskopie)
@@ -673,12 +673,26 @@
   char *buf;
   Py_ssize_t size;

-  if (PyObject_AsCharBuffer(s, (const char **)&buf, &size) != 0)
+  /* special-case Unicode objects: encode them in the default encoding */
+  if (PyUnicode_Check(s)) {
+    s = PyUnicode_AsEncodedString(s, NULL, NULL);
+    if (s == NULL)
       return NULL;
+  } else {
+    Py_INCREF(s);
+  }

+  if (PyObject_AsReadBuffer(s, (const char **)&buf, &size)) {
+    PyErr_Format(PyExc_TypeError, "expected read buffer, %.200s found",
+                 s->ob_type->tp_name);
+    return NULL;
+  }
+
   self = PyObject_New(Iobject, &Itype);
-  if (!self) return NULL;
-  Py_INCREF(s);
+  if (!self) {
+    Py_DECREF(s);
+    return NULL;
+  }
   self->buf=buf;
   self->string_size=size;
   self->pbuf=s;


Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From guido at python.org  Mon Aug  6 20:22:31 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 6 Aug 2007 11:22:31 -0700
Subject: [Python-Dev] cStringIO.StringIO() buffer behavior
In-Reply-To: <f96jo0$vqb$1@sea.gmane.org>
References: <f94o54$i2o$1@sea.gmane.org> <46B5FCDC.6050807@v.loewis.de>
	<f96jo0$vqb$1@sea.gmane.org>
Message-ID: <ca471dc20708061122k535e4323sdf231f1045348a90@mail.gmail.com>

On 8/6/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Guido van Rossum schrieb:
> > Methinks that this was a fundamental limitation of cStringIO, not  a
> > bug. Certainly not something to be "fixed" in a bugfix release.
>
> I'm sorry.

No problem. Somebody else should have flagged this, so it's our
collective responsibility.

> Okay, I propose the following patch:
>
> Index: Modules/cStringIO.c
[...]

My proposal is much more radical -- get rid of cStringIO altogether.
(And also of StringIO.py.) There aren't that many places using it any
more, and almost all of these are easily replaced with io.StringIO (or
io.BytesIO!). There's already a fixer in 2to3 to do this.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Mon Aug  6 20:24:15 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 6 Aug 2007 11:24:15 -0700
Subject: [Python-Dev] cStringIO.StringIO() buffer behavior
In-Reply-To: <ca471dc20708061122k535e4323sdf231f1045348a90@mail.gmail.com>
References: <f94o54$i2o$1@sea.gmane.org> <46B5FCDC.6050807@v.loewis.de>
	<f96jo0$vqb$1@sea.gmane.org>
	<ca471dc20708061122k535e4323sdf231f1045348a90@mail.gmail.com>
Message-ID: <ca471dc20708061124v6c76cfcak27fbd666c3c744d4@mail.gmail.com>

Oops, never mind. This was in the context of 2.5 and 2.6, but my reply
was in the context of 3.0. Still, in the light of cStringIO
disappearing, it would be good to keep cStringIO is stable as possible
(probably restoring 2.5.0 behavior) so as to avoid breaking 3rd party
code more than once.

On 8/6/07, Guido van Rossum <guido at python.org> wrote:
> On 8/6/07, Georg Brandl <g.brandl at gmx.net> wrote:
> > Guido van Rossum schrieb:
> > > Methinks that this was a fundamental limitation of cStringIO, not  a
> > > bug. Certainly not something to be "fixed" in a bugfix release.
> >
> > I'm sorry.
>
> No problem. Somebody else should have flagged this, so it's our
> collective responsibility.
>
> > Okay, I propose the following patch:
> >
> > Index: Modules/cStringIO.c
> [...]
>
> My proposal is much more radical -- get rid of cStringIO altogether.
> (And also of StringIO.py.) There aren't that many places using it any
> more, and almost all of these are easily replaced with io.StringIO (or
> io.BytesIO!). There's already a fixer in 2to3 to do this.
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From alexandre at peadrop.com  Mon Aug  6 21:43:23 2007
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Mon, 6 Aug 2007 15:43:23 -0400
Subject: [Python-Dev] cStringIO.StringIO() buffer behavior
In-Reply-To: <f96jo0$vqb$1@sea.gmane.org>
References: <f94o54$i2o$1@sea.gmane.org> <46B5FCDC.6050807@v.loewis.de>
	<f96jo0$vqb$1@sea.gmane.org>
Message-ID: <acd65fa20708061243g10ab3e64v9ca861dbf1a76256@mail.gmail.com>

On 8/6/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Okay, I propose the following patch:
> [...]

I think your patch is complicated for nothing. It would be much more
straightforward to use PyString_AsStringAndSize to encode the Unicode
string with the default encoding. I think it would be necessary to
port the fix to O_write and O_writelines.

-- Alexandre

Index: Modules/cStringIO.c
===================================================================
--- Modules/cStringIO.c (revision 56754)
+++ Modules/cStringIO.c (working copy)
@@ -665,8 +674,15 @@
   char *buf;
   Py_ssize_t size;

-  if (PyObject_AsCharBuffer(s, (const char **)&buf, &size) != 0)
-      return NULL;
+  /* Special case for unicode objects. */
+  if (PyUnicode_Check(s)) {
+         if (PyString_AsStringAndSize(s, &buf, &size) == -1)
+                 return NULL;
+  }
+  else {
+         if (PyObject_AsReadBuffer(s, (const void **)&buf, &size) == -1)
+                 return NULL;
+  }

   self = PyObject_New(Iobject, &Itype);
   if (!self) return NULL;

From guido at python.org  Tue Aug  7 01:55:18 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 6 Aug 2007 16:55:18 -0700
Subject: [Python-Dev] Pleaswe help with the countdown to zero failing tests
	in the struni branch!
Message-ID: <ca471dc20708061655w62420da3m1d87ffa5eff669c6@mail.gmail.com>

We're down to 11 failing test in the struni branch. I'd like to get
this down to zero ASAP so that we can retire the old p3yk (yes, with
typo!) branch and rename py3k-struni to py3k.

Please help! Here's the list of failing tests:

test_ctypes
Recently one test started failing again, after Martin changed
PyUnicode_FromStringAndSize() to use UTF8 instead of Latin1.

test_email
test_email_codecs
test_email_renamed
Can someone contact the email-sig and ask for help with these?

test_minidom
Recently started failing again; probably shallow.

test_sqlite
Virgin territory, probably best done by whoever wrote the code or at
least someone with time to spare.

test_tarfile
Virgin territory again (but different owner :-).

test_urllib2_localnet
test_urllib2net
I think Jeremy Hylton may be close to fixing these, he's done a lot of
work on urllib and httplib.

test_xml_etree_c
Virgin territory again.

There are also a few tests that only fail on CYGWIN or OSX; I won't
bother listing these.

If you want to help, please refer to this wiki page:
http://wiki.python.org/moin/Py3kStrUniTests

There are also other tasks; see http://wiki.python.org/moin/Py3kToDo

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From fijall at gmail.com  Tue Aug  7 23:23:44 2007
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 7 Aug 2007 23:23:44 +0200
Subject: [Python-Dev] os.tmpfile() problem
Message-ID: <693bc9ab0708071423t4027f99cs8aef222942b42e59@mail.gmail.com>

I've got slight problem with os.tmpfile(). What I would like to do is to get
the filedesc of tmpfile.

First approach:

os.tmpfile().fileno() of course does not work out, because fileno() does not
keep object alive. The solution is to keep os.tmpfile() result somewhere for
an arbitrary amount of time, which is quite obscure. This is problem with
all file operations, but fortunately if I want a filedesc, I can do just
os.open() which will not close the file for me.

I've got several obscure solutions, noone satisfies me really:

* If I use .fileno() than I'm on my own and I need to close file myself

* .fileno() returns a int-like object which keeps alive file (well, this
will explode when keeping this as an index in a list, which does not keep
the object alive and so on)

* have os._tmpfile() or whatever which returns filedesc

What do you think?

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070807/eeaa0959/attachment.htm 

From guido at python.org  Tue Aug  7 23:28:23 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 7 Aug 2007 14:28:23 -0700
Subject: [Python-Dev] os.tmpfile() problem
In-Reply-To: <693bc9ab0708071423t4027f99cs8aef222942b42e59@mail.gmail.com>
References: <693bc9ab0708071423t4027f99cs8aef222942b42e59@mail.gmail.com>
Message-ID: <ca471dc20708071428i2e75b84dv825512f5da070dbc@mail.gmail.com>

This seems a question for comp.lang.python or help at python.org (does
that still exist?).

Also, you might consider the APIs available in the tempfile module
rather than os.tempfile().

On 8/7/07, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I've got slight problem with os.tmpfile(). What I would like to do is to get
> the filedesc of tmpfile.
>
> First approach:
>
> os.tmpfile().fileno() of course does not work out, because fileno() does not
> keep object alive. The solution is to keep os.tmpfile() result somewhere for
> an arbitrary amount of time, which is quite obscure. This is problem with
> all file operations, but fortunately if I want a filedesc, I can do just
> os.open() which will not close the file for me.
>
> I've got several obscure solutions, noone satisfies me really:
>
> * If I use .fileno() than I'm on my own and I need to close file myself
>
> * .fileno() returns a int-like object which keeps alive file (well, this
> will explode when keeping this as an index in a list, which does not keep
> the object alive and so on)
>
> * have os._tmpfile() or whatever which returns filedesc
>
> What do you think?
>
> Cheers,
> fijal
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Wed Aug  8 00:41:40 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 7 Aug 2007 15:41:40 -0700
Subject: [Python-Dev] Pleaswe help with the countdown to zero failing
	tests in the struni branch!
In-Reply-To: <ca471dc20708061655w62420da3m1d87ffa5eff669c6@mail.gmail.com>
References: <ca471dc20708061655w62420da3m1d87ffa5eff669c6@mail.gmail.com>
Message-ID: <ca471dc20708071541s68ad50c5sb8b5c1becc9fca9d@mail.gmail.com>

Here's a followup.

We need help from someone with a 64-bit Linux box; these tests are
failing on 64-bit only: test_io, test_largefile, test_ossaudiodev,
test_poll, test_shelve, test_socket_ssl.

I suspect that the _fileio.c module probably is one of the culprits.

Other news:

On 8/6/07, Guido van Rossum <guido at python.org> wrote:
> We're down to 11 failing test in the struni branch. I'd like to get
> this down to zero ASAP so that we can retire the old p3yk (yes, with
> typo!) branch and rename py3k-struni to py3k.
>
> Please help! Here's the list of failing tests:
>
> test_ctypes
> Recently one test started failing again, after Martin changed
> PyUnicode_FromStringAndSize() to use UTF8 instead of Latin1.
>
> test_email
> test_email_codecs
> test_email_renamed
> Can someone contact the email-sig and ask for help with these?
>
> test_minidom
> Recently started failing again; probably shallow.
>
> test_sqlite
> Virgin territory, probably best done by whoever wrote the code or at
> least someone with time to spare.
>
> test_tarfile
> Virgin territory again (but different owner :-).

Lars Gustaebel fixed this except for a few bz2-related tests.

> test_urllib2_localnet
> test_urllib2net
> I think Jeremy Hylton may be close to fixing these, he's done a lot of
> work on urllib and httplib.
>
> test_xml_etree_c
> Virgin territory again.
>
> There are also a few tests that only fail on CYGWIN or OSX; I won't
> bother listing these.

The two OSX tests listed at the time were fixed, thanks to those volunteers!

We now only have an OSX-specific failure in test_csv.

> If you want to help, please refer to this wiki page:
> http://wiki.python.org/moin/Py3kStrUniTests
>
> There are also other tasks; see http://wiki.python.org/moin/Py3kToDo

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From nmm1 at cus.cam.ac.uk  Wed Aug  8 11:28:16 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Wed, 08 Aug 2007 10:28:16 +0100
Subject: [Python-Dev] Regular expressions, Unicode etc.
Message-ID: <E1IIhpw-0001jT-2W@virgo.cus.cam.ac.uk>

I have needed to push my stack to teach REs (don't ask), and am
taking a look at the RE code.  I may be able to extend it to support
RFE 694374 and (more importantly) atomic groups and possessive
quantifiers.  While I regard such things as revolting beyond belief,
they make a HELL of a difference to the efficiency of recognising
things like HTML tags in a morass of mixed text.

The other approach, which is to stick to true regular expressions,
and wholly or partially convert to DFAs, has already been rendered
impossible by even the limited Perl/PCRE extensions that Python
has adopted.

My first question is whether this would clash with any ongoing
work, including being superseded by any changes in Python 3000.

Note that I am NOT proposing to do a fixed task, but will produce
a proper proposal only when I know what I can achieve for a small
amount of work.  If the SRE engine turns out to be unsuitable to
extend in these ways, I shall quietly abandon the project.



My second one is about Unicode.  I really, but REALLY regard it as
a serious defect that there is no escape for printing characters.
Any code that checks arbitrary text is likely to need them - yes,
I know why Perl and hence PCRE doesn't have that, but let's skip
that.  That is easy to add, though choosing a letter is tricky.
Currently \c and \C, for 'character' (I would prefer 'text' or
'printable', but \t is obviously insane and \P is asking for
incompatibility with Perl and Java).

But attempting to rebuild the Unicode database hasn't worked.
Tools/unicode is, er, a trifle incomplete and out of date.  The
only file I need to change is Objects/unicodetype_db.h, but the
init attempts to run Tools/unicode/makeunicodedata.py have not
been successful.

I may be able to reverse engineer the mechanism enough to get
the files off the Unicode site and run it, but I don't want to
spend forever on it.  Any clues?


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Wed Aug  8 13:15:40 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Wed, 08 Aug 2007 12:15:40 +0100
Subject: [Python-Dev] Regular expressions, Unicode etc.
Message-ID: <E1IIjVs-0002o2-5c@virgo.cus.cam.ac.uk>

Further to the above, I found the Unicode sources, have rebuilt
the files, but it involved some fairly serious hacking to the
building mechanism and I have had to disable the Unicode 3.2
support.  And, of course, that means that 4 of the tests fail.

This area needs addressing, not least because Python should
clearly be upgraded to Unicode 5.0.0 (which is what I am using)
at some stage.

I am not sure how best to report a bug that essentially says
"The build mechanisms for Unicode have suffered bit-rot, no longer
work and need redesigning."  I could certainly do that, but it's
not helpful - people already know that, from the comments :-(


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From g.brandl at gmx.net  Wed Aug  8 14:52:47 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 08 Aug 2007 14:52:47 +0200
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IIjVs-0002o2-5c@virgo.cus.cam.ac.uk>
References: <E1IIjVs-0002o2-5c@virgo.cus.cam.ac.uk>
Message-ID: <f9ceam$j8u$1@sea.gmane.org>

Nick Maclaren schrieb:
> Further to the above, I found the Unicode sources, have rebuilt
> the files, but it involved some fairly serious hacking to the
> building mechanism and I have had to disable the Unicode 3.2
> support.  And, of course, that means that 4 of the tests fail.
> 
> This area needs addressing, not least because Python should
> clearly be upgraded to Unicode 5.0.0 (which is what I am using)
> at some stage.
> 
> I am not sure how best to report a bug that essentially says
> "The build mechanisms for Unicode have suffered bit-rot, no longer
> work and need redesigning."  I could certainly do that, but it's
> not helpful - people already know that, from the comments :-(

FWIW, there is a patch on the tracker at python.org/sf/1571184 that may be
helpful to you.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From martin at v.loewis.de  Wed Aug  8 20:41:33 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 08 Aug 2007 20:41:33 +0200
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IIhpw-0001jT-2W@virgo.cus.cam.ac.uk>
References: <E1IIhpw-0001jT-2W@virgo.cus.cam.ac.uk>
Message-ID: <46BA0E5D.60109@v.loewis.de>

> My second one is about Unicode.  I really, but REALLY regard it as
> a serious defect that there is no escape for printing characters.
> Any code that checks arbitrary text is likely to need them - yes,
> I know why Perl and hence PCRE doesn't have that, but let's skip
> that.  That is easy to add, though choosing a letter is tricky.
> Currently \c and \C, for 'character' (I would prefer 'text' or
> 'printable', but \t is obviously insane and \P is asking for
> incompatibility with Perl and Java).

Before discussing the escape, I'd like to see a specification of
it first - what characters precisely would classify as "printing"?

> But attempting to rebuild the Unicode database hasn't worked.
> Tools/unicode is, er, a trifle incomplete and out of date.  The
> only file I need to change is Objects/unicodetype_db.h, but the
> init attempts to run Tools/unicode/makeunicodedata.py have not
> been successful.
> 
> I may be able to reverse engineer the mechanism enough to get
> the files off the Unicode site and run it, but I don't want to
> spend forever on it.  Any clues?

I see that you managed to do something here, so I'm not sure
what kind of help you still need.

Regards,
Martin

From martin at v.loewis.de  Wed Aug  8 20:48:46 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 08 Aug 2007 20:48:46 +0200
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IIjVs-0002o2-5c@virgo.cus.cam.ac.uk>
References: <E1IIjVs-0002o2-5c@virgo.cus.cam.ac.uk>
Message-ID: <46BA100E.4020006@v.loewis.de>

> Further to the above, I found the Unicode sources, have rebuilt
> the files, but it involved some fairly serious hacking to the
> building mechanism and I have had to disable the Unicode 3.2
> support.  And, of course, that means that 4 of the tests fail.
> 
> This area needs addressing, not least because Python should
> clearly be upgraded to Unicode 5.0.0 (which is what I am using)
> at some stage.

I recommend you use the 4.1 version of the database; this should
work out of the box, with no change to the build environment at
all.

As for updating it - that has to wait until the next release
of Python. At that point, 5.1 might be releasesd, so 5.0 might
get skipped altogether.

> I am not sure how best to report a bug that essentially says
> "The build mechanisms for Unicode have suffered bit-rot, no longer
> work and need redesigning."  I could certainly do that, but it's
> not helpful - people already know that, from the comments :-(

I would likely close such a report as "works for me" (after testing
it does - it did when I last ran it, which was before the release
of Python 2.5).
It did not suffer from bit-rot - it still works just fine for
the version of the database that is supported.

As for the need for redesigning - I don't see that need. What specific
aspect do you think needs redesigning? If you merely meant to say
"I don't understand the code" - this is not enough reason, I
remember it took me some time to understand it as well, but now
I see that it does precisely what it needs to do, and precisely
in the way it needs to do that.

Regards,
Martin

From mike.klaas at gmail.com  Wed Aug  8 20:56:58 2007
From: mike.klaas at gmail.com (Mike Klaas)
Date: Wed, 8 Aug 2007 11:56:58 -0700
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IIhpw-0001jT-2W@virgo.cus.cam.ac.uk>
References: <E1IIhpw-0001jT-2W@virgo.cus.cam.ac.uk>
Message-ID: <9CBC9283-52BF-48AB-A39F-0DAE0E4EAFAE@gmail.com>

On 8-Aug-07, at 2:28 AM, Nick Maclaren wrote:

> I have needed to push my stack to teach REs (don't ask), and am
> taking a look at the RE code.  I may be able to extend it to support
> RFE 694374 and (more importantly) atomic groups and possessive
> quantifiers.  While I regard such things as revolting beyond belief,
> they make a HELL of a difference to the efficiency of recognising
> things like HTML tags in a morass of mixed text.

+1.  I would use such a feature.

> The other approach, which is to stick to true regular expressions,
> and wholly or partially convert to DFAs, has already been rendered
> impossible by even the limited Perl/PCRE extensions that Python
> has adopted.

Impossible?  Surely, a sufficiently-competent re engine could detect  
when a DFA is possible to construct?

-Mike

From nmm1 at cus.cam.ac.uk  Wed Aug  8 21:29:50 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Wed, 08 Aug 2007 20:29:50 +0100
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: Your message of "Wed, 08 Aug 2007 20:41:33 +0200."
	<46BA0E5D.60109@v.loewis.de> 
Message-ID: <E1IIrE7-0004pR-1S@libra.cus.cam.ac.uk>

[ I would appreciate not getting private copies as well. ]

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
>
> Before discussing the escape, I'd like to see a specification of
> it first - what characters precisely would classify as "printing"?

For basic ASCII and locale-based testing, whatever isprint() says.
Just as for isalpha().

For Unicode, whatever people agree!  I use the criterion that it
has a defined category that doesn't start with 'C' - which is what
I think that most people will accept.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Wed Aug  8 21:31:49 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Wed, 08 Aug 2007 20:31:49 +0100
Subject: [Python-Dev] =?iso-8859-1?q?cc=3A_=22Martin_v=2E_L=F6wis=22_=3Cma?=
	=?iso-8859-1?q?rtin=40v=2Eloewis=2Ede=3E?=
Message-ID: <E1IIrG1-0004q6-1w@libra.cus.cam.ac.uk>

Re: [Python-Dev] Regular expressions, Unicode etc.
=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
>
> I recommend you use the 4.1 version of the database; this should
> work out of the box, with no change to the build environment at
> all.

I tried that, of course.  See below.

> As for updating it - that has to wait until the next release
> of Python. At that point, 5.1 might be releasesd, so 5.0 might
> get skipped altogether.

Very true.

> I would likely close such a report as "works for me" (after testing
> it does - it did when I last ran it, which was before the release
> of Python 2.5).

I think that you will find that you are using a non-standard
environment and set of Python sources.  I started off with the
standard distribution.

> It did not suffer from bit-rot - it still works just fine for
> the version of the database that is supported.

Really?  I have just checked 2.5.1, and the same defects are there.

> As for the need for redesigning - I don't see that need. What specific
> aspect do you think needs redesigning? If you merely meant to say
> "I don't understand the code" - this is not enough reason, I
> remember it took me some time to understand it as well, but now
> I see that it does precisely what it needs to do, and precisely
> in the way it needs to do that.

Well, here are a selection of the issues that I found:

The Makefile includes the command:
    ncftpget -R ftp.unicode.org . Public/MAPPINGS
Not merely is ncftpget not a standard utility, the current mappings
are no longer at that location.  Indeed, I can see nothing useful in
that directory at present, though I haven't searched it in depth!

Looking through www.unicode.org, I could find the relevant files
for 5.0.0, but for no other version.  No, I am NOT going to type
in over a megabyte of data from the PDF!

makeunicodedata.py has a reference to the Unicode 3.2 files, but
they are not present in the standard distribution, the Makefile
doesn't fetch them, and I can't find them.

makeunicodedata.py refers to (for example) UnicodeData.txt and
Modules/unicodedata_db.h as such, which rather requires it to be
run in a particular directory.  I can find nothing in any file
even referring to this.

Having run it, running 'make all' does not rebuild Python correctly.
I couldn't be bothered to work out why, so I hit it with the usual
trick, 'make distclean'.

And, of course, it SHOULD be possible to upgrade the Unicode data
without having to change version of Python!


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Wed Aug  8 21:47:11 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Wed, 08 Aug 2007 20:47:11 +0100
Subject: [Python-Dev] Regular expressions, Unicode etc.
Message-ID: <E1IIrUt-0004wR-Km@libra.cus.cam.ac.uk>

I am not on "Python 3000", so am restricting.

Mike Klaas <mike.klaas at gmail.com> wrote:
>
> > I have needed to push my stack to teach REs (don't ask), and am
> > taking a look at the RE code.  I may be able to extend it to support
> > RFE 694374 and (more importantly) atomic groups and possessive
> > quantifiers.  While I regard such things as revolting beyond belief,
> > they make a HELL of a difference to the efficiency of recognising
> > things like HTML tags in a morass of mixed text.
> 
> +1.  I would use such a feature.

I think that I am getting somewhere, but I really dislike the style
of _sre.c.  It has a very complex semi-stack, semi-finite-state
design and no comments on how it is supposed to work.

And its memory management looks like a recipe for leaks, so I may
well introduce some of them.

> > The other approach, which is to stick to true regular expressions,
> > and wholly or partially convert to DFAs, has already been rendered
> > impossible by even the limited Perl/PCRE extensions that Python
> > has adopted.
> 
> Impossible?  Surely, a sufficiently-competent re engine could detect  
> when a DFA is possible to construct?

I doubt it.  While it isn't equivalent to the halting problem, it IS
an intractable one!  There are two problems:

Firstly, things like backreferences are an absolute no-no.  They
are not regular, and REs with them in cannot be converted to DFAs.
That could be 'solved' by a parser that kicked out such constructions,
but it would get screams from many users.

Secondly, anything involving explicit or implicit negation can lead
to (if I recall) a super-exponential explosion in the size of the
DFA.  That could be 'solved' by imposing a limit, but few people
would be able to predict when it would bite.

Thirdly, I would require notice of the question of whether capturing
parentheses could be supported, and what semantic changes would be
to which were set and how.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From brett at python.org  Wed Aug  8 22:28:12 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 8 Aug 2007 13:28:12 -0700
Subject: [Python-Dev] Please help verify SF data dump imported into (future)
	new tracker
Message-ID: <bbaeab100708081328v70d0fc28v1c69e74e3895bd68@mail.gmail.com>

We are getting very close to moving over to the new tracker (hopefully
by the end of the month; no firm date yet, though, as we are still
planning things out)!

Part of the transition is taking a data dump provided by SourceForge
and loading it into our Roundup instance.  But we need to make some
effort to make sure SF's data dump is accurate and that our import is
good.

If you can, please go to SourceForge and choose some issue (bug,
patch, whatever), and then look up the corresponding issue at
http://bugs.python.org/ .  If there is any discrepancy, please report
it at http://psf.upfronthosting.co.za/roundup/meta (the link is also
listed at the new tracker as where to report tracker problems) or to
this email.

-Brett

P.S.: If you want to help with the transitionin other ways, you can
also help with the tracker docs at
http://wiki.python.org/moin/TrackerDocs.

From mike.klaas at gmail.com  Wed Aug  8 22:29:50 2007
From: mike.klaas at gmail.com (Mike Klaas)
Date: Wed, 8 Aug 2007 13:29:50 -0700
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IIrUt-0004wR-Km@libra.cus.cam.ac.uk>
References: <E1IIrUt-0004wR-Km@libra.cus.cam.ac.uk>
Message-ID: <04F52551-D458-490C-9945-6DAD1E46F7D1@gmail.com>

In 8-Aug-07, at 12:47 PM, Nick Maclaren wrote:

>
>>> The other approach, which is to stick to true regular expressions,
>>> and wholly or partially convert to DFAs, has already been rendered
>>> impossible by even the limited Perl/PCRE extensions that Python
>>> has adopted.
>>
>> Impossible?  Surely, a sufficiently-competent re engine could detect
>> when a DFA is possible to construct?
>
> I doubt it.  While it isn't equivalent to the halting problem, it IS
> an intractable one!  There are two problems:
>
> Firstly, things like backreferences are an absolute no-no.  They
> are not regular, and REs with them in cannot be converted to DFAs.
> That could be 'solved' by a parser that kicked out such constructions,
> but it would get screams from many users.
>
> Secondly, anything involving explicit or implicit negation can lead
> to (if I recall) a super-exponential explosion in the size of the
> DFA.  That could be 'solved' by imposing a limit, but few people
> would be able to predict when it would bite.

Right.  The analysis I envisioned would be more along the lines of  
"if troublesome RE extensions are used, do not attempt to construct a  
DFA".  It could even be exposed via an alternate api (re.compile_dfa 
()) that admitted a subset of the usual grammar.

> Thirdly, I would require notice of the question of whether capturing
> parentheses could be supported, and what semantic changes would be
> to which were set and how.

Capturing groups are rather integral to the python regex api and, as  
you say, a major difficulty for DFA-based implementations.  Sounds  
like a task best left to a thirdparty package.

-Mike

From martin at v.loewis.de  Wed Aug  8 22:38:03 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 08 Aug 2007 22:38:03 +0200
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IIrE7-0004pR-1S@libra.cus.cam.ac.uk>
References: <E1IIrE7-0004pR-1S@libra.cus.cam.ac.uk>
Message-ID: <46BA29AB.40804@v.loewis.de>

>> Before discussing the escape, I'd like to see a specification of
>> it first - what characters precisely would classify as "printing"?
> 
> For basic ASCII and locale-based testing, whatever isprint() says.
> Just as for isalpha().

In the mediate term, locale-based testing will go away/be not
implementable (in particular, Py3k won't have a byte-oriented
character string type, so we can't use isprint). In general,
isprint is unsuitable since it doesn't support multi-byte
character sets.

> For Unicode, whatever people agree!  I use the criterion that it
> has a defined category that doesn't start with 'C' - which is what
> I think that most people will accept.

-1. There must be a better specification than that.

Can you please explain the concept of "printing character"? If
you have a Unicode code point, how do you determine whether it
is printing? If rendering it would generate black pixels on white
background?

Regards,
Martin

From nmm1 at cus.cam.ac.uk  Wed Aug  8 23:16:37 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Wed, 08 Aug 2007 22:16:37 +0100
Subject: [Python-Dev] Regular expressions, Unicode etc.
Message-ID: <E1IIstR-00047Y-Du@draco.cus.cam.ac.uk>

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
> 
> >> Before discussing the escape, I'd like to see a specification of
> >> it first - what characters precisely would classify as "printing"?
> > 
> > For basic ASCII and locale-based testing, whatever isprint() says.
> > Just as for isalpha().
> 
> In the mediate term, locale-based testing will go away/be not
> implementable (in particular, Py3k won't have a byte-oriented
> character string type, so we can't use isprint). In general,
> isprint is unsuitable since it doesn't support multi-byte
> character sets.

Well, iswprint isn't so restricted :-)  I don't see the relevance
of this, as EXACTLY the same problem applies to isalnum and \w.
If you can solve one problem (and you have to solve the latter),
you can solve the other.

> > For Unicode, whatever people agree!  I use the criterion that it
> > has a defined category that doesn't start with 'C' - which is what
> > I think that most people will accept.
> 
> -1. There must be a better specification than that.
> 
> Can you please explain the concept of "printing character"? If
> you have a Unicode code point, how do you determine whether it
> is printing? If rendering it would generate black pixels on white
> background?

Eh?  This is a character set we are talking about.  The proposed
extensions to include font and colour are an aberration that I shall
thankfully be long retired before they hit.

Unicode has a two letter classification of each character, with
the main category being in upper case and the subsidiary one in
lower.  Let's ignore the latter, as it is irrelevant.  The main
categories are 'Z' (spaces), 'L' (letters), 'N' (numbers),
'S' (Symbols), 'P' (punctuation), 'M' (marks) and 'C' control
characters.

There are some pretty weird entries in 'L' and 'N' and the
difference between 'S', P' and 'M' is arcane, to a degree.  But
all of the categories except 'C' are things that display, and
'C' is mainly the ASCII controls we know and, er, love - with
some similar extras.

Obviously, unclassified characters should not be called printing,
and equally obviously controls shouldn't.  There is no clear
reason why the others should not be - especially as the difference
between a modifying accent and a free-standing one is something
so obscure that most people don't even know that there IS one.

The point about an escape for printing characters is to check
for bad characters in text input, and the rule I mentioned is
fine for that.  What's the problem with it?


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From martin at v.loewis.de  Wed Aug  8 23:54:06 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 08 Aug 2007 23:54:06 +0200
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IIstR-00047Y-Du@draco.cus.cam.ac.uk>
References: <E1IIstR-00047Y-Du@draco.cus.cam.ac.uk>
Message-ID: <46BA3B7E.2090208@v.loewis.de>

>> In the mediate term, locale-based testing will go away/be not
>> implementable (in particular, Py3k won't have a byte-oriented
>> character string type, so we can't use isprint). In general,
>> isprint is unsuitable since it doesn't support multi-byte
>> character sets.
> 
> Well, iswprint isn't so restricted :-) 

Yes. However, it is even more difficult to convert from
Py_UNICODE to wchar_t in general.

> I don't see the relevance
> of this, as EXACTLY the same problem applies to isalnum and \w.

There is no problem for isalnum: it will just go away if
byte-oriented characters go away. Fortunately, we have a
replacement for the Unicode case.

The relevance is that your specification of "printing character"
as "isprint returns true" is nearly useless, as it only applies
to byte-oriented characters.

> If you can solve one problem (and you have to solve the latter),
> you can solve the other.

Unicode-isalnum is defined as isalpha|isdecimal|isdigit|isnumeric.
isalpha means categories Ll, Lu, Lt, Lo, Lm. isdecimal means
character has the decimal property. isigit means the character has
the digit property. isnumeric means the character has the numeric
property.

>> Can you please explain the concept of "printing character"? If
>> you have a Unicode code point, how do you determine whether it
>> is printing? If rendering it would generate black pixels on white
>> background?
> 
> Eh?  This is a character set we are talking about.  The proposed
> extensions to include font and colour are an aberration that I shall
> thankfully be long retired before they hit.

It was a proposal for a definition. English is not my native
language, and "printing character" means nothing to me. So
I kindly asked for a definition, and suggested one possibility.
I would not have guessed that you consider white-space characters
as "printing", as they don't actually print anything.

> The point about an escape for printing characters is to check
> for bad characters in text input, and the rule I mentioned is
> fine for that.  What's the problem with it?

The problem is that you did not quite mention a rule, or else
I missed it.

You seem to be asking for being able to express "not a control
character". I propose that this is best done with UTS#18,
in which you would write

  [\P{C}] # or \P{Other}

If this is what you want, I'm all in favor of having it
implemented.

Regards,
Martin

From martin at v.loewis.de  Thu Aug  9 00:03:51 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 09 Aug 2007 00:03:51 +0200
Subject: [Python-Dev]
 =?iso-8859-1?q?cc=3A_=22Martin_v=2E_L=F6wis=22_=3Cma?=
 =?iso-8859-1?q?rtin=40v=2Eloewis=2Ede=3E?=
In-Reply-To: <E1IIrG1-0004q6-1w@libra.cus.cam.ac.uk>
References: <E1IIrG1-0004q6-1w@libra.cus.cam.ac.uk>
Message-ID: <46BA3DC7.4020709@v.loewis.de>

>> I would likely close such a report as "works for me" (after testing
>> it does - it did when I last ran it, which was before the release
>> of Python 2.5).
> 
> I think that you will find that you are using a non-standard
> environment and set of Python sources.

Please trust me that I didn't. See below.

> Well, here are a selection of the issues that I found:
> 
> The Makefile includes the command:
>     ncftpget -R ftp.unicode.org . Public/MAPPINGS
> Not merely is ncftpget not a standard utility, the current mappings
> are no longer at that location.  Indeed, I can see nothing useful in
> that directory at present, though I haven't searched it in depth!

Ah, the makefile. I don't think you use it create the Unicode database.

It's only good for generating the codecs (Lib/encodings)

AFAICT, the mappings are still where they always were: at the
location given in the Makefile. (e.g.
ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-15.TXT
)

For generating the Unicode database, you need to download the
files manually

> Looking through www.unicode.org, I could find the relevant files
> for 5.0.0, but for no other version.  No, I am NOT going to type
> in over a megabyte of data from the PDF!

And nobody asks you to. Just use

http://www.unicode.org/Public/4.1.0/ucd/

(also available through ftp)

Did you really believe the Unicode consortium doesn't have the
old versions of the character database online? Do you think
they are complete fools?

> makeunicodedata.py has a reference to the Unicode 3.2 files, but
> they are not present in the standard distribution, the Makefile
> doesn't fetch them, and I can't find them.

Googling for "unicode 3.2 ucd" gives me

http://unicode.org/Public/3.2-Update/

as the top hit (of course, you have to know that they call
the character database "ucd" to invoke that query).

> makeunicodedata.py refers to (for example) UnicodeData.txt and
> Modules/unicodedata_db.h as such, which rather requires it to be
> run in a particular directory.  I can find nothing in any file
> even referring to this.

Yes, that's something you have to know. Put the files into the
root directory of the source tree, then run makeunicodedata.py

> And, of course, it SHOULD be possible to upgrade the Unicode data
> without having to change version of Python!

Well.

Regards,
Martin

From nmm1 at cus.cam.ac.uk  Thu Aug  9 10:27:46 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Thu, 09 Aug 2007 09:27:46 +0100
Subject: [Python-Dev] Unicode database
Message-ID: <E1IJ3Mw-0002EL-Jq@libra.cus.cam.ac.uk>

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
>
> > I think that you will find that you are using a non-standard
> > environment and set of Python sources.
>
> Please trust me that I didn't. See below.

I always trust people as much as I trust myself, but I do tend to
check up.  See below.

> Ah, the makefile. I don't think you use it create the Unicode database.
> 
> It's only good for generating the codecs (Lib/encodings)

Yes, but it DOES attempt to download the mappings, and is the ONLY
script which attempts to do so.

beelzebub$find Python-2.5.1 -type f | wc
   3458    3460  135981
beelzebub$find Python-2.5.1 -type f | xargs grep ftp.unicode.org
Python-2.5.1/Doc/lib/libunicodedata.tex:4.1.0 which is publicly available from \url{ftp://ftp.unicode.org/}.
grep: Python-2.5.1/Mac/Icons/Disk: No such file or directory
grep: Image.icns: No such file or directory
grep: Python-2.5.1/Mac/Icons/Python: No such file or directory
grep: Folder.icns: No such file or directory
Python-2.5.1/Misc/NEWS:  at ftp.unicode.org and contain a few updates (e.g. the Mac OS
Python-2.5.1/Tools/unicode/Makefile:# files available at ftp://ftp.unicode.org/
Python-2.5.1/Tools/unicode/Makefile:    ncftpget -R ftp.unicode.org . Public/MAPPINGS
Python-2.5.1/Tools/unicode/gencodec.py:site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec
Python-2.5.1/Tools/unicode/python-mappings/TIS-620.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT the
Python-2.5.1/Tools/unicode/python-mappings/TIS-620.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT
Python-2.5.1/Tools/unicode/python-mappings/KOI8-U.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MISC/KOI8-R.TXT
Python-2.5.1/Tools/unicode/python-mappings/CP1140.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP037.TXT
Python-2.5.1/Modules/unicodedata.c:4.1.0 which is publically available from ftp://ftp.unicode.org/.\n

> AFAICT, the mappings are still where they always were: at the
> location given in the Makefile. (e.g.
> ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-15.TXT
> )

Then you DEFINITELY are using a non-standard set of files.  That
above was from the source of Python 2.5.1 that I have just downloaded.

> Did you really believe the Unicode consortium doesn't have the
> old versions of the character database online? Do you think
> they are complete fools?

Please don't be offensive.  I said that I had failed to find them,
after searching the Unicode Web site.  Now that you have give me
the actual file name, I can find them, but searching on the version
and request for that database leads to unhelpful files.

> Googling for "unicode 3.2 ucd" gives me
> 
> http://unicode.org/Public/3.2-Update/
> 
> as the top hit (of course, you have to know that they call
> the character database "ucd" to invoke that query).

Generally, I distrust Google for such things, as it is as likely
to lead to you the wrong information as the right one.  For example,
that hit you found was on a different logical server, and could
well be an incorrect version of the database.  It is VERY common
for such things to 'escape' into Google.

Have you checked whether or not that file is correct with the
Unicode consortium?


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Thu Aug  9 11:10:10 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Thu, 09 Aug 2007 10:10:10 +0100
Subject: [Python-Dev] Regular expressions, Unicode etc.
Message-ID: <E1IJ41y-0002fB-LR@libra.cus.cam.ac.uk>

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
>
> There is no problem for isalnum: it will just go away if
> byte-oriented characters go away. Fortunately, we have a
> replacement for the Unicode case.

As we do for isprint.

> The relevance is that your specification of "printing character"
> as "isprint returns true" is nearly useless, as it only applies
> to byte-oriented characters.

Eh?  That's ALL I used it to specify!  I used a Unicode-based
specification for Unicode.

> Unicode-isalnum is defined as isalpha|isdecimal|isdigit|isnumeric.
> isalpha means categories Ll, Lu, Lt, Lo, Lm. isdecimal means
> character has the decimal property. isigit means the character has
> the digit property. isnumeric means the character has the numeric
> property.

I sincerely hope it isn't!

Using a mixture of categories and properties is truly horrible,
because it isn't unlikely that some future version of Unicode will
introduce anomalies, even if there aren't any there already.  And
the character aliases file doesn't include any properties called
'digit' or 'decimal' or anything much like them, so they need a
painful amount of reverse engineering to determine what characters
they bind to.  It LOOKS as if they are the subcategories, which
would be OK.

A much cleaner and more future-proof specification would be any
category beginning with 'L' or 'N'.  For example, Unicode doesn't
CURRENTLY have a category for indeterminate numbers or sacred
case, such as are used in some languages, but it isn't implausible
that it would add them :-)

> It was a proposal for a definition. English is not my native
> language, and "printing character" means nothing to me. So
> I kindly asked for a definition, and suggested one possibility.
> I would not have guessed that you consider white-space characters
> as "printing", as they don't actually print anything.

Ah.  It's not an ordinary English term.  It's a computer language
one, so I assumed that you would know it.

It is older than C, but C standardised its use to mean any of the
characters which are intended to display (or leave a blank) with
standard, single positioning semantics.  Almost all languages
derived from C use it in the same sense, and Python has a fair
amount of C ancestry.

> The problem is that you did not quite mention a rule, or else
> I missed it.

I did, and you did!  I said that it should be any character with
a defined category that is not 'control'.

> You seem to be asking for being able to express "not a control
> character". I propose that this is best done with UTS#18,
> in which you would write
> 
>   [\P{C}] # or \P{Other}
>
> If this is what you want, I'm all in favor of having it
> implemented.

Excellent!  We are agreed.  Yes, that is equivalent.

I am NOT volunteering to add the support of that to the parser,
especially now I have discovered the format of the intermediate
data :-(  It would be a foul task, and it isn't clear what syntax
to use, anyway.

There is the horrible POSIX syntax, which I blame (perhaps wrongly)
on HP-UX, and the Java one, which I believe is a modified subset
of the example in UTS#8.  But that says:

    All syntax and API presented in this document is only for the
    purpose of illustration; there is absolutely no requirement to
    follow such syntax or API.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From mal at egenix.com  Thu Aug  9 11:23:50 2007
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 09 Aug 2007 11:23:50 +0200
Subject: [Python-Dev] Unicode database
In-Reply-To: <E1IJ3Mw-0002EL-Jq@libra.cus.cam.ac.uk>
References: <E1IJ3Mw-0002EL-Jq@libra.cus.cam.ac.uk>
Message-ID: <46BADD26.9030200@egenix.com>

Nick Maclaren wrote:
>> Ah, the makefile. I don't think you use it create the Unicode database.
>>
>> It's only good for generating the codecs (Lib/encodings)
> 
> Yes, but it DOES attempt to download the mappings, and is the ONLY
> script which attempts to do so.

Of course it does. The Tools/unicode/Makefile is meant to simplify
recreating the codecs from the (possibly updated) mapping on the Unicode
site.

If it doesn't work for you, that may well be possible, since I wrote
the Makefile and the other related stuff in that directory to help me
with updating the codecs from the mappings. It's only checked in for
convenience.

> beelzebub$find Python-2.5.1 -type f | wc
>    3458    3460  135981
> beelzebub$find Python-2.5.1 -type f | xargs grep ftp.unicode.org
> Python-2.5.1/Doc/lib/libunicodedata.tex:4.1.0 which is publicly available from \url{ftp://ftp.unicode.org/}.
> grep: Python-2.5.1/Mac/Icons/Disk: No such file or directory
> grep: Image.icns: No such file or directory
> grep: Python-2.5.1/Mac/Icons/Python: No such file or directory
> grep: Folder.icns: No such file or directory
> Python-2.5.1/Misc/NEWS:  at ftp.unicode.org and contain a few updates (e.g. the Mac OS
> Python-2.5.1/Tools/unicode/Makefile:# files available at ftp://ftp.unicode.org/
> Python-2.5.1/Tools/unicode/Makefile:    ncftpget -R ftp.unicode.org . Public/MAPPINGS
> Python-2.5.1/Tools/unicode/gencodec.py:site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec
> Python-2.5.1/Tools/unicode/python-mappings/TIS-620.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT the
> Python-2.5.1/Tools/unicode/python-mappings/TIS-620.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT
> Python-2.5.1/Tools/unicode/python-mappings/KOI8-U.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MISC/KOI8-R.TXT
> Python-2.5.1/Tools/unicode/python-mappings/CP1140.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP037.TXT
> Python-2.5.1/Modules/unicodedata.c:4.1.0 which is publically available from ftp://ftp.unicode.org/.\n
> 
>> AFAICT, the mappings are still where they always were: at the
>> location given in the Makefile. (e.g.
>> ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-15.TXT
>> )
> 
> Then you DEFINITELY are using a non-standard set of files.  That
> above was from the source of Python 2.5.1 that I have just downloaded.

No idea where you get that impression from, but then I'm not really
sure what you're after anyway ;-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 09 2007)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From guido at python.org  Thu Aug  9 17:11:52 2007
From: guido at python.org (Guido van Rossum)
Date: Thu, 9 Aug 2007 08:11:52 -0700
Subject: [Python-Dev] Move to a "py3k" branch *DONE*
In-Reply-To: <ca471dc20708090743k30562834uee2abd624f2f41b5@mail.gmail.com>
References: <ca471dc20708090743k30562834uee2abd624f2f41b5@mail.gmail.com>
Message-ID: <ca471dc20708090811i62d00662tac2303546f011fc5@mail.gmail.com>

Please spread the word. The py3k-struni branch is dead! Don't use it any more.

--Guido

---------- Forwarded message ----------
From: Guido van Rossum <guido at python.org>
Date: Aug 9, 2007 7:43 AM
Subject: Move to a "py3k" branch *DONE*
To: Python 3000 <python-3000 at python.org>
Cc: Neal Norwitz <nnorwitz at gmail.com>


This is done. The new py3k branch is ready for business.

If you currently have the py3k-struni branch checked out (at its top
level), *don't update*, but issue the following commands:

  svn switch svn+ssh://pythondev at svn.python.org/python/branches/py3k
  svn update

Only a small amount of activity should result (unless you didn't svn
update for a long time).

For the p3yk branch, the same instructions will work, but the svn
update will update most of your tree. A "make clean" is recommended in
this case.

Left to do:

- update the wikis
- clean out the old branches
- switch the buildbot and the doc builder to use the new branch (Neal)

There are currently about 7 failing unit tests left:

test_bsddb
test_bsddb3
test_email
test_email_codecs
test_email_renamed
test_sqlite
test_urllib2_localnet

See http://wiki.python.org/moin/Py3kStrUniTests for detailed status
regarding these.

--Guido

On 8/9/07, Guido van Rossum <guido at python.org> wrote:
> I am starting now. Please, no more checkins to either p3yk ot py3k-struni.
>
> On 8/8/07, Guido van Rossum <guido at python.org> wrote:
> > I would like to move to a new branch soon for all Py3k development.
> >
> > I plan to name the branch "py3k".  It will be branched from
> > py3k-struni.  I will do one last set of merges from the trunk via p3yk
> > (note typo!) and py3k-struni, and then I will *delete* the old py3k
> > and py3k-struni branches (you will still be able to access their last
> > known good status by syncing back to a previous revision).  I will
> > temporarily shut up some unit tests to avoid getting endless spam from
> > Neal's buildbot.
> >
> > After the switch, you should be able to switch your workspaces to the
> > new branch using the "svn switch" command.
> >
> > If anyone is in the middle of something that would become painful due
> > to this changeover, let me know ASAP and I'll delay.
> >
> > I will send out another message when I start the move, and another
> > when I finish it.
> >
> > --
> > --Guido van Rossum (home page: http://www.python.org/~guido/)
> >
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>


--
--Guido van Rossum (home page: http://www.python.org/~guido/)


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Fri Aug 10 00:51:47 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 10 Aug 2007 00:51:47 +0200
Subject: [Python-Dev] Unicode database
In-Reply-To: <E1IJ3Mw-0002EL-Jq@libra.cus.cam.ac.uk>
References: <E1IJ3Mw-0002EL-Jq@libra.cus.cam.ac.uk>
Message-ID: <46BB9A83.4030901@v.loewis.de>

>> Ah, the makefile. I don't think you use it create the Unicode database.
>>
>> It's only good for generating the codecs (Lib/encodings)
> 
> Yes, but it DOES attempt to download the mappings, and is the ONLY
> script which attempts to do so.

Sure. But (again): you don't need to have the mappings at all for
what you want to achieve. So there is no point in downloading them

> beelzebub$find Python-2.5.1 -type f | xargs grep ftp.unicode.org
> Python-2.5.1/Doc/lib/libunicodedata.tex:4.1.0 which is publicly available from \url{ftp://ftp.unicode.org/}.
> grep: Python-2.5.1/Mac/Icons/Disk: No such file or directory
> grep: Image.icns: No such file or directory
> grep: Python-2.5.1/Mac/Icons/Python: No such file or directory
> grep: Folder.icns: No such file or directory
> Python-2.5.1/Misc/NEWS:  at ftp.unicode.org and contain a few updates (e.g. the Mac OS
> Python-2.5.1/Tools/unicode/Makefile:# files available at ftp://ftp.unicode.org/
> Python-2.5.1/Tools/unicode/Makefile:    ncftpget -R ftp.unicode.org . Public/MAPPINGS
> Python-2.5.1/Tools/unicode/gencodec.py:site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec
> Python-2.5.1/Tools/unicode/python-mappings/TIS-620.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT the
> Python-2.5.1/Tools/unicode/python-mappings/TIS-620.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT
> Python-2.5.1/Tools/unicode/python-mappings/KOI8-U.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MISC/KOI8-R.TXT
> Python-2.5.1/Tools/unicode/python-mappings/CP1140.TXT:#       ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP037.TXT
> Python-2.5.1/Modules/unicodedata.c:4.1.0 which is publically available from ftp://ftp.unicode.org/.\n
> 
>> AFAICT, the mappings are still where they always were: at the
>> location given in the Makefile. (e.g.
>> ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-15.TXT
>> )
> 
> Then you DEFINITELY are using a non-standard set of files.  That
> above was from the source of Python 2.5.1 that I have just downloaded.

I don't understand. Why does this follow? What should I read out
of the grep lines above, and why does my citing of a URL prove
that I did something to my build environment?

Regards,
Martin

From martin at v.loewis.de  Fri Aug 10 00:59:44 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 10 Aug 2007 00:59:44 +0200
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IJ41y-0002fB-LR@libra.cus.cam.ac.uk>
References: <E1IJ41y-0002fB-LR@libra.cus.cam.ac.uk>
Message-ID: <46BB9C60.2030405@v.loewis.de>

Nick Maclaren schrieb:
>> The relevance is that your specification of "printing character"
>> as "isprint returns true" is nearly useless, as it only applies
>> to byte-oriented characters.
> 
> Eh?  That's ALL I used it to specify!  I used a Unicode-based
> specification for Unicode.

Your specification was "For Unicode, whatever people agree!"

I would not call that "Unicode-based".

>> Unicode-isalnum is defined as isalpha|isdecimal|isdigit|isnumeric.
>> isalpha means categories Ll, Lu, Lt, Lo, Lm. isdecimal means
>> character has the decimal property. isigit means the character has
>> the digit property. isnumeric means the character has the numeric
>> property.
> 
> I sincerely hope it isn't!

Please read the code.

>> It was a proposal for a definition. English is not my native
>> language, and "printing character" means nothing to me. So
>> I kindly asked for a definition, and suggested one possibility.
>> I would not have guessed that you consider white-space characters
>> as "printing", as they don't actually print anything.
> 
> Ah.  It's not an ordinary English term.  It's a computer language
> one, so I assumed that you would know it.

I know the term "printable character", which is what I read
in definitions of the isprint() routine. "printing character"
I never heard before.

Regards,
Martin


From greg.ewing at canterbury.ac.nz  Fri Aug 10 02:28:49 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 10 Aug 2007 12:28:49 +1200
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <46BB9C60.2030405@v.loewis.de>
References: <E1IJ41y-0002fB-LR@libra.cus.cam.ac.uk>
	<46BB9C60.2030405@v.loewis.de>
Message-ID: <46BBB141.700@canterbury.ac.nz>

Martin v. L?wis wrote:
> I know the term "printable character", which is what I read
> in definitions of the isprint() routine. "printing character"
> I never heard before.

Hmmm... I guess this means your brain is using a
part-of-speech-sensitive word->technical_meaning
mapping.

Perhaps this will be fixed in English 3.0...

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | Carpe post meridiem!          	  |
Christchurch, New Zealand	   | (I'm not a morning person.)          |
greg.ewing at canterbury.ac.nz	   +--------------------------------------+

From foom at fuhm.net  Fri Aug 10 07:02:16 2007
From: foom at fuhm.net (James Y Knight)
Date: Fri, 10 Aug 2007 01:02:16 -0400
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IIrUt-0004wR-Km@libra.cus.cam.ac.uk>
References: <E1IIrUt-0004wR-Km@libra.cus.cam.ac.uk>
Message-ID: <C66FD236-B0C9-4FA9-870B-1BF94001B4BF@fuhm.net>

On Aug 8, 2007, at 3:47 PM, Nick Maclaren wrote:
> Firstly, things like backreferences are an absolute no-no.  They
> are not regular, and REs with them in cannot be converted to DFAs.
> That could be 'solved' by a parser that kicked out such constructions,
> but it would get screams from many users.

People keep saying things like this as if GNU grep and tcl's regular  
expression matchers didn't exist.
See http://www.tcl.tk/man/tcl8.5/TclCmd/re_syntax.htm for example.

time python -c 'import re; print re.match("("+"a?"*26+"a"*26+")b\\1",  
"a"*26+"b"+"a"*26).group(0)'
aaaaaaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaaaaaa

real    0m5.913s
user    0m5.905s
sys     0m0.006s

time echo 'aaaaaaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaaaaaa' |  
grep -E '(a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a?a? 
aaaaaaaaaaaaaaaaaaaaaaaaaa)b\1'
aaaaaaaaaaaaaaaaaaaaaaaaaabaaaaaaaaaaaaaaaaaaaaaaaaaa

real    0m0.002s
user    0m0.002s
sys     0m0.000s

James

From greg.ewing at canterbury.ac.nz  Fri Aug 10 09:40:28 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 10 Aug 2007 19:40:28 +1200
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <C66FD236-B0C9-4FA9-870B-1BF94001B4BF@fuhm.net>
References: <E1IIrUt-0004wR-Km@libra.cus.cam.ac.uk>
	<C66FD236-B0C9-4FA9-870B-1BF94001B4BF@fuhm.net>
Message-ID: <46BC166C.7000300@canterbury.ac.nz>

James Y Knight wrote:
> On Aug 8, 2007, at 3:47 PM, Nick Maclaren wrote:
> > Firstly, things like backreferences are an absolute no-no.  They
> > are not regular, and REs with them in cannot be converted to DFAs.
>
> People keep saying things like this as if GNU grep and tcl's regular  
> expression matchers didn't exist.

But do these work by conversion to a DFA?

--
Greg

From nmm1 at cus.cam.ac.uk  Fri Aug 10 10:12:33 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 10 Aug 2007 09:12:33 +0100
Subject: [Python-Dev] Regular expressions, Unicode etc.
Message-ID: <E1IJPbl-0006uZ-KV@draco.cus.cam.ac.uk>

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
>
> Your specification was "For Unicode, whatever people agree!"
>
> I would not call that "Unicode-based".

Can we drop this, please?  I am happy to agree that I was being unclear
(it is a common failing of mine), but I did provide the specification
I coded.  Specifically, and in full, I said:

    For Unicode, whatever people agree!  I use the criterion that it
    has a defined category that doesn't start with 'C' - which is what
    I think that most people will accept.

That is equivalent to the definition you gave.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Fri Aug 10 10:23:42 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 10 Aug 2007 09:23:42 +0100
Subject: [Python-Dev] Unicode database
Message-ID: <E1IJPmY-00076H-8h@draco.cus.cam.ac.uk>

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
>
> Sure. But (again): you don't need to have the mappings at all for
> what you want to achieve. So there is no point in downloading them

Sigh.  No, I don't.  But, if I want to be able to merge anything
back into the main Python source, it is a VERY good idea to use the
existing mechanisms and not invent new ones.

The easiest thing would have been to hack re.py to create a Unicode
table using unicodedata.py directly, and that would indeed be a rather
cleaner solution in the long term.  But it would have meant that there
were now multiple different ways of generating the Unicode data for
_sre.c, and that would have led to inconsistencies.

As I pointed out, there is already a problem where upgrading the data
needs a complete rebuild to get all of the Unicode data back in step;
'make all' in itself does not work.  That is precisely the sort of
problem that is caused by having duplicate update mechanisms.


Now, IF I can work out how the _sre.c engine works enough to put
atomic/possessive quantifiers in, this problem will return.  My
question would be how best to make a suitable proposal that, inter
alia, includes changes that can't be made by the normal building
mechanisms.

And I still don't have a clue about that one.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Fri Aug 10 10:28:58 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 10 Aug 2007 09:28:58 +0100
Subject: [Python-Dev] Regular expressions, Unicode etc.
Message-ID: <E1IJPre-0007As-1g@draco.cus.cam.ac.uk>

James Y Knight <foom at fuhm.net> wrote:
>
> > Firstly, things like backreferences are an absolute no-no.  They
> > are not regular, and REs with them in cannot be converted to DFAs.
> > That could be 'solved' by a parser that kicked out such constructions,
> > but it would get screams from many users.
> 
> People keep saying things like this as if GNU grep and tcl's regular  
> expression matchers didn't exist.
> See http://www.tcl.tk/man/tcl8.5/TclCmd/re_syntax.htm for example.

PCRE also has a breadth-first engine, but it does not convert the
NFA to a DFA (its author is a close colleague of mine).  Those
engines won't do the conversion, either, and I am prepared to bet
that I could produce a pattern that would either run very slowly
or expose the semantics differences in most of them.

I did NOT say that there were not, alternative, approaches.  What
I said was correct - you cannot convert such extended expressions
to DFAs.  You can convert them to things that are sort of NFA/DFA
hybrids, which might or might not be a good way to proceed.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From guido at python.org  Fri Aug 10 20:23:45 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 10 Aug 2007 11:23:45 -0700
Subject: [Python-Dev] Universal newlines support in Python 3.0
Message-ID: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>

Python 3.0 currently has limited universal newlines support: by
default, \r\n is translated into \n for text files, but this can be
controlled by the newline= keyword parameter. For details on how, see
PEP 3116. The PEP prescribes that a lone \r must also be translated,
though this hasn't been implemented yet (any volunteers?).

However, the old universal newlines feature also set an attibute named
'newlines' on the file object to a tuple of up to three elements
giving the actual line endings that were observed on the file so far
(\r, \n, or \r\n). This feature is not in PEP 3116, and it is not
implemented. I'm tempted to kill it. Does anyone have a use case for
this? Has anyone even ever used this?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From stephen at xemacs.org  Fri Aug 10 21:15:45 2007
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 11 Aug 2007 04:15:45 +0900
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python 3.0
In-Reply-To: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
Message-ID: <87wsw3p5em.fsf@uwakimon.sk.tsukuba.ac.jp>

Guido van Rossum writes:

 > However, the old universal newlines feature also set an attibute named
 > 'newlines' on the file object to a tuple of up to three elements
 > giving the actual line endings that were observed on the file so far
 > (\r, \n, or \r\n). This feature is not in PEP 3116, and it is not
 > implemented. I'm tempted to kill it. Does anyone have a use case for
 > this?

I have run into files that intentionally have more than one newline
convention used (mbox and Babyl mail folders, with messages received
from various platforms).  However, most of the time multiple newline
conventions is a sign that the file is either corrupt or isn't text.
If so, then saving the file may corrupt it.  The newlines attribute
could be used to check for this condition.

 > Has anyone even ever used this?

Not I.  When I care about such issues I prefer that the codec raise an
exception at the time of detection.


From martin at v.loewis.de  Sat Aug 11 01:01:19 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 11 Aug 2007 01:01:19 +0200
Subject: [Python-Dev] Unicode database
In-Reply-To: <E1IJPmY-00076H-8h@draco.cus.cam.ac.uk>
References: <E1IJPmY-00076H-8h@draco.cus.cam.ac.uk>
Message-ID: <46BCEE3F.7020109@v.loewis.de>

>> Sure. But (again): you don't need to have the mappings at all for
>> what you want to achieve. So there is no point in downloading them
> 
> Sigh.  No, I don't.  But, if I want to be able to merge anything
> back into the main Python source, it is a VERY good idea to use the
> existing mechanisms and not invent new ones.

I think you still don't understand. Why I keep calling "mappings"
is *unrelated* to unicodedata. unicodedata is a different database, and
not related at all to the makefile. It never was.

> As I pointed out, there is already a problem where upgrading the data
> needs a complete rebuild to get all of the Unicode data back in step;
> 'make all' in itself does not work.  That is precisely the sort of
> problem that is caused by having duplicate update mechanisms.

Right. Downloading the necessary files is a completely manual process,
not supported at all by "make all", which is designed to do something
entirely different.

> Now, IF I can work out how the _sre.c engine works enough to put
> atomic/possessive quantifiers in, this problem will return.  My
> question would be how best to make a suitable proposal that, inter
> alia, includes changes that can't be made by the normal building
> mechanisms.
> 
> And I still don't have a clue about that one.

You lost me somewhere. What are "changes that can't be made by the
normal building process", and what is "this problem" that will
return?

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Sat Aug 11 03:28:07 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 11 Aug 2007 13:28:07 +1200
Subject: [Python-Dev] Regular expressions, Unicode etc.
In-Reply-To: <E1IJPre-0007As-1g@draco.cus.cam.ac.uk>
References: <E1IJPre-0007As-1g@draco.cus.cam.ac.uk>
Message-ID: <46BD10A7.4030708@canterbury.ac.nz>

Nick Maclaren wrote:
> You can convert them to things that are sort of NFA/DFA
> hybrids,

If you could express it as an NFA, then you could
(in principle) convert it to a DFA. So whatever it's
using can't be an NFA either.

--
Greg

From kbk at shore.net  Sat Aug 11 04:08:34 2007
From: kbk at shore.net (Kurt B. Kaiser)
Date: Fri, 10 Aug 2007 22:08:34 -0400 (EDT)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200708110208.l7B28YEW028649@hampton.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  404 open ( +0) /  3855 closed ( +8) /  4259 total ( +8)
Bugs    : 1065 open ( +6) /  6790 closed ( +6) /  7855 total (+12)
RFE     :  263 open ( +0) /   295 closed ( +0) /   558 total ( +0)

New / Reopened Patches
______________________

MSVC++8 x86 tkinter build patch for trunk  (2007-08-05)
       http://python.org/sf/1767787  opened by  brotchie

test_asyncore fix  (2007-08-05)
CLOSED http://python.org/sf/1767834  opened by  Hasan Diwan

Fix for failing test_scriptpackages in py3k-struni  (2007-08-07)
CLOSED http://python.org/sf/1768976  opened by  Antti Rasinen

Fix for failing test_plistlib in py3k-struni  (2007-08-07)
CLOSED http://python.org/sf/1769016  opened by  brotchie

struni: test_xml_etree_c  (2007-08-08)
CLOSED http://python.org/sf/1769767  opened by  Joe Gregorio

Remove cStringIO usage  (2007-08-08)
CLOSED http://python.org/sf/1770008  reopened by  tiran

Remove cStringIO usage  (2007-08-08)
CLOSED http://python.org/sf/1770008  opened by  Christian Heimes

ctypes: c_char now uses bytes and not str (unicode)  (2007-08-08)
CLOSED http://python.org/sf/1770355  opened by  STINNER Victor

Misc improvements for the io module  (2007-08-10)
       http://python.org/sf/1771364  opened by  Christian Heimes

Patches Closed
______________

test_asyncore fix  (2007-08-05)
       http://python.org/sf/1767834  closed by  gvanrossum

test_csv struni fixes + unicode support in _csv  (2007-08-03)
       http://python.org/sf/1767398  closed by  gvanrossum

urllib2-howto - correction  (2007-08-02)
       http://python.org/sf/1765839  closed by  gbrandl

Fix for failing test_scriptpackages in py3k-struni  (2007-08-06)
       http://python.org/sf/1768976  closed by  nnorwitz

Fix for failing test_plistlib in py3k-struni  (2007-08-07)
       http://python.org/sf/1769016  closed by  gvanrossum

struni: test_xml_etree_c  (2007-08-07)
       http://python.org/sf/1769767  closed by  nnorwitz

Remove cStringIO usage  (2007-08-08)
       http://python.org/sf/1770008  closed by  gvanrossum

Remove cStringIO usage  (2007-08-08)
       http://python.org/sf/1770008  closed by  gvanrossum

ctypes: c_char now uses bytes and not str (unicode)  (2007-08-08)
       http://python.org/sf/1770355  closed by  haypo

New / Reopened Bugs
___________________

SocketServer.DatagramRequestHandler  (2007-08-04)
       http://python.org/sf/1767511  opened by  Alzheimer

Badly formed XML using etree and utf-16  (2007-08-05)
       http://python.org/sf/1767933  opened by  BugoK

Byte code WITH_CLEANUP missing, MAKE_CLOSURE wrong  (2007-08-05)
       http://python.org/sf/1768121  opened by  L. Peter Deutsch

tutorial  (2007-08-06)
CLOSED http://python.org/sf/1768767  opened by  Michael R Bax

Python - Operation time out problem   (2007-08-06)
       http://python.org/sf/1768858  opened by  MASK

A paragraph about packages should be updated.  (2007-08-07)
CLOSED http://python.org/sf/1769002  opened by  Noam Raphael

decimal.Decimal("trash") produces informationless exception  (2007-08-08)
       http://python.org/sf/1770009  opened by  John Machin

platform.mac_ver() returning incorrect patch version  (2007-08-08)
       http://python.org/sf/1770190  opened by  Gus Tabares

Decimal.__int__ overflows for large values  (2007-08-08)
       http://python.org/sf/1770416  opened by  Jason G

words able to decode but unable to encode in GB18030  (2007-08-09)
       http://python.org/sf/1770551  opened by  Z-flagship

Errors in site.py not reported properly  (2007-08-09)
       http://python.org/sf/1771260  opened by  Adam Olsen

bsddb can't use unicode keys  (2007-08-10)
       http://python.org/sf/1771381  opened by  Erol Aktay

another 'nothing to repeat'  (2007-08-10)
CLOSED http://python.org/sf/1771483  opened by  viciousdog

minor bug in turtle  (2007-08-10)
CLOSED http://python.org/sf/1771558  opened by  Jeremy Sanders

Bugs Closed
___________

String.capwords() does not capitalize first word  (2007-08-03)
       http://python.org/sf/1767363  closed by  gbrandl

subprocess.Popen.wait fails sporadically with threads  (2007-07-16)
       http://python.org/sf/1754642  closed by  gbrandl

subprocess raising "No Child Process" OSError  (2007-07-14)
       http://python.org/sf/1753891  closed by  gbrandl

tutorial  (2007-08-06)
       http://python.org/sf/1768767  deleted by  mrbax

A paragraph about packages should be updated.  (2007-08-07)
       http://python.org/sf/1769002  closed by  gbrandl

cStringIO no longer accepts array.array objects  (2007-06-03)
       http://python.org/sf/1730114  closed by  gbrandl

another 'nothing to repeat'  (2007-08-10)
       http://python.org/sf/1771483  deleted by  viciousdog

minor bug in turtle  (2007-08-10)
       http://python.org/sf/1771558  closed by  gbrandl


From g.brandl at gmx.net  Sat Aug 11 11:08:24 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 11 Aug 2007 11:08:24 +0200
Subject: [Python-Dev] Exception pickling patch
Message-ID: <f9jua7$hst$1@sea.gmane.org>

Can somebody please review this patch:

https://sourceforge.net/support/tracker.php?aid=1692335

It aims to fix the pickling of exceptions whose __init__ methods don't call
Exception.__init__ at all, or with a different number of arguments.

This should be fixed before 2.5.2.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From tony at PageDNA.com  Sat Aug 11 18:45:37 2007
From: tony at PageDNA.com (Tony Lownds)
Date: Sat, 11 Aug 2007 09:45:37 -0700
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
In-Reply-To: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
Message-ID: <E1DD53D0-168D-4DC1-BF76-E7A3DE64F359@PageDNA.com>


On Aug 10, 2007, at 11:23 AM, Guido van Rossum wrote:

> Python 3.0 currently has limited universal newlines support: by
> default, \r\n is translated into \n for text files, but this can be
> controlled by the newline= keyword parameter. For details on how, see
> PEP 3116. The PEP prescribes that a lone \r must also be translated,
> though this hasn't been implemented yet (any volunteers?).
>

I'm working on this, but now I'm not sure how the file is supposed to  
be read when
the newline parameter is \r or \r\n. Here's the PEP language:

   buffer is a reference to the BufferedIOBase object to be wrapped  
with the TextIOWrapper.
   encoding refers to an encoding to be used for translating between  
the byte-representation
   and character-representation. If it is None, then the system's  
locale setting will be used
   as the default. newline can be None, '\n', '\r', or '\r\n' (all  
other values are illegal);
   it indicates the translation for '\n' characters written. If None,  
a system-specific default
   is chosen, i.e., '\r\n' on Windows and '\n' on Unix/Linux. Setting  
newline='\n' on input
   means that no CRLF translation is done; lines ending in '\r\n'  
will be returned as '\r\n'.
   ('\r' support is still needed for some OSX applications that  
produce files using '\r' line
   endings; Excel (when exporting to text) and Adobe Illustrator EPS  
files are the most common examples.

Is this ok: when newline='\r\n' or newline='\r' is passed, only that  
string is used to determine
the end of lines. No translation to '\n' is done.

> However, the old universal newlines feature also set an attibute named
> 'newlines' on the file object to a tuple of up to three elements
> giving the actual line endings that were observed on the file so far
> (\r, \n, or \r\n). This feature is not in PEP 3116, and it is not
> implemented. I'm tempted to kill it. Does anyone have a use case for
> this? Has anyone even ever used this?
>

This strikes me as a pragmatic feature, making it easy to read a file
and write back the same line ending. I can include in patch.

http://www.google.com/codesearch?hl=en&q=+lang:python+%22.newlines%22 
+show:cz2Fhijwr3s:yutdXigOmYY:YDns9IyEkLQ&sa=N&cd=12&ct=rc&cs_p=http://f 
tp.gnome.org/pub/gnome/sources/meld/1.0/ 
meld-1.0.0.tar.bz2&cs_f=meld-1.0.0/filediff.py#a0

http://www.google.com/codesearch?hl=en&q=+lang:python+%22.newlines%22 
+show:SLyZnjuFadw:kOTmKU8aU2I:VX_dFr3mrWw&sa=N&cd=37&ct=rc&cs_p=http://s 
vn.python.org/projects/ctypes/trunk&cs_f=ctypeslib/ctypeslib/ 
dynamic_module.py#a0

Thanks
-Tony


From guido at python.org  Sat Aug 11 19:29:38 2007
From: guido at python.org (Guido van Rossum)
Date: Sat, 11 Aug 2007 10:29:38 -0700
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
In-Reply-To: <E1DD53D0-168D-4DC1-BF76-E7A3DE64F359@PageDNA.com>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
	<E1DD53D0-168D-4DC1-BF76-E7A3DE64F359@PageDNA.com>
Message-ID: <ca471dc20708111029n5722ff04h824eb8788a4c824e@mail.gmail.com>

On 8/11/07, Tony Lownds <tony at pagedna.com> wrote:
>
> On Aug 10, 2007, at 11:23 AM, Guido van Rossum wrote:
>
> > Python 3.0 currently has limited universal newlines support: by
> > default, \r\n is translated into \n for text files, but this can be
> > controlled by the newline= keyword parameter. For details on how, see
> > PEP 3116. The PEP prescribes that a lone \r must also be translated,
> > though this hasn't been implemented yet (any volunteers?).
> >
>
> I'm working on this, but now I'm not sure how the file is supposed to
> be read when
> the newline parameter is \r or \r\n. Here's the PEP language:
>
>    buffer is a reference to the BufferedIOBase object to be wrapped
> with the TextIOWrapper.
>    encoding refers to an encoding to be used for translating between
> the byte-representation
>    and character-representation. If it is None, then the system's
> locale setting will be used
>    as the default. newline can be None, '\n', '\r', or '\r\n' (all
> other values are illegal);
>    it indicates the translation for '\n' characters written. If None,
> a system-specific default
>    is chosen, i.e., '\r\n' on Windows and '\n' on Unix/Linux. Setting
> newline='\n' on input
>    means that no CRLF translation is done; lines ending in '\r\n'
> will be returned as '\r\n'.
>    ('\r' support is still needed for some OSX applications that
> produce files using '\r' line
>    endings; Excel (when exporting to text) and Adobe Illustrator EPS
> files are the most common examples.
>
> Is this ok: when newline='\r\n' or newline='\r' is passed, only that
> string is used to determine
> the end of lines. No translation to '\n' is done.

I *think* it would be more useful if it always returned lines ending
in \n (not \r\n or \r). Wouldn't it? Although this is not how it
currently behaves; when you set newline='\r\n', it returns the \r\n
unchanged, so it would make sense to do this too when newline='\r'.
Caveat user I guess.

> > However, the old universal newlines feature also set an attibute named
> > 'newlines' on the file object to a tuple of up to three elements
> > giving the actual line endings that were observed on the file so far
> > (\r, \n, or \r\n). This feature is not in PEP 3116, and it is not
> > implemented. I'm tempted to kill it. Does anyone have a use case for
> > this? Has anyone even ever used this?
> >
>
> This strikes me as a pragmatic feature, making it easy to read a file
> and write back the same line ending. I can include in patch.

OK, if you think you can, that's good. It's not always sufficient (not
if there was a mix of line endings) but it's a start.

> http://www.google.com/codesearch?hl=en&q=+lang:python+%22.newlines%22
> +show:cz2Fhijwr3s:yutdXigOmYY:YDns9IyEkLQ&sa=N&cd=12&ct=rc&cs_p=http://f
> tp.gnome.org/pub/gnome/sources/meld/1.0/
> meld-1.0.0.tar.bz2&cs_f=meld-1.0.0/filediff.py#a0
>
> http://www.google.com/codesearch?hl=en&q=+lang:python+%22.newlines%22
> +show:SLyZnjuFadw:kOTmKU8aU2I:VX_dFr3mrWw&sa=N&cd=37&ct=rc&cs_p=http://s
> vn.python.org/projects/ctypes/trunk&cs_f=ctypeslib/ctypeslib/
> dynamic_module.py#a0

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tony at pagedna.com  Sat Aug 11 20:41:08 2007
From: tony at pagedna.com (Tony Lownds)
Date: Sat, 11 Aug 2007 11:41:08 -0700
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
In-Reply-To: <ca471dc20708111029n5722ff04h824eb8788a4c824e@mail.gmail.com>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
	<E1DD53D0-168D-4DC1-BF76-E7A3DE64F359@PageDNA.com>
	<ca471dc20708111029n5722ff04h824eb8788a4c824e@mail.gmail.com>
Message-ID: <B416984A-309B-42FA-90D2-B37549D8F139@pagedna.com>


On Aug 11, 2007, at 10:29 AM, Guido van Rossum wrote:
>> Is this ok: when newline='\r\n' or newline='\r' is passed, only that
>> string is used to determine
>> the end of lines. No translation to '\n' is done.
>
> I *think* it would be more useful if it always returned lines ending
> in \n (not \r\n or \r). Wouldn't it? Although this is not how it
> currently behaves; when you set newline='\r\n', it returns the \r\n
> unchanged, so it would make sense to do this too when newline='\r'.
> Caveat user I guess.

Because there's an easy way to translate, having the option to not  
translate
apply to all valid newline values is probably more useful. I do think  
it's easier
to define the behavior this way.

> OK, if you think you can, that's good. It's not always sufficient (not
> if there was a mix of line endings) but it's a start.

Right

-Tony

From status at bugs.python.org  Sun Aug 12 02:00:55 2007
From: status at bugs.python.org (Tracker)
Date: Sun, 12 Aug 2007 00:00:55 +0000 (UTC)
Subject: [Python-Dev] Summary of Tracker Issues
Message-ID: <20070812000055.9ED9C781B4@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (08/05/07 - 08/12/07)
Tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 1295 open ( +8) / 11130 closed ( +2) / 12425 total (+10)

Average duration of open issues: 690 days.
Median duration of open issues: 553 days.

Open Issues Breakdown
   open  1295 ( +8)
pending     0 ( +0)

Issues Created Or Reopened (10)
_______________________________

x                                                                08/08/07
       http://bugs.python.org/issue1000    created  gbrandl        

MSVC++8 x86 tkinter build patch for trunk                        08/05/07
       http://bugs.python.org/issue1767787 created  brotch         

test_asyncore fix                                                08/05/07
       http://bugs.python.org/issue1767834 created  hdiwan650      

Badly formed XML using etree and utf-16                          08/05/07
       http://bugs.python.org/issue1767933 created  bugok          

Byte code WITH_CLEANUP missing, MAKE_CLOSURE wrong               08/06/07
       http://bugs.python.org/issue1768121 created  lpd            

tutorial                                                         08/06/07
CLOSED http://bugs.python.org/issue1768767 created  mrbax          

Python - Operation time out problem                              08/06/07
       http://bugs.python.org/issue1768858 created  mohammedsk     

Fix for failing test_scriptpackages in py3k-struni               08/07/07
CLOSED http://bugs.python.org/issue1768976 created  arsatiki       

A paragraph about packages should be updated.                    08/07/07
       http://bugs.python.org/issue1769002 created  noamr          

Fix for failing test_plistlib in py3k-struni                     08/07/07
       http://bugs.python.org/issue1769016 created  brotch         



Issues Now Closed (7)
_____________________

fix 1668596: copy datafiles properly when package_dir is ' '      83 days
       http://bugs.python.org/issue1720897 loewis         

unicode(None,charset) raise TypeError                             15 days
       http://bugs.python.org/issue1758804 sf-robot       

socket close fixed                                                 6 days
       http://bugs.python.org/issue1763387 jyasskin       

urllib2-howto - correction                                         4 days
       http://bugs.python.org/issue1765839 gbrandl        

test_csv struni fixes + unicode support in _csv                    3 days
       http://bugs.python.org/issue1767398 gvanrossum     

tutorial                                                           0 days
       http://bugs.python.org/issue1768767 mrbax          

Fix for failing test_scriptpackages in py3k-struni                 0 days
       http://bugs.python.org/issue1768976 nnorwitz       



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070812/d9278580/attachment.htm 

From martin at v.loewis.de  Sun Aug 12 12:29:07 2007
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 12 Aug 2007 12:29:07 +0200
Subject: [Python-Dev] Dropping support for Win9x
Message-ID: <46BEE0F3.7070904@v.loewis.de>

I'd like to remove support for Windows 9x (95, 98(SE), ME)
soon from the Python trunk. This would primarily affect all
wide-string APIs (which would be considered present
unconditionally), as well as certain "new" Win32 functions;
in this cleanup, I would also drop support for NT+ before
Windows 2000 (i.e. NT 3.1, 3.5(1), 4.0).

I'm not sure whether w9xpopen should be left in place; Tim
suggested it should, however, I don't see why (something
with alternative command line interpreters IIRC).

The 2.5 installer already gives a warning on W9x that this
will be the last release.

If you object to this plan, please speak up.

Regards,
Martin

From p.f.moore at gmail.com  Sun Aug 12 18:58:44 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 12 Aug 2007 17:58:44 +0100
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
In-Reply-To: <ca471dc20708111029n5722ff04h824eb8788a4c824e@mail.gmail.com>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
	<E1DD53D0-168D-4DC1-BF76-E7A3DE64F359@PageDNA.com>
	<ca471dc20708111029n5722ff04h824eb8788a4c824e@mail.gmail.com>
Message-ID: <79990c6b0708120958p588aabd1ic6dadf2f65de86d3@mail.gmail.com>

On 11/08/07, Guido van Rossum <guido at python.org> wrote:
> On 8/11/07, Tony Lownds <tony at pagedna.com> wrote:
> > Is this ok: when newline='\r\n' or newline='\r' is passed, only that
> > string is used to determine
> > the end of lines. No translation to '\n' is done.
>
> I *think* it would be more useful if it always returned lines ending
> in \n (not \r\n or \r). Wouldn't it? Although this is not how it
> currently behaves; when you set newline='\r\n', it returns the \r\n
> unchanged, so it would make sense to do this too when newline='\r'.
> Caveat user I guess.

Neither this wording, nor the PEP are clear to me, but I'm
assuming/hoping that there will be a way to spell the current
behaviour for universal newlines on input[1], namely that files can
have *either* bare \n, *or* the combination \r\n, to delimit lines.
Whichever is used (I have no need for mixed-style files) gets
translated to \n so that the program sees the same data regardless.

[1] ... at least the bit I care about :-)

This behaviour is immensely useful for uniform treatment of Windows
text files, which are an inconsistent mess of \n-only and \r\n
conventions.

Specifically, I'm looking to replicate this behaviour:

>xxd crlf
0000000: 610d 0a62 0d0a                           a..b..

>xxd lf
0000000: 610a 620a                                a.b.

>python
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> open('crlf').read()
'a\nb\n'
>>> open('lf').read()
'a\nb\n'
>>>

As demonstrated, this is the default in Python 2.5. I'd hope it was so
in 3.0 as well.

Sorry I can't test this for myself - I don't have the time/toolset to
build my own Py3k on Windows...

Paul.

From g.brandl at gmx.net  Sun Aug 12 20:24:07 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 12 Aug 2007 20:24:07 +0200
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
In-Reply-To: <79990c6b0708120958p588aabd1ic6dadf2f65de86d3@mail.gmail.com>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>	<E1DD53D0-168D-4DC1-BF76-E7A3DE64F359@PageDNA.com>	<ca471dc20708111029n5722ff04h824eb8788a4c824e@mail.gmail.com>
	<79990c6b0708120958p588aabd1ic6dadf2f65de86d3@mail.gmail.com>
Message-ID: <f9nj86$l8b$1@sea.gmane.org>

Paul Moore schrieb:

> Specifically, I'm looking to replicate this behaviour:
> 
>>xxd crlf
> 0000000: 610d 0a62 0d0a                           a..b..
> 
>>xxd lf
> 0000000: 610a 620a                                a.b.
> 
>>python
> Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
> (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> open('crlf').read()
> 'a\nb\n'
>>>> open('lf').read()
> 'a\nb\n'
>>>>
> 
> As demonstrated, this is the default in Python 2.5. I'd hope it was so
> in 3.0 as well.

Note that Python does nothing special in the above case. For non-Windows
platforms, you'd get two different results -- the conversion from \r\n to
\n is done by the Windows C runtime since the default open() mode is text mode.

Only with mode 'U' does Python use its own universal newline mode.

With Python 3.0, the C library is not used and Python uses universal newline
mode by default.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From p.f.moore at gmail.com  Sun Aug 12 21:12:29 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 12 Aug 2007 20:12:29 +0100
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
In-Reply-To: <f9nj86$l8b$1@sea.gmane.org>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
	<E1DD53D0-168D-4DC1-BF76-E7A3DE64F359@PageDNA.com>
	<ca471dc20708111029n5722ff04h824eb8788a4c824e@mail.gmail.com>
	<79990c6b0708120958p588aabd1ic6dadf2f65de86d3@mail.gmail.com>
	<f9nj86$l8b$1@sea.gmane.org>
Message-ID: <79990c6b0708121212m2490d6f0tb151c3c1d5aa1ea3@mail.gmail.com>

On 12/08/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Note that Python does nothing special in the above case. For non-Windows
> platforms, you'd get two different results -- the conversion from \r\n to
> \n is done by the Windows C runtime since the default open() mode is text mode.
>
> Only with mode 'U' does Python use its own universal newline mode.

Pah. You're right - I almost used 'U' and then "discovered" that I
didn't need it (and got bitten by a portability bug as a result :-()

> With Python 3.0, the C library is not used and Python uses universal newline
> mode by default.

That's what I expected, but I was surprised to find that the PEP is
pretty unclear on this. The phrase "universal newlines" is mentioned
only once, and never defined. Knowing the meaning, I can see how the
PEP is intended to say that universal newlines on input is the default
(and you set the newline argument to specify a *specific*,
non-universal, newline value) - but I missed it on first reading.

Thanks for the clarification.
Paul.

From skip at pobox.com  Mon Aug 13 18:55:26 2007
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 13 Aug 2007 11:55:26 -0500
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
 3.0
In-Reply-To: <79990c6b0708120958p588aabd1ic6dadf2f65de86d3@mail.gmail.com>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
	<E1DD53D0-168D-4DC1-BF76-E7A3DE64F359@PageDNA.com>
	<ca471dc20708111029n5722ff04h824eb8788a4c824e@mail.gmail.com>
	<79990c6b0708120958p588aabd1ic6dadf2f65de86d3@mail.gmail.com>
Message-ID: <18112.36094.979628.85609@montanaro.dyndns.org>


    Paul> ...  that files can have *either* bare \n, *or* the combination
    Paul> \r\n, to delimit lines.

As someone else pointed out, \r needs to be supported as well.  Many Mac
applications (Excel comes to mind) still emit text files with \r as the line
terminator.

Skip

From guido at python.org  Mon Aug 13 19:25:41 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 13 Aug 2007 10:25:41 -0700
Subject: [Python-Dev] Python 3000 Sprint @ Google
Message-ID: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>

It's official! The second annual Python Sprint @ Google is happening
again: August 22-25 (Wed-Sat).  We're sprinting at two locations, this
time Google headquarters in Mountain View and the Google office in
Chicago (thanks to Brian Fitzpatrick). We'll connect the two sprints
with full-screen videoconferencing. The event is *free* and includes
Google's *free gourmet food*.

Anyone with a reasonable Python experience is invited to attend. The
primary goal is to work on Python 3000, to polish off the first alpha
release; other ideas are welcome too. Experienced Python core
developers will be available for mentoring. (The goal is not to learn
Python; it is to learn *contributing* to Python.)

For more information and to sign up, please see the wiki page on python.org:
http://wiki.python.org/moin/GoogleSprint

Sign-up via the wiki page is strongly recommended to avoid lines
getting badges. Please read the whole wiki page to make sure you're
prepared.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From trentm at activestate.com  Mon Aug 13 19:37:31 2007
From: trentm at activestate.com (Trent Mick)
Date: Mon, 13 Aug 2007 10:37:31 -0700
Subject: [Python-Dev] Dropping support for Win9x
In-Reply-To: <46BEE0F3.7070904@v.loewis.de>
References: <46BEE0F3.7070904@v.loewis.de>
Message-ID: <46C096DB.9070608@activestate.com>

Martin v. L?wis wrote:
> I'd like to remove support for Windows 9x (95, 98(SE), ME)
> soon from the Python trunk. This would primarily affect all
> wide-string APIs (which would be considered present
> unconditionally), as well as certain "new" Win32 functions;
> in this cleanup, I would also drop support for NT+ before
> Windows 2000 (i.e. NT 3.1, 3.5(1), 4.0).
> 
> I'm not sure whether w9xpopen should be left in place; Tim
> suggested it should, however, I don't see why (something
> with alternative command line interpreters IIRC).

I'm not entirely sure, but I think that w9xpopen gets used when it looks 
like the shell (as per %ComSpec%) is command.com rather than cmd.exe. My 
understanding is that a Win9x machine *upgraded* (rather than a clean 
re-install) to Win2k or XP will retain command.com as the %ComSpec% setting.

If so, that *might* be sufficient reason to keep w9xpopen around. I 
don't have a strong opinion though: I'm all for dropping win9x support 
and would be happy with either a doc note that users need to ensure they 
aren't using command.com, or a warning in the installer if this is detected.

Trent

-- 
Trent Mick
trentm at activestate.com

From rowen at cesmail.net  Mon Aug 13 19:46:08 2007
From: rowen at cesmail.net (Russell E Owen)
Date: Mon, 13 Aug 2007 10:46:08 -0700
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
	<87wsw3p5em.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <rowen-D017E8.10460713082007@sea.gmane.org>

In article <87wsw3p5em.fsf at uwakimon.sk.tsukuba.ac.jp>,
 "Stephen J. Turnbull" <stephen at xemacs.org> wrote:

> Guido van Rossum writes:
> 
>  > However, the old universal newlines feature also set an attibute named
>  > 'newlines' on the file object to a tuple of up to three elements
>  > giving the actual line endings that were observed on the file so far
>  > (\r, \n, or \r\n). This feature is not in PEP 3116, and it is not
>  > implemented. I'm tempted to kill it. Does anyone have a use case for
>  > this?
> 
> I have run into files that intentionally have more than one newline
> convention used (mbox and Babyl mail folders, with messages received
> from various platforms).  However, most of the time multiple newline
> conventions is a sign that the file is either corrupt or isn't text.
> If so, then saving the file may corrupt it.  The newlines attribute
> could be used to check for this condition.

There is at least one Mac source code editor (SubEthaEdit) that is all 
too happy to add one kind of newline to a file that started out with a 
different line ending character. As a result I have seen a fair number 
of text files with mixed line endings. I don't see as many these days, 
though; perhaps because the current version of SubEthaEdit handles 
things a bit better. So perhaps it won't matter much for Python 3000.

-- Russell


From guido at python.org  Mon Aug 13 22:15:03 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 13 Aug 2007 13:15:03 -0700
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
In-Reply-To: <rowen-D017E8.10460713082007@sea.gmane.org>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
	<87wsw3p5em.fsf@uwakimon.sk.tsukuba.ac.jp>
	<rowen-D017E8.10460713082007@sea.gmane.org>
Message-ID: <ca471dc20708131315i6dba68d6u9efae161d1b647ca@mail.gmail.com>

On 8/13/07, Russell E Owen <rowen at cesmail.net> wrote:
> In article <87wsw3p5em.fsf at uwakimon.sk.tsukuba.ac.jp>,
>  "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
>
> > Guido van Rossum writes:
> >
> >  > However, the old universal newlines feature also set an attibute named
> >  > 'newlines' on the file object to a tuple of up to three elements
> >  > giving the actual line endings that were observed on the file so far
> >  > (\r, \n, or \r\n). This feature is not in PEP 3116, and it is not
> >  > implemented. I'm tempted to kill it. Does anyone have a use case for
> >  > this?
> >
> > I have run into files that intentionally have more than one newline
> > convention used (mbox and Babyl mail folders, with messages received
> > from various platforms).  However, most of the time multiple newline
> > conventions is a sign that the file is either corrupt or isn't text.
> > If so, then saving the file may corrupt it.  The newlines attribute
> > could be used to check for this condition.
>
> There is at least one Mac source code editor (SubEthaEdit) that is all
> too happy to add one kind of newline to a file that started out with a
> different line ending character. As a result I have seen a fair number
> of text files with mixed line endings. I don't see as many these days,
> though; perhaps because the current version of SubEthaEdit handles
> things a bit better. So perhaps it won't matter much for Python 3000.

I've seen similar behavior in MS VC++ (long ago, dunno what it does
these days). It would read files with \r\n and \n line endings, and
whenever you edited a line, that line also got a \r\n ending. But
unchanged lines that started out with \n-only endings would keep the
\n only. And there was no way for the end user to see or control this.

To emulate this behavior in Python you'd have to read the file in
binary mode *or* we'd have to have an additional flag specifying to
return line endings as encountered in the file. The newlines attribute
(as defined in 2.x) doesn't help, because it doesn't tell which lines
used which line ending. I think the newline feature in PEP 3116 falls
short too; it seems mostly there to override the line ending *written*
(from the default os.sep).

I think we may need different flags for input and for output.

For input, we'd need two things: (a) which are acceptable line
endings; (b) whether to translate acceptable line endings to \n or
not. For output, we need two things again: (c) whether to translate
line endings at all; (d) which line endings to translate. I guess we
could map (c) to (b) and (d) to (a) for a signature that's the same
for input and output (and makes sense for read+write files as well).
The default would be (a)=={'\n', '\r\n', '\r'} and (b)==True.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From gh at ghaering.de  Mon Aug 13 23:25:14 2007
From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=)
Date: Mon, 13 Aug 2007 23:25:14 +0200
Subject: [Python-Dev] Python 3000: confused about str8, str, bytes
Message-ID: <46C0CC3A.70809@ghaering.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I maintain the sqlite module in the standard library, which makes heavy use
of PyString_* C API. Now I've made it work under Python 3000 insofar as
tests pass, but the new Python string semantics mean I have more work to do
here and make some API choices.

I've read in another thread that the future of str8 is not decided yet. To
be honest I was confused when I saw it first, it's documented nowhere as
far as I can see.

Is that decided yet? Is str8 going away?

What will happen with the Python C API? Will PyString_* become what
PyUnicode_* is in Python 2.x?

- -- Gerhard
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGwMw6dIO4ozGCH14RAoyEAJ0eoqZ8gSqKh5/HIXxhbG5xpMedLgCgquQV
Qv+CGyoD8eSXaoAKzn2WBSM=
=w4HB
-----END PGP SIGNATURE-----

From brett at python.org  Tue Aug 14 00:13:10 2007
From: brett at python.org (Brett Cannon)
Date: Mon, 13 Aug 2007 15:13:10 -0700
Subject: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
In-Reply-To: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
Message-ID: <bbaeab100708131513q22017848y7bc91001886c6e34@mail.gmail.com>

On 8/13/07, Guido van Rossum <guido at python.org> wrote:
> It's official! The second annual Python Sprint @ Google is happening
> again: August 22-25 (Wed-Sat).

I can't attend this year (damn doctor's appt.), but I will try to be
on Google Talk (username of bcannon) in case I can help out somehow
remotely.

-Brett

From guido at python.org  Tue Aug 14 01:31:32 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 13 Aug 2007 16:31:32 -0700
Subject: [Python-Dev] Python 3000: confused about str8, str, bytes
In-Reply-To: <46C0CC3A.70809@ghaering.de>
References: <46C0CC3A.70809@ghaering.de>
Message-ID: <ca471dc20708131631j70e2037bo319bd18f7d87072c@mail.gmail.com>

When I said it wasn't decided I was totally serious. No decision has
been reached.

However, I strongly recommend that you try to write all your code
using PyUnicode and PyBytes, avoiding PyString completely. Even if
str8/PyString will remain in existence, it will be a last resort type
for backwards compatibility.

We *may* at some point rename PyUnicode to PyString (or PyText, to
avoid confusion), but don't count on this. If we do, we'll provide a
tool or service to do the conversion for you.

--Guido

On 8/13/07, Gerhard H?ring <gh at ghaering.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I maintain the sqlite module in the standard library, which makes heavy use
> of PyString_* C API. Now I've made it work under Python 3000 insofar as
> tests pass, but the new Python string semantics mean I have more work to do
> here and make some API choices.
>
> I've read in another thread that the future of str8 is not decided yet. To
> be honest I was confused when I saw it first, it's documented nowhere as
> far as I can see.
>
> Is that decided yet? Is str8 going away?
>
> What will happen with the Python C API? Will PyString_* become what
> PyUnicode_* is in Python 2.x?
>
> - -- Gerhard
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFGwMw6dIO4ozGCH14RAoyEAJ0eoqZ8gSqKh5/HIXxhbG5xpMedLgCgquQV
> Qv+CGyoD8eSXaoAKzn2WBSM=
> =w4HB
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From barry at python.org  Tue Aug 14 15:58:32 2007
From: barry at python.org (Barry Warsaw)
Date: Tue, 14 Aug 2007 09:58:32 -0400
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
In-Reply-To: <ca471dc20708131315i6dba68d6u9efae161d1b647ca@mail.gmail.com>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
	<87wsw3p5em.fsf@uwakimon.sk.tsukuba.ac.jp>
	<rowen-D017E8.10460713082007@sea.gmane.org>
	<ca471dc20708131315i6dba68d6u9efae161d1b647ca@mail.gmail.com>
Message-ID: <FAEFC901-BAE1-4D56-8FEE-6098E307DAF2@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Aug 13, 2007, at 4:15 PM, Guido van Rossum wrote:

> I've seen similar behavior in MS VC++ (long ago, dunno what it does
> these days). It would read files with \r\n and \n line endings, and
> whenever you edited a line, that line also got a \r\n ending. But
> unchanged lines that started out with \n-only endings would keep the
> \n only. And there was no way for the end user to see or control this.
>
> To emulate this behavior in Python you'd have to read the file in
> binary mode *or* we'd have to have an additional flag specifying to
> return line endings as encountered in the file. The newlines attribute
> (as defined in 2.x) doesn't help, because it doesn't tell which lines
> used which line ending. I think the newline feature in PEP 3116 falls
> short too; it seems mostly there to override the line ending *written*
> (from the default os.sep).
>
> I think we may need different flags for input and for output.
>
> For input, we'd need two things: (a) which are acceptable line
> endings; (b) whether to translate acceptable line endings to \n or
> not. For output, we need two things again: (c) whether to translate
> line endings at all; (d) which line endings to translate. I guess we
> could map (c) to (b) and (d) to (a) for a signature that's the same
> for input and output (and makes sense for read+write files as well).
> The default would be (a)=={'\n', '\r\n', '\r'} and (b)==True.

I haven't thought about the output side of the equation, but I've  
already hit a situation where I'd like to see the input side (b)  
option implemented.

I'm still sussing out the email package changes (down to 7F/9E of 247  
tests!) but in trying to fix things I found myself wanting to open  
files in text mode so that I got strings out of the file instead of  
bytes.  This was all fine except that some of the tests started  
failing because of the EOL translation that happens unconditionally  
now.   The file contained \r\n and the test was ensuring these EOLs  
were preserved in the parsed text.  I switched back to opening the  
file in binary mode, and doing a crufty conversion of bytes to  
strings (which I suspect is error prone but gets me farther along).

It would have been perfect, I think, if I could have opened the file  
in text mode so that read() gave me strings, with universal newlines  
and preservation of line endings (i.e. no translation to \n).

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)

iQCVAwUBRsG1CXEjvBPtnXfVAQKF3AP/X+/E44KI2EB3w0i3N5cGBCajJbMV93fk
j2S/lfQf4tjBH3ZFEhUnybcJxsNukYY65T4MdzKh+IgJHV5s0rQtl2Hzr85e7Y0O
i5Z3N4TAKc11PjSIk6vKrkgwPCEMzvwIQ5DFxeQBF5kOF6cZuXKaeDzB6z/GBYNv
YiJEnOeZkW8=
=u6OL
-----END PGP SIGNATURE-----

From alan.mcintyre at gmail.com  Tue Aug 14 20:03:06 2007
From: alan.mcintyre at gmail.com (Alan McIntyre)
Date: Tue, 14 Aug 2007 14:03:06 -0400
Subject: [Python-Dev] SimpleXMLRPCServer failure on G4 OS X
Message-ID: <1d36917a0708141103j493d558crdf7440cdcae2061a@mail.gmail.com>

Hi all,

There are some new tests for xmlrpclib/SimpleXMLRPCServer that fail
only on the G4 OS X buildbot.  Unfortunately, the SimpleXMLRPCServer
returns a vanilla 500 error for any local exceptions, so there's no
obvious (to me) way to find out what's going wrong without having a G4
Mac on hand (which I don't).

Can anybody recommend any way to look into this, or suggest any
G4-specific issues I could check for?

Thanks,
Alan

From lists at cheimes.de  Tue Aug 14 22:46:56 2007
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 14 Aug 2007 22:46:56 +0200
Subject: [Python-Dev] Documentation switch imminent
In-Reply-To: <f9t2nn$ksg$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org>
Message-ID: <46C214C0.3050703@cheimes.de>

Georg Brandl wrote:
> Infos for people who will write docs in the new trees can be found in the
> new "Documenting Python" document, at the moment still available from
> http://pydoc.gbrandl.de:3000/documenting/, especially the "Differences"
> section at http://pydoc.gbrandl.de:3000/documenting/fromlatex/ (which
> is not complete, patches are welcome :)

http://pydoc.gbrandl.de:3000/documenting/fromlatex/ doesn't work for me:

Keyword Not Found

The keyword documenting/fromlatex is not directly associated with a page.


Christian


From brett at python.org  Wed Aug 15 01:57:19 2007
From: brett at python.org (Brett Cannon)
Date: Tue, 14 Aug 2007 16:57:19 -0700
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <f9t2nn$ksg$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org>
Message-ID: <bbaeab100708141657n16a012fbqdceece1dff992a06@mail.gmail.com>

On 8/14/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Now that the converted documentation is fairly bug-free, I want to
> make the switch.
>
> I will replace the old Doc/ trees in the trunk and py3k branches
> tomorrow, moving over the reST ones found at
> svn+ssh://svn.python.org/doctools/Doc-{26,3k}.

First, that address is wrong; missing a 'trunk' in there.

Second, are we going to keep the docs in a separate tree forever, or
is this just for now?  I am not thinking so much about the tools, but
whether we will need to do two separate commits in order to make code
changes *and* change the docs?  Or are you going to add an externals
dependency in the trees to their respective doc directories?

-Brett

From brett at python.org  Wed Aug 15 19:16:10 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 15 Aug 2007 10:16:10 -0700
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <f9v16k$rq5$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org>
	<bbaeab100708141657n16a012fbqdceece1dff992a06@mail.gmail.com>
	<f9v16k$rq5$1@sea.gmane.org>
Message-ID: <bbaeab100708151016o1329d493m393bdb76149c20c0@mail.gmail.com>

On 8/15/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Brett Cannon schrieb:
> > On 8/14/07, Georg Brandl <g.brandl at gmx.net> wrote:
> >> Now that the converted documentation is fairly bug-free, I want to
> >> make the switch.
> >>
> >> I will replace the old Doc/ trees in the trunk and py3k branches
> >> tomorrow, moving over the reST ones found at
> >> svn+ssh://svn.python.org/doctools/Doc-{26,3k}.
> >
> > First, that address is wrong; missing a 'trunk' in there.
>
> Sorry again.
>

Not a problem.  I also noticed, though, that the user (pythondev) is
missing as well.  =)

> > Second, are we going to keep the docs in a separate tree forever, or
> > is this just for now?
>
> They will be moved (in a few minutes...) to the location where the
> Latex docs are now.
>

Yep, just did an update.

> > I am not thinking so much about the tools, but
> > whether we will need to do two separate commits in order to make code
> > changes *and* change the docs?  Or are you going to add an externals
> > dependency in the trees to their respective doc directories?
>
> No separate commits will be needed to commit changes to the docs.
> However, the tool to build the docs will not be in the tree under Doc/,
> but continue to be maintained in the doctools/ toplevel project.
>

OK.

> I spoke with Martin about including them as externals, but we agreed that
> they are not needed and cost too much time on every "svn up".  Instead,
> the Doc/ makefile checks out the tools in a separate directory and runs
> them from there. (The Doc/README.txt file explains this in more detail.)

Seems simple enough!  Thanks again for doing this, Georg (and the doc SIG)!

-Brett

From martin at v.loewis.de  Wed Aug 15 19:51:02 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 15 Aug 2007 19:51:02 +0200
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <f9v2rd$2dl$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org> <f9v2rd$2dl$1@sea.gmane.org>
Message-ID: <46C33D06.9030607@v.loewis.de>

> Okay, I made the switch.  I tagged the state of both Python branches
> before the switch as tags/py{26,3k}-before-rstdocs/.

Update instructions:

1. svn diff Doc; any pending changes will need to be redone
2. svn up; this will remove the tex sources, and then likely
   fail if there were still other files present in Doc, e.g.
   from building the documentation
3. review any files left in Doc
4. rm -rf Doc
5. svn up

If you are certain there is nothing of interest in your sandbox
copy of Doc, you can start with step 4.

Regards,
Martin

From brett at python.org  Wed Aug 15 23:40:22 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 15 Aug 2007 14:40:22 -0700
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <46C33D06.9030607@v.loewis.de>
References: <f9t2nn$ksg$1@sea.gmane.org> <f9v2rd$2dl$1@sea.gmane.org>
	<46C33D06.9030607@v.loewis.de>
Message-ID: <bbaeab100708151440u7df9e6fco8cc611ae98a2ae95@mail.gmail.com>

On 8/15/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > Okay, I made the switch.  I tagged the state of both Python branches
> > before the switch as tags/py{26,3k}-before-rstdocs/.
>
> Update instructions:
>
> 1. svn diff Doc; any pending changes will need to be redone
> 2. svn up; this will remove the tex sources, and then likely
>    fail if there were still other files present in Doc, e.g.
>    from building the documentation
> 3. review any files left in Doc
> 4. rm -rf Doc
> 5. svn up
>
> If you are certain there is nothing of interest in your sandbox
> copy of Doc, you can start with step 4.

Why the 'rm' call?  When I did ``svn update`` it deleted the files for
me.  Is this to ditch some metadata?

-Brett

From martin at v.loewis.de  Wed Aug 15 23:54:01 2007
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 15 Aug 2007 23:54:01 +0200
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <bbaeab100708151440u7df9e6fco8cc611ae98a2ae95@mail.gmail.com>
References: <f9t2nn$ksg$1@sea.gmane.org> <f9v2rd$2dl$1@sea.gmane.org>	
	<46C33D06.9030607@v.loewis.de>
	<bbaeab100708151440u7df9e6fco8cc611ae98a2ae95@mail.gmail.com>
Message-ID: <46C375F9.3020908@v.loewis.de>

>> 1. svn diff Doc; any pending changes will need to be redone
>> 2. svn up; this will remove the tex sources, and then likely
>>    fail if there were still other files present in Doc, e.g.
>>    from building the documentation
>> 3. review any files left in Doc
>> 4. rm -rf Doc
>> 5. svn up
>>
>> If you are certain there is nothing of interest in your sandbox
>> copy of Doc, you can start with step 4.
> 
> Why the 'rm' call?  When I did ``svn update`` it deleted the files for
> me.  Is this to ditch some metadata?

No, it's to delete any files in this tree not under version control,
see step 2. If you had any such files, step 2 would abort with an
error message

svn: Konnte Verzeichnis ?Doc? nicht hinzuf?gen: ein Objekt mit demselben
Namen existiert bereits

(or some such)

Regards,
Martin

From janssen at parc.com  Thu Aug 16 03:29:37 2007
From: janssen at parc.com (Bill Janssen)
Date: Wed, 15 Aug 2007 18:29:37 PDT
Subject: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
In-Reply-To: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
Message-ID: <07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>

I'd really like an excuse to implement server-side SSL support one of
these days.  Could that be a sprint activity?  Probably against 2.6 (I
doubt the Modules/_ssl.c file will change much for 3K).

The idea is that if you call socket.ssl() on a socket that's bound to
an address, the socket is assumed to be server-side, the cert passed
in is assumed to be a server-side cert, and the SSLObject returned has
a couple of extra methods, listen() and accept().  Calling accept() does
the SSL dance with the remote side, and returns an SSLObject.

Does this need a PEP?

Bill

From guido at python.org  Thu Aug 16 04:03:31 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 15 Aug 2007 19:03:31 -0700
Subject: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
In-Reply-To: <2235611917962225454@unknownmsgid>
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<2235611917962225454@unknownmsgid>
Message-ID: <ca471dc20708151903l9986620v70fa26037f123cd3@mail.gmail.com>

Sounds like a good plan. I'm not a great coach though since I didn't
write _ssl.c and I've never used openssl directly. But I can help you
with the Python stuff of course!

--Guido

On 8/15/07, Bill Janssen <janssen at parc.com> wrote:
> I'd really like an excuse to implement server-side SSL support one of
> these days.  Could that be a sprint activity?  Probably against 2.6 (I
> doubt the Modules/_ssl.c file will change much for 3K).
>
> The idea is that if you call socket.ssl() on a socket that's bound to
> an address, the socket is assumed to be server-side, the cert passed
> in is assumed to be a server-side cert, and the SSLObject returned has
> a couple of extra methods, listen() and accept().  Calling accept() does
> the SSL dance with the remote side, and returns an SSLObject.
>
> Does this need a PEP?
>
> Bill
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From janssen at parc.com  Thu Aug 16 04:45:55 2007
From: janssen at parc.com (Bill Janssen)
Date: Wed, 15 Aug 2007 19:45:55 PDT
Subject: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
In-Reply-To: <ca471dc20708151903l9986620v70fa26037f123cd3@mail.gmail.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<2235611917962225454@unknownmsgid>
	<ca471dc20708151903l9986620v70fa26037f123cd3@mail.gmail.com>
Message-ID: <07Aug15.194559pdt."57996"@synergy1.parc.xerox.com>

> Sounds like a good plan. I'm not a great coach though since I didn't
> write _ssl.c and I've never used openssl directly. But I can help you
> with the Python stuff of course!

Thanks (though I think I can handle the Python end of it, too :-).

It's been a while since I wrote any Python C code, though -- are there
better tools these days for debugging reference counting?  Anyone know?

Bill


From nnorwitz at gmail.com  Thu Aug 16 05:13:04 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 15 Aug 2007 20:13:04 -0700
Subject: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
In-Reply-To: <-8815054654740213484@unknownmsgid>
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<2235611917962225454@unknownmsgid>
	<ca471dc20708151903l9986620v70fa26037f123cd3@mail.gmail.com>
	<-8815054654740213484@unknownmsgid>
Message-ID: <ee2a432c0708152013x69987579wb67d5f88dbfd9208@mail.gmail.com>

On 8/15/07, Bill Janssen <janssen at parc.com> wrote:
> > Sounds like a good plan. I'm not a great coach though since I didn't
> > write _ssl.c and I've never used openssl directly. But I can help you
> > with the Python stuff of course!
>
> Thanks (though I think I can handle the Python end of it, too :-).
>
> It's been a while since I wrote any Python C code, though -- are there
> better tools these days for debugging reference counting?  Anyone know?

The way I typically do it is to configure --with-pydebug.  That shows
the ref count in the interpreter and allows running tests with the -R
flag to regrtest.  When regrtest reports leaks, narrow down the
(Python) code which causes a leak using bisection, find the C code
which corresponds, and visually inspect the C code.  Most leaks are
pretty obvious this way.  With good tests, this doesn't take much
time.

For pure memory leaks, valgrind works pretty well.

n

From foom at fuhm.net  Thu Aug 16 06:06:32 2007
From: foom at fuhm.net (James Y Knight)
Date: Thu, 16 Aug 2007 00:06:32 -0400
Subject: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
In-Reply-To: <07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <118A99D6-8BBB-47A2-A2F1-36C82931C610@fuhm.net>


On Aug 15, 2007, at 9:29 PM, Bill Janssen wrote:

> I'd really like an excuse to implement server-side SSL support one of
> these days.  Could that be a sprint activity?  Probably against 2.6 (I
> doubt the Modules/_ssl.c file will change much for 3K).
>
> The idea is that if you call socket.ssl() on a socket that's bound to
> an address, the socket is assumed to be server-side, the cert passed
> in is assumed to be a server-side cert, and the SSLObject returned has
> a couple of extra methods, listen() and accept().  Calling accept()  
> does
> the SSL dance with the remote side, and returns an SSLObject.
>
> Does this need a PEP?

Maybe one of the three existing Python/SSL libraries should be stdlib- 
ified instead of starting another new one from scratch?

Just a thought...

James


From nnorwitz at gmail.com  Thu Aug 16 07:07:05 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 15 Aug 2007 22:07:05 -0700
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <f9v2rd$2dl$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org> <f9v2rd$2dl$1@sea.gmane.org>
Message-ID: <ee2a432c0708152207j66c26fbdu6e9bcec6dae0ccf5@mail.gmail.com>

On 8/15/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Georg Brandl schrieb:
> >
> > Neal will change his build scripts, so that the 2.6 and 3.0 devel
> > documentation pages at docs.python.org will be built from these new
> > trees soon.
>
> Okay, I made the switch.  I tagged the state of both Python branches
> before the switch as tags/py{26,3k}-before-rstdocs/.

http://docs.python.org/dev/
http://docs.python.org/dev/3.0/

The upgrade went smoothly.  Below are all the issues I noticed.  I had
to install a version of python 2.5 since that is a minimum
requirement.  I had to change from a plain 'make' in the Doc directory
to 'make html'.  The output is in build/html rather than html/ now.

2.6 output:
trying to load pickled env... failed: [Errno 2] No such file or
directory: 'build/doctrees/environment.pickle'

writing output...
... library/contextlib.rst<string>:3: Warning: 'with' will become a
reserved keyword in Python 2.6
tutorial/errors.rst<string>:1: Warning: 'with' will become a reserved
keyword in Python

3.0 output:
Traceback (most recent call last):
  File "tools/sphinx-build.py", line 13, in <module>
    from sphinx import main
  File "/home/neal/python/py3k/Doc/tools/sphinx/__init__.py", line 16,
in <module>
    from .builder import builders
  File "/home/neal/python/py3k/Doc/tools/sphinx/builder.py", line 35,
in <module>
    from .environment import BuildEnvironment
  File "/home/neal/python/py3k/Doc/tools/sphinx/environment.py", line
34, in <module>
    from docutils.parsers.rst.states import Body
  File "/home/neal/python/py3k/Doc/tools/docutils/parsers/rst/__init__.py",
line 77, in <module>
    from docutils.parsers.rst import states
  File "/home/neal/python/py3k/Doc/tools/docutils/parsers/rst/states.py",
line 110, in <module>
    import roman
ImportError: No module named roman

After this error, I just linked my tools directory to the one in 2.6
(trunk) and that worked. I'm not sure if this will create problems in
the future.

trying to load pickled env... failed: [Errno 2] No such file or
directory: 'build/doctrees/environment.pickle'

writing output...
... library/contextlib.rst<string>:3: Warning: 'with' will become a
reserved keyword in Python 2.6
library/shutil.rst<string>:17: Warning: 'as' will become a reserved
keyword in Python 2.6
library/subprocess.rst<string>:7: Warning: 'as' will become a reserved
keyword in Python 2.6
tutorial/errors.rst<string>:1: Warning: 'with' will become a reserved
keyword in Python 2.6

I realize none of these are a big deal.  However, it would be nice if
it was cleaned up so that people unfamiliar with building the docs
aren't surprised.

n

From barry at python.org  Thu Aug 16 06:40:33 2007
From: barry at python.org (Barry Warsaw)
Date: Thu, 16 Aug 2007 00:40:33 -0400
Subject: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
In-Reply-To: <07Aug15.194559pdt."57996"@synergy1.parc.xerox.com>
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<2235611917962225454@unknownmsgid>
	<ca471dc20708151903l9986620v70fa26037f123cd3@mail.gmail.com>
	<07Aug15.194559pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <888D07AF-6168-4140-809A-217B3E43408B@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Aug 15, 2007, at 10:45 PM, Bill Janssen wrote:
>
> It's been a while since I wrote any Python C code, though -- are there
> better tools these days for debugging reference counting?  Anyone  
> know?

No, but /that/ would make an awesome sprint topic <wink>.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)

iQCVAwUBRsPVQnEjvBPtnXfVAQIp9QP/Y9DCqvBdbDdVvTQp8gt4so2HW/AqRyZU
IF3SI/rrzMneslZRbU9PBlKbhq7oE/zwThpPss73W+64CoF2Z7N2dEGJJZncp+RK
bo1jyzG2bituz1ZqXRFW8t373XAWLrMusABXNAD5Ypfd1PfbmziFaa6ttyu2jl5O
4QWxPaw4qU0=
=+Zfi
-----END PGP SIGNATURE-----

From g.brandl at gmx.net  Thu Aug 16 08:40:24 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 16 Aug 2007 08:40:24 +0200
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <ee2a432c0708152207j66c26fbdu6e9bcec6dae0ccf5@mail.gmail.com>
References: <f9t2nn$ksg$1@sea.gmane.org> <f9v2rd$2dl$1@sea.gmane.org>
	<ee2a432c0708152207j66c26fbdu6e9bcec6dae0ccf5@mail.gmail.com>
Message-ID: <fa0rgk$9v6$1@sea.gmane.org>

Neal Norwitz schrieb:
> On 8/15/07, Georg Brandl <g.brandl at gmx.net> wrote:
>> Georg Brandl schrieb:
>> >
>> > Neal will change his build scripts, so that the 2.6 and 3.0 devel
>> > documentation pages at docs.python.org will be built from these new
>> > trees soon.
>>
>> Okay, I made the switch.  I tagged the state of both Python branches
>> before the switch as tags/py{26,3k}-before-rstdocs/.
> 
> http://docs.python.org/dev/
> http://docs.python.org/dev/3.0/

Great!

> 3.0 output:
> Traceback (most recent call last):
>   File "tools/sphinx-build.py", line 13, in <module>
>     from sphinx import main
>   File "/home/neal/python/py3k/Doc/tools/sphinx/__init__.py", line 16,
> in <module>
>     from .builder import builders
>   File "/home/neal/python/py3k/Doc/tools/sphinx/builder.py", line 35,
> in <module>
>     from .environment import BuildEnvironment
>   File "/home/neal/python/py3k/Doc/tools/sphinx/environment.py", line
> 34, in <module>
>     from docutils.parsers.rst.states import Body
>   File "/home/neal/python/py3k/Doc/tools/docutils/parsers/rst/__init__.py",
> line 77, in <module>
>     from docutils.parsers.rst import states
>   File "/home/neal/python/py3k/Doc/tools/docutils/parsers/rst/states.py",
> line 110, in <module>
>     import roman
> ImportError: No module named roman
> 
> After this error, I just linked my tools directory to the one in 2.6
> (trunk) and that worked. I'm not sure if this will create problems in
> the future.

No, it shouldn't. I added roman.py in trunk, but didn't
touch py3k since I don't want to disturb svnmerge more than necessary.

> trying to load pickled env... failed: [Errno 2] No such file or
> directory: 'build/doctrees/environment.pickle'

That is expected.

> writing output...
> ... library/contextlib.rst<string>:3: Warning: 'with' will become a
> reserved keyword in Python 2.6
> library/shutil.rst<string>:17: Warning: 'as' will become a reserved
> keyword in Python 2.6
> library/subprocess.rst<string>:7: Warning: 'as' will become a reserved
> keyword in Python 2.6
> tutorial/errors.rst<string>:1: Warning: 'with' will become a reserved
> keyword in Python 2.6
> 
> I realize none of these are a big deal.  However, it would be nice if
> it was cleaned up so that people unfamiliar with building the docs
> aren't surprised.

I'll have the with/as problem fixed soon, it should be nothing more than
setting the future flag for the call to compile().

Thanks,
Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From janssen at parc.com  Thu Aug 16 17:52:02 2007
From: janssen at parc.com (Bill Janssen)
Date: Thu, 16 Aug 2007 08:52:02 PDT
Subject: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
In-Reply-To: <118A99D6-8BBB-47A2-A2F1-36C82931C610@fuhm.net> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<118A99D6-8BBB-47A2-A2F1-36C82931C610@fuhm.net>
Message-ID: <07Aug16.085210pdt."57996"@synergy1.parc.xerox.com>

> Maybe one of the three existing Python/SSL libraries should be stdlib- 
> ified instead of starting another new one from scratch?

Yep, that's my intent.  This should just be a change to _ssl.c.

Bill

From janssen at parc.com  Thu Aug 16 17:52:35 2007
From: janssen at parc.com (Bill Janssen)
Date: Thu, 16 Aug 2007 08:52:35 PDT
Subject: [Python-Dev] [Python-3000] Python 3000 Sprint @ Google
In-Reply-To: <888D07AF-6168-4140-809A-217B3E43408B@python.org> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<2235611917962225454@unknownmsgid>
	<ca471dc20708151903l9986620v70fa26037f123cd3@mail.gmail.com>
	<07Aug15.194559pdt."57996"@synergy1.parc.xerox.com>
	<888D07AF-6168-4140-809A-217B3E43408B@python.org>
Message-ID: <07Aug16.085238pdt."57996"@synergy1.parc.xerox.com>

Barry Warsaw suggested:
> > It's been a while since I wrote any Python C code, though -- are there
> > better tools these days for debugging reference counting?  Anyone  
> > know?
> 
> No, but /that/ would make an awesome sprint topic <wink>.

Indeed!

Bill

From alexandre at peadrop.com  Fri Aug 17 01:43:10 2007
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Thu, 16 Aug 2007 19:43:10 -0400
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <ee2a432c0708152207j66c26fbdu6e9bcec6dae0ccf5@mail.gmail.com>
References: <f9t2nn$ksg$1@sea.gmane.org> <f9v2rd$2dl$1@sea.gmane.org>
	<ee2a432c0708152207j66c26fbdu6e9bcec6dae0ccf5@mail.gmail.com>
Message-ID: <acd65fa20708161643s6d7c0c42x135eb0957980aa14@mail.gmail.com>

On 8/16/07, Neal Norwitz <nnorwitz at gmail.com> wrote:
> On 8/15/07, Georg Brandl <g.brandl at gmx.net> wrote:
> > Okay, I made the switch.  I tagged the state of both Python branches
> > before the switch as tags/py{26,3k}-before-rstdocs/.
>
> http://docs.python.org/dev/
> http://docs.python.org/dev/3.0/
>

Is it just me, or the markup of the new docs is quite heavy?

alex% wget -q -O- http://docs.python.org/api/genindex.html | wc -c
77868
alex% wget -q -O- http://docs.python.org/dev/3.0/genindex.html | wc -c
918359

Firefox, on my fairly recent machine, takes ~5 seconds rendering the
index of the new docs from disk, compared to a fraction of a second
for the old one.

-- Alexandre

From dasn at lavabit.com  Fri Aug 17 08:41:55 2007
From: dasn at lavabit.com (Dasn)
Date: Fri, 17 Aug 2007 14:41:55 +0800
Subject: [Python-Dev] mailbox._create_temporary without checking the file
	permission
Message-ID: <20070817064155.GA5550@lavabit.com>

Hi, guys. 

_create_temporary is not tracking the perm bits of the original mbox.

$ ls -l me
-rw-------  1 dasn  users  274886 Aug 16 08:43 me
$ python
Python 2.5.1 (r251:54863,  May  8 2007,  07:32:21)
[GCC 3.3.5 (propolice)] on openbsd4
Type "help",  "copyright",  "credits" or "license" for more information.
>>> from mailbox import mbox
>>> m=mbox('me')
>>> m.pop(0)
<mailbox.mboxMessage instance at 0x84865ecc>
>>> m.flush()
>>>^D
$ ls -l me
-rwxr-xr-x  1 dasn  users  268438 Aug 16 09:26 me*
$

-- 
Dasn


From dasn at lavabit.com  Fri Aug 17 11:33:19 2007
From: dasn at lavabit.com (Dasn)
Date: Fri, 17 Aug 2007 17:33:19 +0800
Subject: [Python-Dev] mailbox._create_temporary without checking the
	file permission
In-Reply-To: <20070817064155.GA5550@lavabit.com>
References: <20070817064155.GA5550@lavabit.com>
Message-ID: <20070817093319.GA26162@lavabit.com>

On 17/08/07 14:41 +0800, Dasn wrote:
>Hi, guys. 
>
>_create_temporary is not tracking the perm bits of the original mbox.
>
>$ ls -l me
>-rw-------  1 dasn  users  274886 Aug 16 08:43 me
>$ python
>Python 2.5.1 (r251:54863,  May  8 2007,  07:32:21)
>[GCC 3.3.5 (propolice)] on openbsd4
>Type "help",  "copyright",  "credits" or "license" for more information.
>>>> from mailbox import mbox
>>>> m=mbox('me')
>>>> m.pop(0)
><mailbox.mboxMessage instance at 0x84865ecc>
>>>> m.flush()
>>>>^D
>$ ls -l me
>-rwxr-xr-x  1 dasn  users  268438 Aug 16 09:26 me*
>$
>

I think there are 2 problems should be considered in the _create_temporary:
 1. what to do if we have no write permission to the directory (e.g.
/var/mail/), what about using tempfile module?
 2. keep the temp file as the same mode as the original one.

-- 
Dasn


From ncoghlan at gmail.com  Fri Aug 17 16:00:25 2007
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 18 Aug 2007 00:00:25 +1000
Subject: [Python-Dev] [Python-3000]   Documentation switch imminent
In-Reply-To: <fa3egl$56d$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org>	<f9v2rd$2dl$1@sea.gmane.org>	<ee2a432c0708152207j66c26fbdu6e9bcec6dae0ccf5@mail.gmail.com>	<acd65fa20708161643s6d7c0c42x135eb0957980aa14@mail.gmail.com>
	<fa3egl$56d$1@sea.gmane.org>
Message-ID: <46C5A9F9.1070002@gmail.com>

Georg Brandl wrote:
>> Firefox, on my fairly recent machine, takes ~5 seconds rendering the
>> index of the new docs from disk, compared to a fraction of a second
>> for the old one.
> 
> But you're right that rendering is slow there.  It may be caused by the
> more complicated CSS... perhaps the index should be split up in several
> pages.

Splitting out the C API index would probably be a reasonable start. (It 
may also be worth considering ignoring a leading Py or _Py in that index 
- many of the C API index entries end up under just two index groups).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From skip at pobox.com  Fri Aug 17 16:18:59 2007
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 17 Aug 2007 09:18:59 -0500
Subject: [Python-Dev] Source file markers for Tru64?
Message-ID: <18117.44627.786049.522772@montanaro.dyndns.org>

I'm working on removing Tru64 support from the Python 3 code base.  There
aren't that many markers which actually say "Tru64".  I see __digitial__ and
OSF/1 as well.  What others are there?

Thx,

Skip

From alexandre at peadrop.com  Fri Aug 17 18:28:43 2007
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Fri, 17 Aug 2007 12:28:43 -0400
Subject: [Python-Dev] [Python-3000]  Documentation switch imminent
In-Reply-To: <fa3egl$56d$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org> <f9v2rd$2dl$1@sea.gmane.org>
	<ee2a432c0708152207j66c26fbdu6e9bcec6dae0ccf5@mail.gmail.com>
	<acd65fa20708161643s6d7c0c42x135eb0957980aa14@mail.gmail.com>
	<fa3egl$56d$1@sea.gmane.org>
Message-ID: <acd65fa20708170928y6dc91623tefa112d4fe7528e5@mail.gmail.com>

On 8/17/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Alexandre Vassalotti schrieb:
> > On 8/16/07, Neal Norwitz <nnorwitz at gmail.com> wrote:
> >> On 8/15/07, Georg Brandl <g.brandl at gmx.net> wrote:
> >> > Okay, I made the switch.  I tagged the state of both Python branches
> >> > before the switch as tags/py{26,3k}-before-rstdocs/.
> >>
> >> http://docs.python.org/dev/
> >> http://docs.python.org/dev/3.0/
> >>
> >
> > Is it just me, or the markup of the new docs is quite heavy?
>
> Docutils markup tends to be a bit verbose, yes, but the index is not
> even generated by them.
>
> > alex% wget -q -O- http://docs.python.org/api/genindex.html | wc -c
> > 77868
> > alex% wget -q -O- http://docs.python.org/dev/3.0/genindex.html | wc -c
> > 918359
>
> The new index includes all documents (api, lib, ref, ...), so the ratio
> is more like 678000 : 950000 (using 2.6 here), and the difference can be
> explained quite easily because (a) sphinx uses different anchor names
> ("mailbox.Mailbox.__contains__" vs "l2h-849") and the hrefs have to
> include subdirs like "reference/".

Ah, I didn't notice that index included all the documents. That
explains the huge size increase. However, would it be possible to keep
the indexes separated? I noticed that I find I want more quickly when
the indexes are separated.

> I've now removed leading spaces in the index output, and the character
> count is down to 850000.
>
> > Firefox, on my fairly recent machine, takes ~5 seconds rendering the
> > index of the new docs from disk, compared to a fraction of a second
> > for the old one.
>
> But you're right that rendering is slow there.  It may be caused by the
> more complicated CSS... perhaps the index should be split up in several
> pages.
>

I disabled CSS-support (with View->Page Style->No Style), but it
didn't affect the initial rendering speed. However, scrolling was
*much* faster without CSS.

-- Alexandre

From martin at v.loewis.de  Fri Aug 17 20:02:46 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 17 Aug 2007 20:02:46 +0200
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <18117.44627.786049.522772@montanaro.dyndns.org>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
Message-ID: <46C5E2C6.4070106@v.loewis.de>

skip at pobox.com schrieb:
> I'm working on removing Tru64 support from the Python 3 code base. 

Can you please add it to PEP 11 first? Please also add code that
makes the build break in a version where the code is still in place,
so that users can know that support for their platform is to be
removed (in 3.0, I presume).

> What others are there?

Potentially, code that checks whether the processor is alpha
might also be relevant - but that could also talk about other
systems (Linux or VMS).

Regards,
Martin


From skip at pobox.com  Fri Aug 17 22:20:54 2007
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 17 Aug 2007 15:20:54 -0500
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <46C5E2C6.4070106@v.loewis.de>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
	<46C5E2C6.4070106@v.loewis.de>
Message-ID: <18118.806.945501.14325@montanaro.dyndns.org>


    Martin> skip at pobox.com schrieb:
    >> I'm working on removing Tru64 support from the Python 3 code base.

    Martin> Can you please add it to PEP 11 first?

Sorry, I was going by the Py3kDeprecated wiki page.

    Martin> Please also add code that makes the build break in a version
    Martin> where the code is still in place, so that users can know that
    Martin> support for their platform is to be removed (in 3.0, I presume).

How do I do that?  Does that go in the Python 2.6 code?

    >> What others are there?

    Martin> Potentially, code that checks whether the processor is alpha
    Martin> might also be relevant - but that could also talk about other
    Martin> systems (Linux or VMS).

Skip

From martin at v.loewis.de  Fri Aug 17 23:20:15 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 17 Aug 2007 23:20:15 +0200
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <18118.806.945501.14325@montanaro.dyndns.org>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
	<46C5E2C6.4070106@v.loewis.de>
	<18118.806.945501.14325@montanaro.dyndns.org>
Message-ID: <46C6110F.6050203@v.loewis.de>

>     Martin> Can you please add it to PEP 11 first?
> 
> Sorry, I was going by the Py3kDeprecated wiki page.

Hmm. Who wrote this page, and why? I would hope that a PEP has more
official status than a wiki page.

>     Martin> Please also add code that makes the build break in a version
>     Martin> where the code is still in place, so that users can know that
>     Martin> support for their platform is to be removed (in 3.0, I presume).
> 
> How do I do that?  Does that go in the Python 2.6 code?

If you want to unsupport the system in 2.6, yes. You put something in
configure.in which users can't fail to notice, e.g. a single-line abort
that they need to uncomment before they can proceed. See earlier Python
releases for how this was done (I would have to dig myself to find out
what releases and what systems).

Regards,
Martin

From guido at python.org  Fri Aug 17 23:42:50 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 17 Aug 2007 14:42:50 -0700
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <46C6110F.6050203@v.loewis.de>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
	<46C5E2C6.4070106@v.loewis.de>
	<18118.806.945501.14325@montanaro.dyndns.org>
	<46C6110F.6050203@v.loewis.de>
Message-ID: <ca471dc20708171442i1f3b6d7dl5048743bdfcde9fa@mail.gmail.com>

According to the wiki history, BeOS was added to the wiki page by Skip.

See http://wiki.python.org/moin/Py3kDeprecated?action=info

(rev 6)

On 8/17/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> >     Martin> Can you please add it to PEP 11 first?
> >
> > Sorry, I was going by the Py3kDeprecated wiki page.
>
> Hmm. Who wrote this page, and why? I would hope that a PEP has more
> official status than a wiki page.
>
> >     Martin> Please also add code that makes the build break in a version
> >     Martin> where the code is still in place, so that users can know that
> >     Martin> support for their platform is to be removed (in 3.0, I presume).
> >
> > How do I do that?  Does that go in the Python 2.6 code?
>
> If you want to unsupport the system in 2.6, yes. You put something in
> configure.in which users can't fail to notice, e.g. a single-line abort
> that they need to uncomment before they can proceed. See earlier Python
> releases for how this was done (I would have to dig myself to find out
> what releases and what systems).
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From steve at holdenweb.com  Sat Aug 18 00:58:44 2007
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 17 Aug 2007 18:58:44 -0400
Subject: [Python-Dev] Documentation switch imminent
In-Reply-To: <acd65fa20708170928y6dc91623tefa112d4fe7528e5@mail.gmail.com>
References: <f9t2nn$ksg$1@sea.gmane.org>
	<f9v2rd$2dl$1@sea.gmane.org>	<ee2a432c0708152207j66c26fbdu6e9bcec6dae0ccf5@mail.gmail.com>	<acd65fa20708161643s6d7c0c42x135eb0957980aa14@mail.gmail.com>	<fa3egl$56d$1@sea.gmane.org>
	<acd65fa20708170928y6dc91623tefa112d4fe7528e5@mail.gmail.com>
Message-ID: <46C62824.90002@holdenweb.com>

Alexandre Vassalotti wrote:
> On 8/17/07, Georg Brandl <g.brandl at gmx.net> wrote:
[...]
> Ah, I didn't notice that index included all the documents. That
> explains the huge size increase. However, would it be possible to keep
> the indexes separated? I noticed that I find I want more quickly when
> the indexes are separated.
> 
Which is fine when you know which section to expect to find your content 
in. But let's retain an "all-documentation" index if we can, as this is 
particularly helpful to the newcomers who aren't that familiar with the 
structure of the documentation.

>> I've now removed leading spaces in the index output, and the character
>> count is down to 850000.
>>
>>> Firefox, on my fairly recent machine, takes ~5 seconds rendering the
>>> index of the new docs from disk, compared to a fraction of a second
>>> for the old one.
>> But you're right that rendering is slow there.  It may be caused by the
>> more complicated CSS... perhaps the index should be split up in several
>> pages.
>>
> 
> I disabled CSS-support (with View->Page Style->No Style), but it
> didn't affect the initial rendering speed. However, scrolling was
> *much* faster without CSS.
> 
Probably because the positional calculations are more straightforward then.

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

From steve at holdenweb.com  Sat Aug 18 00:58:44 2007
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 17 Aug 2007 18:58:44 -0400
Subject: [Python-Dev] Documentation switch imminent
In-Reply-To: <acd65fa20708170928y6dc91623tefa112d4fe7528e5@mail.gmail.com>
References: <f9t2nn$ksg$1@sea.gmane.org>
	<f9v2rd$2dl$1@sea.gmane.org>	<ee2a432c0708152207j66c26fbdu6e9bcec6dae0ccf5@mail.gmail.com>	<acd65fa20708161643s6d7c0c42x135eb0957980aa14@mail.gmail.com>	<fa3egl$56d$1@sea.gmane.org>
	<acd65fa20708170928y6dc91623tefa112d4fe7528e5@mail.gmail.com>
Message-ID: <46C62824.90002@holdenweb.com>

Alexandre Vassalotti wrote:
> On 8/17/07, Georg Brandl <g.brandl at gmx.net> wrote:
[...]
> Ah, I didn't notice that index included all the documents. That
> explains the huge size increase. However, would it be possible to keep
> the indexes separated? I noticed that I find I want more quickly when
> the indexes are separated.
> 
Which is fine when you know which section to expect to find your content 
in. But let's retain an "all-documentation" index if we can, as this is 
particularly helpful to the newcomers who aren't that familiar with the 
structure of the documentation.

>> I've now removed leading spaces in the index output, and the character
>> count is down to 850000.
>>
>>> Firefox, on my fairly recent machine, takes ~5 seconds rendering the
>>> index of the new docs from disk, compared to a fraction of a second
>>> for the old one.
>> But you're right that rendering is slow there.  It may be caused by the
>> more complicated CSS... perhaps the index should be split up in several
>> pages.
>>
> 
> I disabled CSS-support (with View->Page Style->No Style), but it
> didn't affect the initial rendering speed. However, scrolling was
> *much* faster without CSS.
> 
Probably because the positional calculations are more straightforward then.

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------


From skip at pobox.com  Sat Aug 18 01:52:44 2007
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 17 Aug 2007 18:52:44 -0500
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <ca471dc20708171442i1f3b6d7dl5048743bdfcde9fa@mail.gmail.com>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
	<46C5E2C6.4070106@v.loewis.de>
	<18118.806.945501.14325@montanaro.dyndns.org>
	<46C6110F.6050203@v.loewis.de>
	<ca471dc20708171442i1f3b6d7dl5048743bdfcde9fa@mail.gmail.com>
Message-ID: <18118.13516.884648.728284@montanaro.dyndns.org>


    Guido> According to the wiki history, BeOS was added to the wiki page by
    Guido> Skip.  See http://wiki.python.org/moin/Py3kDeprecated?action=info

    Guido> (rev 6)

I didn't make it up.  I'm pretty sure I saw it somewhere in either the wiki,
the python-3000 mailing list or in the source code itself.  I'll dig around
some and figure out where I found it.  In the meantime, if someone else
stumbles upon the smoking gun let me know.

    >> >     Martin> Please also add code that makes the build break in a
    >> >     Martin> version where the code is still in place, so that users
    >> >     Martin> can know that support for their platform is to be
    >> >     Martin> removed (in 3.0, I presume).
    >> >
    >> > How do I do that?  Does that go in the Python 2.6 code?
    >> 
    >> If you want to unsupport the system in 2.6, yes. You put something in
    >> configure.in which users can't fail to notice, e.g. a single-line
    >> abort that they need to uncomment before they can proceed. 

Thanks, I'll check out configure.in for that as well.

Skip

From skip at pobox.com  Sat Aug 18 02:03:33 2007
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 17 Aug 2007 19:03:33 -0500
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <ca471dc20708171442i1f3b6d7dl5048743bdfcde9fa@mail.gmail.com>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
	<46C5E2C6.4070106@v.loewis.de>
	<18118.806.945501.14325@montanaro.dyndns.org>
	<46C6110F.6050203@v.loewis.de>
	<ca471dc20708171442i1f3b6d7dl5048743bdfcde9fa@mail.gmail.com>
Message-ID: <18118.14165.619336.182393@montanaro.dyndns.org>


    I didn't make it up.  I'm pretty sure I saw it somewhere in either the
    wiki, the python-3000 mailing list or in the source code itself.  I'll
    dig around some and figure out where I found it.  In the meantime, if
    someone else stumbles upon the smoking gun let me know.

PEP 3100:

    Remove support for old systems, including: BeOS, RISCOS, (SGI) Irix,
    Tru64

I'll hold off on doing anything more until I can update PEP 11.

Skip

From guido at python.org  Sat Aug 18 02:13:29 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 17 Aug 2007 17:13:29 -0700
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <18118.14165.619336.182393@montanaro.dyndns.org>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
	<46C5E2C6.4070106@v.loewis.de>
	<18118.806.945501.14325@montanaro.dyndns.org>
	<46C6110F.6050203@v.loewis.de>
	<ca471dc20708171442i1f3b6d7dl5048743bdfcde9fa@mail.gmail.com>
	<18118.14165.619336.182393@montanaro.dyndns.org>
Message-ID: <ca471dc20708171713v2da0228age9b302e9e8117068@mail.gmail.com>

Neal added it in rev 43214, when it was still PEP 3000. I'm sure he
didn't read PEP 11 and beyond that I doubt he remembers from where he
pulled this info. :-)

On 8/17/07, skip at pobox.com <skip at pobox.com> wrote:
>
>     I didn't make it up.  I'm pretty sure I saw it somewhere in either the
>     wiki, the python-3000 mailing list or in the source code itself.  I'll
>     dig around some and figure out where I found it.  In the meantime, if
>     someone else stumbles upon the smoking gun let me know.
>
> PEP 3100:
>
>     Remove support for old systems, including: BeOS, RISCOS, (SGI) Irix,
>     Tru64
>
> I'll hold off on doing anything more until I can update PEP 11.
>
> Skip
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From bjourne at gmail.com  Sat Aug 18 02:31:10 2007
From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=)
Date: Sat, 18 Aug 2007 00:31:10 +0000
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <f9t2nn$ksg$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org>
Message-ID: <740c3aec0708171731qc9324c3o17debfafe4c1530d@mail.gmail.com>

It is fantastic! Totally super work. I just have one small request;
pretty please do not set the font. I'm very happy with my browsers
default (Verdana), and Bitstream Vera Sans renders badly for me.

-- 
mvh Bj?rn

From skip at pobox.com  Sat Aug 18 05:31:24 2007
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 17 Aug 2007 22:31:24 -0500
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <ca471dc20708171713v2da0228age9b302e9e8117068@mail.gmail.com>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
	<46C5E2C6.4070106@v.loewis.de>
	<18118.806.945501.14325@montanaro.dyndns.org>
	<46C6110F.6050203@v.loewis.de>
	<ca471dc20708171442i1f3b6d7dl5048743bdfcde9fa@mail.gmail.com>
	<18118.14165.619336.182393@montanaro.dyndns.org>
	<ca471dc20708171713v2da0228age9b302e9e8117068@mail.gmail.com>
Message-ID: <18118.26636.961444.339565@montanaro.dyndns.org>


    Guido> Neal added it in rev 43214, when it was still PEP 3000. I'm sure
    Guido> he didn't read PEP 11 and beyond that I doubt he remembers from
    Guido> where he pulled this info. :-)

Okay, next question.  Have we documented somewhere the official maintainers
of the various minor platforms (where that duty falls to one or a small
group of people)?  I realize for some platforms (Irix, Tru64/OSF/1/Digital
Unix) it may not be a single person, but for others (OS/2, AtheOS, BeOS), it
appears there is such a person.  It would be nice information to collect if
not.  (Maybe as a section of PEP 11?)

Skip

From nnorwitz at gmail.com  Sat Aug 18 07:12:33 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Fri, 17 Aug 2007 22:12:33 -0700
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <ca471dc20708171713v2da0228age9b302e9e8117068@mail.gmail.com>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
	<46C5E2C6.4070106@v.loewis.de>
	<18118.806.945501.14325@montanaro.dyndns.org>
	<46C6110F.6050203@v.loewis.de>
	<ca471dc20708171442i1f3b6d7dl5048743bdfcde9fa@mail.gmail.com>
	<18118.14165.619336.182393@montanaro.dyndns.org>
	<ca471dc20708171713v2da0228age9b302e9e8117068@mail.gmail.com>
Message-ID: <ee2a432c0708172212qf799ebam9a2bf79c20901d53@mail.gmail.com>

On 8/17/07, Guido van Rossum <guido at python.org> wrote:
> Neal added it in rev 43214, when it was still PEP 3000. I'm sure he
> didn't read PEP 11 and beyond that I doubt he remembers from where he
> pulled this info. :-)

When I added it in 43214, it was in an outstanding questions section.
It wasn't meant to become gospel. :-)  We have pretty crappy support
for many versions of Unix.  There are various problems with Solaris,
AIX, HP-UX, etc.  Not all the tests work on many of these platforms.
When people report problems, they are almost always left unanswered.
Since it's difficult to get access to boxes with the problems, our
support isn't likely to improve.  That was part of the motivation to
consider removing the code.  Another motivating factor was to
eliminate code and make python easier to maintain.

It may be reasonable that little used platforms are maintained as
patches outside the core.  VMS did this for a long time.  I'm not sure
if stock python has full VMS support or if there are still some
patches necessary.

Below are more details in case anyone cares.

The checkin spawned this thread:
http://mail.python.org/pipermail/python-dev/2006-March/062741.html

One result was that we got a Tru64 buildbot out of it (which I have
access to and help maintain).  I'm not against keeping lesser
platforms.  But if they aren't used, they should be removed.

Here is HP's support for Tru64:  http://h30097.www3.hp.com/support.html

The supported versions show the last released version was V5.1B-4 Dec
2006.  HP still supports B-3 and B-4 for a minimum of 42 months from
release (Jan 2005 for B-3).  The end of support is still TBD for B-3
and B-4.

n
--
>
> On 8/17/07, skip at pobox.com <skip at pobox.com> wrote:
> >
> >     I didn't make it up.  I'm pretty sure I saw it somewhere in either the
> >     wiki, the python-3000 mailing list or in the source code itself.  I'll
> >     dig around some and figure out where I found it.  In the meantime, if
> >     someone else stumbles upon the smoking gun let me know.
> >
> > PEP 3100:
> >
> >     Remove support for old systems, including: BeOS, RISCOS, (SGI) Irix,
> >     Tru64
> >
> > I'll hold off on doing anything more until I can update PEP 11.
> >
> > Skip
> >
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>

From martin at v.loewis.de  Sat Aug 18 07:52:12 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Aug 2007 07:52:12 +0200
Subject: [Python-Dev] Source file markers for Tru64?
In-Reply-To: <18118.26636.961444.339565@montanaro.dyndns.org>
References: <18117.44627.786049.522772@montanaro.dyndns.org>
	<46C5E2C6.4070106@v.loewis.de>
	<18118.806.945501.14325@montanaro.dyndns.org>
	<46C6110F.6050203@v.loewis.de>
	<ca471dc20708171442i1f3b6d7dl5048743bdfcde9fa@mail.gmail.com>
	<18118.14165.619336.182393@montanaro.dyndns.org>
	<ca471dc20708171713v2da0228age9b302e9e8117068@mail.gmail.com>
	<18118.26636.961444.339565@montanaro.dyndns.org>
Message-ID: <46C6890C.8080101@v.loewis.de>

> Okay, next question.  Have we documented somewhere the official maintainers
> of the various minor platforms (where that duty falls to one or a small
> group of people)?  I realize for some platforms (Irix, Tru64/OSF/1/Digital
> Unix) it may not be a single person, but for others (OS/2, AtheOS, BeOS), it
> appears there is such a person.  It would be nice information to collect if
> not.  (Maybe as a section of PEP 11?)

Sounds fine to me.

Regards,
Martin

From 2006a at usenet.alexanderweb.de  Sat Aug 18 17:02:44 2007
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Sat, 18 Aug 2007 17:02:44 +0200
Subject: [Python-Dev] sys.last_traceback - annoying pdb.pm behaviour
Message-ID: <l9ejp4-ikt.ln1@tuxpaddy.alexanderweb.homeip.net>

Hi there,

when trying to use pdb, I spotted some kind of annoying behaviour of pdb:
for pdb.pm, it still uses sys.last_traceback which is set by code.py

This renders the function unusable in non-interactive contexts ... I suggest
to change pdb.pm to use sys.exc_traceback instead. Any reasons not to do
so?

For Py3k, it might make sense to kill it.

Besides that, some other places in the stdlib reference this attr as well:

idle in 5 places
./lib-tk/Tkinter.py:        sys.last_traceback = tb
./traceback.py:    sys.last_value, sys.last_traceback, limit, file)'."""
./traceback.py:    print_exception(sys.last_type, sys.last_value,
sys.last_traceback,
./pdb.py:    post_mortem(sys.last_traceback)
./code.py:        type, value, sys.last_traceback = sys.exc_info()
./code.py:            sys.last_traceback = tb
./dis.py:            tb = sys.last_traceback

Kind regards,
Alexander


From 2006a at usenet.alexanderweb.de  Sat Aug 18 17:09:01 2007
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Sat, 18 Aug 2007 17:09:01 +0200
Subject: [Python-Dev] sys.last_traceback - annoying pdb behaviour
Message-ID: <dlejp4-ikt.ln1@tuxpaddy.alexanderweb.homeip.net>

Hi there,

when I tried to use pdb, I spotted some kind of annoying behaviour of pdb:
for pdb.pm, it still uses sys.last_traceback which is set by code.py

This renders the function unusable in non-interactive contexts ... I suggest
to change pdb.pm to use sys.exc_traceback instead. Any reasons not to do
so?

For Py3k, it might make sense to kill it completly.

Besides that, some other places in the stdlib reference this attr as well:

idle in 5 places
./lib-tk/Tkinter.py: ? ? ? ?sys.last_traceback = tb
./traceback.py: ? ?sys.last_value, sys.last_traceback, limit, file)'."""
./traceback.py: ? ?print_exception(sys.last_type, sys.last_value,
sys.last_traceback,
./pdb.py: ? ?post_mortem(sys.last_traceback)
./code.py: ? ? ? ?type, value, sys.last_traceback = sys.exc_info()
./code.py: ? ? ? ? ? ?sys.last_traceback = tb
./dis.py: ? ? ? ? ? ?tb = sys.last_traceback

Kind regards,
Alexander


From guido at python.org  Sat Aug 18 21:46:59 2007
From: guido at python.org (Guido van Rossum)
Date: Sat, 18 Aug 2007 12:46:59 -0700
Subject: [Python-Dev] sys.last_traceback - annoying pdb behaviour
In-Reply-To: <dlejp4-ikt.ln1@tuxpaddy.alexanderweb.homeip.net>
References: <dlejp4-ikt.ln1@tuxpaddy.alexanderweb.homeip.net>
Message-ID: <ca471dc20708181246x71f25393l1a580b545c7a6f8a@mail.gmail.com>

sys.last_traceback is the only place where you can reliably find the
latest *uncaught* traceback. There is no intention to get rid of it in
Py3k or to change this behavior. If you have another traceback you'd
like to print, you can use pdb.post_mortem(sys.exc_info()[2]).

On 8/18/07, Alexander Schremmer <2006a at usenet.alexanderweb.de> wrote:
> Hi there,
>
> when I tried to use pdb, I spotted some kind of annoying behaviour of pdb:
> for pdb.pm, it still uses sys.last_traceback which is set by code.py
>
> This renders the function unusable in non-interactive contexts ... I suggest
> to change pdb.pm to use sys.exc_traceback instead. Any reasons not to do
> so?
>
> For Py3k, it might make sense to kill it completly.
>
> Besides that, some other places in the stdlib reference this attr as well:
>
> idle in 5 places
> ./lib-tk/Tkinter.py: sys.last_traceback = tb
> ./traceback.py: sys.last_value, sys.last_traceback, limit, file)'."""
> ./traceback.py: print_exception(sys.last_type, sys.last_value,
> sys.last_traceback,
> ./pdb.py: post_mortem(sys.last_traceback)
> ./code.py: type, value, sys.last_traceback = sys.exc_info()
> ./code.py: sys.last_traceback = tb
> ./dis.py: tb = sys.last_traceback
>
> Kind regards,
> Alexander
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From 2007a at usenet.alexanderweb.de  Sat Aug 18 23:39:04 2007
From: 2007a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Sat, 18 Aug 2007 23:39:04 +0200
Subject: [Python-Dev] sys.last_traceback - annoying pdb behaviour
References: <dlejp4-ikt.ln1@tuxpaddy.alexanderweb.homeip.net>
	<ca471dc20708181246x71f25393l1a580b545c7a6f8a@mail.gmail.com>
Message-ID: <og5kp4-4oe.ln1@tuxpaddy.alexanderweb.homeip.net>

Guido van Rossum wrote:

> On 8/18/07, Alexander Schremmer <2006a at usenet.alexanderweb.de> wrote:

>> when I tried to use pdb, I spotted some kind of annoying behaviour of
>> pdb: for pdb.pm, it still uses sys.last_traceback which is set by code.py
>>
>> This renders the function unusable in non-interactive contexts ... I
>> suggest to change pdb.pm to use sys.exc_traceback instead. Any reasons
>> not to do so?

> sys.last_traceback is the only place where you can reliably find the
> latest *uncaught* traceback. There is no intention to get rid of it in
> Py3k or to change this behavior. If you have another traceback you'd
> like to print, you can use pdb.post_mortem(sys.exc_info()[2]).

Indeed, I did not see that it is also set in pythonrun. How do you think
about changing pdb.pm to use exc_info if last_exception is not set?

Kind regards,
Alexander


From guido at python.org  Sun Aug 19 00:11:48 2007
From: guido at python.org (Guido van Rossum)
Date: Sat, 18 Aug 2007 15:11:48 -0700
Subject: [Python-Dev] sys.last_traceback - annoying pdb behaviour
In-Reply-To: <og5kp4-4oe.ln1@tuxpaddy.alexanderweb.homeip.net>
References: <dlejp4-ikt.ln1@tuxpaddy.alexanderweb.homeip.net>
	<ca471dc20708181246x71f25393l1a580b545c7a6f8a@mail.gmail.com>
	<og5kp4-4oe.ln1@tuxpaddy.alexanderweb.homeip.net>
Message-ID: <ca471dc20708181511y48381627l6192cdcd77e1475c@mail.gmail.com>

On 8/18/07, Alexander Schremmer <2007a at usenet.alexanderweb.de> wrote:
> Indeed, I did not see that it is also set in pythonrun. How do you think
> about changing pdb.pm to use exc_info if last_exception is not set?

That would just confuse you if you're counting on that behavior, and
then some time later last_exception gets set and you suddenly don't
get the behavior you expected. pdb.pm() is *only* for use at the
command-line prompt (that's why its name is so short :-).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip at pobox.com  Sun Aug 19 00:27:28 2007
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 18 Aug 2007 17:27:28 -0500
Subject: [Python-Dev] PEP 11 update - Call for port maintainers to step
	forward
Message-ID: <18119.29264.543717.894262@montanaro.dyndns.org>

I made a quick update to PEP 11, "Removing support for little used
platforms".  I added details about ending support for AtheOS/Syllable and
BeOS.

I also added a yet-to-be-fleshed out section entitled "Platform
Maintainers".  I intend that to the extent possible we document the
responsible parties for various platforms.  Obviously, common platforms like
Windows, Mac OS X, Linux and common Unix platforms (Solaris, *BSD, what
else?)  will continue to be supported by the core Python developer
community, but lesser platforms should have one or more champions, and we
should be able to get ahold of them to determine their continued interest in
supporting Python on their platform(s).  If you are the "owner" of a minor
platform, please drop me a note.  Ones I'm aware of that probably need
specialized support outside the core Python developers include:

    IRIX
    Tru64 (aka OSF/1 and other names (what else?))
    OS2/EMX (Andrew MacIntyre?)
    Cygwin
    MinGW
    HP-UX
    AIX
    Solaris < version 8
    SCO
    Unixware

IRIX and Tru64 are likely to go the way of the dodo if someone doesn't step
up soon to offer support.  I don't expect the others to disappear soon, but
they tend to need more specialized support, especially in more "challenging"
areas (shared library support, threading, etc).

If you maintain the platform-specific aspects for any of these platforms,
please let me know.  If you aren't that person but know who is, please pass
this note along to them.  If I've missed any other platforms (I know I must
have have missed something), let me know that as well.

Thanks,

-- 
Skip Montanaro - skip at pobox.com - http://www.webfast.com/~skip/

From status at bugs.python.org  Sun Aug 19 02:00:57 2007
From: status at bugs.python.org (Tracker)
Date: Sun, 19 Aug 2007 00:00:57 +0000 (UTC)
Subject: [Python-Dev] Summary of Tracker Issues
Message-ID: <20070819000057.B4E487809B@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (08/12/07 - 08/19/07)
Tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 1300 open ( +5) / 11141 closed ( +0) / 12441 total ( +5)

Average duration of open issues: 694 days.
Median duration of open issues: 553 days.

Open Issues Breakdown
   open  1300 ( +5)
pending     0 ( +0)

Issues Created Or Reopened (5)
______________________________

Test                                                             08/17/07
       http://bugs.python.org/issue1000    created  loewis         

urllib2 hangs with some documents.                               08/12/07
       http://bugs.python.org/issue1772481 created  acreature      

dir() on traceback objects returns an empty list                 08/12/07
       http://bugs.python.org/issue1772489 created  collinwinter   

Replacing char* with const char*                                 08/12/07
       http://bugs.python.org/issue1772673 created  ext-           

exec() doesn't take an open file                                 08/12/07
       http://bugs.python.org/issue1772686 created  nnorwitz       



Issues Now Closed (1)
_____________________

words able to decode but unable to encode in GB18030               4 days
       http://bugs.python.org/issue1770551 perky          



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070819/98788979/attachment.html 

From kbk at shore.net  Sun Aug 19 03:58:40 2007
From: kbk at shore.net (Kurt B. Kaiser)
Date: Sat, 18 Aug 2007 21:58:40 -0400 (EDT)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200708190158.l7J1weEW020776@hampton.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  417 open (+13) /  3855 closed ( +0) /  4272 total (+13)
Bugs    : 1069 open ( +4) /  6800 closed (+10) /  7869 total (+14)
RFE     :  262 open ( -1) /   296 closed ( +1) /   558 total ( +0)

New / Reopened Patches
______________________

Replacing char* with const char*  (2007-08-12)
       http://python.org/sf/1772673  opened by  eXt

[python-mode] Properly highlight lambda with no arguments  (2007-08-12)
       http://python.org/sf/1772721  opened by  Michal Kwiatkowski

-q (quiet) option for python interpreter  (2007-08-13)
       http://python.org/sf/1772833  opened by  Marcin Wojdyr

Decimal and long hash, compatibly and efficiently  (2007-08-13)
       http://python.org/sf/1772851  opened by  Mark Dickinson

Remove references to _xmlrpclib from xmlrpclib.py  (2007-08-14)
       http://python.org/sf/1773632  opened by  Alan McIntyre

Unify __builtins__ -> __builtin__  (2007-08-15)
       http://python.org/sf/1774369  opened by  Christian Heimes

Add Checkbutton get() and set(value)  (2007-08-14)
       http://python.org/sf/1774370  opened by  Graham Horler

Make it possible to use SVK to develop Python  (2007-08-14)
       http://python.org/sf/1774414  opened by  Collin Winter

Override flags set by IOBase in io.StringIO  (2007-08-15)
       http://python.org/sf/1774828  opened by  Alexandre Vassalotti

Convert str to bytes in io.BytesIO.__init__  (2007-08-15)
       http://python.org/sf/1774833  opened by  Alexandre Vassalotti

zipfile: Allow reading duplicate filenames  (2007-08-15)
       http://python.org/sf/1775025  opened by  Graham Horler

utt-32 codecs  (2007-08-16)
       http://python.org/sf/1775604  opened by  Walter D?rwald

Minor corrections to smtplib  (2007-08-17)
       http://python.org/sf/1776581  opened by  Alan McIntyre

New / Reopened Bugs
___________________

urllib2 hangs with some documents.  (2007-08-12)
       http://python.org/sf/1772481  opened by  Creature

dir() on traceback objects returns an empty list  (2007-08-11)
CLOSED http://python.org/sf/1772489  opened by  Collin Winter

exec() doesn't take an open file  (2007-08-12)
CLOSED http://python.org/sf/1772686  opened by  Neal Norwitz

chr(128) in u'only ascii' -> TypeError with misleading msg  (2007-08-13)
       http://python.org/sf/1772788  opened by  Pekka Laukkanen

Using telnetlib fails with unicode strings containing only a  (2007-08-13)
       http://python.org/sf/1772794  opened by  Pekka Laukkanen

Py Lib Ref, 3.6.1 String Methods startswith() typo  (2007-08-13)
CLOSED http://python.org/sf/1772890  opened by  arxaaron

xmlrpclib crash when PyXML installed  (2007-08-13)
       http://python.org/sf/1772916  opened by  lplatypus

Binding <Control-space> fails  (2007-08-15)
       http://python.org/sf/1774736  opened by  Ali Gholami Rudi

Not exiting when running tests  (2007-08-15)
       http://python.org/sf/1774840  opened by  Jos? Pablo "Pupeno" Fern?ndez

Display CallTips for classes using metaclasses.  (2007-08-16)
       http://python.org/sf/1775388  opened by  Noam Raphael

Buffer overflow when listing deeply nested directory  (2007-08-17)
       http://python.org/sf/1776160  opened by  Bj?rn Lindqvist

glob.glob inconsistent  (2007-08-18)
       http://python.org/sf/1776674  opened by  norbidur

tempfile.TemporaryFile differs between platforms  (2007-08-18)
       http://python.org/sf/1776696  opened by  Kenneth Loafman

memoryview('test') is causing a segfault  (2007-08-19)
       http://python.org/sf/1777057  opened by  Christian Heimes

Bugs Closed
___________

ERROR - Microsoft Visual C++ Runtime Library  (2007-05-18)
       http://python.org/sf/1721161  closed by  gbrandl

dir() on traceback objects returns an empty list  (2007-08-11)
       http://python.org/sf/1772489  closed by  collinwinter

words able to decode but unable to encode in GB18030  (2007-08-09)
       http://python.org/sf/1770551  closed by  perky

exec() doesn't take an open file  (2007-08-12)
       http://python.org/sf/1772686  closed by  bcannon

Py Lib Ref, 3.6.1 String Methods startswith() typo  (2007-08-13)
       http://python.org/sf/1772890  closed by  gbrandl

decimal.Decimal("trash") produces informationless exception  (2007-08-08)
       http://python.org/sf/1770009  closed by  facundobatista

mark deprecated modules in indexes  (2003-10-02)
       http://python.org/sf/816725  closed by  gbrandl

Python function/method/constant names as HTML-tag IDs  (2005-07-24)
       http://python.org/sf/1243945  closed by  gbrandl

Collapse distutils docs  (2005-12-07)
       http://python.org/sf/1375258  closed by  gbrandl

some section links (previous, up, next) missing last word  (2006-09-15)
       http://python.org/sf/1559142  closed by  gbrandl

RFE Closed
__________

-q (quiet) option for python interpreter  (2007-05-30)
       http://python.org/sf/1728488  closed by  wojdyr


From orsenthil at users.sourceforge.net  Sun Aug 19 21:10:31 2007
From: orsenthil at users.sourceforge.net (O.R.Senthil Kumaran)
Date: Mon, 20 Aug 2007 00:40:31 +0530
Subject: [Python-Dev] RFC - proposal for urilib. unified module of urlparse,
	urllib and	urllib2
Message-ID: <20070819191031.GD3437@gmail.com>

Hello All,

I am drafting a PEP proposing a module 'urilib', which will be the unified module of urlparse, urllib and urllib2.

a) _all_ functions will include from urlparse,urllib and urllib2.
b) overlapping functionality between urllib and urllib2 to be ironed out.
c) RFC3986 and RFC3987 compliant.

Cleanup task of urllib was my G-SoC, and this unified module is a suggestion with python 2.6 in mind.

I would like to know your thoughts on
- How should we plan to maintain the backward compatiblity?
- What other features should be kept in mind while designing?

Thanks,
Senthil



-- 
O.R.Senthil Kumaran
http://uthcode.sarovar.org

From janssen at parc.com  Sun Aug 19 22:34:08 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 19 Aug 2007 13:34:08 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug15.182939pdt."57996"@synergy1.parc.xerox.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>

> The idea is that if you call socket.ssl() on a socket that's bound to
> an address, the socket is assumed to be server-side, the cert passed
> in is assumed to be a server-side cert, and the SSLObject returned has
> a couple of extra methods, listen() and accept().  Calling accept() does
> the SSL dance with the remote side, and returns an SSLObject.

After looking at the code a bit more, I think it's wrong to use
SSLObject for both of these uses.  Instead, it makes more sense to
introduce a different C-implemented object, SSLServerPort, which
supports only the method "accept".

So the Pythonic change will be that calling socket.ssl() will call the
"getpeername" method on the socket, and if it doesn't have one, will
assume it's server-side, and construct and return an SSLServerPort
instance instead of doing the SSL dance and returning SSObject.  Code
using this will then call listen on the SSLServerPort, and when a
connection is requested, will call "accept" to do the server-side SSL
dance, which if successful will return an SSLObject.

I intend to add a method to SSLObject, "peer", which will be
functionally equivalent to the current "server" method, which returns
a string version of the name of the server the socket is connected to.
We'd deprecate the use of "server", and replace it with "peer" over
time.  Similarly, internally in _ssl.c, the "server_cert" slot of
SSLObject will become "peer_cert".

I'm very tempted to add an optional parameter to socket.ssl(), which
will be a callback function which will be passed the remote side's IP
address and credentials, and which may raise an exception if it
doesn't like the credentials.  The exception would then be re-raised
from socket.ssl() (on the client side) or SSLServerPort.accept() (on
the server side).  The callback function would of course default to
None which would give the current behavior.  This is different from
the built-in SSL verify function, which just checks the certificate
chain.  This could do things like only allowing particular clients to
reach the server; it would also work on the client side, allowing a
client to check the credentials of the server.  The problem with doing
this is that we'd also need to devise a Python object wrapper for the
cert itself, with an appropriate API.  More work.

Bill

From g.brandl at gmx.net  Sun Aug 19 23:06:16 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 19 Aug 2007 23:06:16 +0200
Subject: [Python-Dev] RFC - proposal for urilib. unified module of
 urlparse, urllib and urllib2
In-Reply-To: <20070819191031.GD3437@gmail.com>
References: <20070819191031.GD3437@gmail.com>
Message-ID: <faabc3$rup$1@sea.gmane.org>

O.R.Senthil Kumaran schrieb:
> Hello All,
> 
> I am drafting a PEP proposing a module 'urilib', which will be the unified
> module of urlparse, urllib and urllib2.
> 
> a) _all_ functions will include from urlparse,urllib and urllib2. b)
> overlapping functionality between urllib and urllib2 to be ironed out. c)
> RFC3986 and RFC3987 compliant.
> 
> Cleanup task of urllib was my G-SoC, and this unified module is a suggestion
> with python 2.6 in mind.
> 
> I would like to know your thoughts on - How should we plan to maintain the
> backward compatiblity? - What other features should be kept in mind while
> designing?

In any case, you should probably be ready to write and maintain a PEP to keep
track of suggestions and proposals in case there are many of them, or
inclusion in 2.6 is debated.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From janssen at parc.com  Mon Aug 20 00:15:43 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 19 Aug 2007 15:15:43 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug19.133416pdt."57996"@synergy1.parc.xerox.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>

> I'm very tempted to add an optional parameter to socket.ssl(), which
> will be a callback function which will be passed the remote side's IP
> address and credentials, and which may raise an exception if it
> doesn't like the credentials.  The exception would then be re-raised
> from socket.ssl() (on the client side) or SSLServerPort.accept() (on
> the server side).  The callback function would of course default to
> None which would give the current behavior.  This is different from
> the built-in SSL verify function, which just checks the certificate
> chain.  This could do things like only allowing particular clients to
> reach the server; it would also work on the client side, allowing a
> client to check the credentials of the server.  The problem with doing
> this is that we'd also need to devise a Python object wrapper for the
> cert itself, with an appropriate API.  More work.

A simpler way of doing this would be to just have the "peer" call on
SSLObject return a read-only mapping giving the entities in the peer
certificate (or None, or host and port, if there is no peer
certificate).  Then the application can look at this before working
with it, and decide whether to communicate with the remote end.

It would still be nice to add a flag argument to socket.ssl() to allow
the application to control the level of verification for the remote
certificate.  CERT_NONE would mean that no cert is required, or if
provided not verified (SSL_VERIFY_NONE).  CERT_OPTIONAL would mean
that no cert is required, but if one is provided, it's verified
(SSL_VERIFY_PEER).  CERT_REQUIRED would require a cert to be provided,
and would verify it (SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT).
The default SSL verify procedure would be used for verification.  The
default would be CERT_NONE, which is what is currently used.

Now, does this mean that we'd need to create a database of trusted
certs, similar to Java's trustStore and various OS X keychains?  I'm
not sure how useful verification will be without such a facility?

I see that there was some discussion about this in December, which I
missed earlier.

http://mail.python.org/pipermail/python-list/2006-December/418511.html

And of course there has been discussion about SSL in general:

http://mail.python.org/pipermail/python-dev/2005-June/054083.html
http://mail.python.org/pipermail/python-dev/2001-October/018161.html

There's a list of failings of the current SSL support by Gerhard H?ring:

- no support for specifying the SSL protocol (SSL v2/SSL v3/SSL v2|3)

(I wasn't planning to touch that.  The current code uses v2|3.
However, this might be a good time to fix this, too.  It would be
simple to provide an optional keyword protocol argument to
socket.ssl().)

- no proper certicate checking

(See above discussion of verification.)

- SSL objects are neither compatible with sockets nor are they file-like
  objects

(Don't currently intend to change this.)

- no advanced features like SSL callbacks (to allow the user to accept a
  cerficicate, for example)

(Providing more information about the peer cert would halfway address this.)

And there is the pyOpenSSL library, from
http://mail.python.org/pipermail/python-dev/2001-July/015855.html,
now at http://pyopenssl.sourceforge.net/.  It would be nice if we
could just use that X509 object type to allow the app to get its hands
on the peer cert, but it's GPL'd code.

Bill

From adam at hupp.org  Tue Aug 14 18:16:21 2007
From: adam at hupp.org (Adam Hupp)
Date: Tue, 14 Aug 2007 11:16:21 -0500
Subject: [Python-Dev] [Python-3000] Universal newlines support in Python
	3.0
In-Reply-To: <FAEFC901-BAE1-4D56-8FEE-6098E307DAF2@python.org>
References: <ca471dc20708101123y3628e352n7116839dbfdc9bb3@mail.gmail.com>
	<87wsw3p5em.fsf@uwakimon.sk.tsukuba.ac.jp>
	<rowen-D017E8.10460713082007@sea.gmane.org>
	<ca471dc20708131315i6dba68d6u9efae161d1b647ca@mail.gmail.com>
	<FAEFC901-BAE1-4D56-8FEE-6098E307DAF2@python.org>
Message-ID: <20070814161621.GA26420@mouth.upl.cs.wisc.edu>

On Tue, Aug 14, 2007 at 09:58:32AM -0400, Barry Warsaw wrote:
> This was all fine except that some of the tests started  
> failing because of the EOL translation that happens unconditionally  
> now.   The file contained \r\n and the test was ensuring these EOLs  
> were preserved in the parsed text.  I switched back to opening the  
> file in binary mode, and doing a crufty conversion of bytes to  
> strings (which I suspect is error prone but gets me farther along).
> 
> It would have been perfect, I think, if I could have opened the file  
> in text mode so that read() gave me strings, with universal newlines  
> and preservation of line endings (i.e. no translation to \n).

FWIW this same issue (and solution) came up while fixing the csv
tests.

-- 
Adam Hupp | http://hupp.org/adam/


From brett at python.org  Mon Aug 20 09:51:41 2007
From: brett at python.org (Brett Cannon)
Date: Mon, 20 Aug 2007 00:51:41 -0700
Subject: [Python-Dev] Planning to switch to new tracker on August 23rd
Message-ID: <bbaeab100708200051kadbe485h2cdc3176b157d8a4@mail.gmail.com>

Having squashed the final issues, we are now ready to switch over to
the new tracker!  The plan is to do it on the 23rd.  But before I
announce to the community I wanted to make sure there was not some
specific objection by python-dev or python-3000.  If there is please
let me know by midday Monday so that we can postpone to next week if
needed.

-Brett

From skip at pobox.com  Mon Aug 20 17:52:46 2007
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 20 Aug 2007 10:52:46 -0500
Subject: [Python-Dev] Email addresses in PEPs?
Message-ID: <18121.47310.218893.540750@montanaro.dyndns.org>

Is there a policy about putting email addresses in PEPs?  I have the names
and email addresses of a couple platform maintainers to add to PEP 11.
Having the email addresses there would make it handy to contact said
maintainers, but I realize people don't always appreciate the extra exposure
to spammers.

Skip


From janssen at parc.com  Mon Aug 20 18:46:17 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 20 Aug 2007 09:46:17 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug19.151552pdt."57996"@synergy1.parc.xerox.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>

All right, here's my current design :-).

We add to the socket module a subtype of socket.socket,
socket.SSLSocket.  It has the following constructor:

  SSLSocket (family=AF_INET, type=SOCK_STREAM, proto=0,
	     cert_filename=None, key_filename=None,
	     cert_policy=CERT_NONE, ssl_protocol=PROTOCOL_SSLv23)

where the options for "cert_policy" are:

  CERT_NONE -- no certificates will be required or verified
  CERT_OPTIONAL -- cert not required from peer, but if provided is verified
  CERT_REQUIRED -- verifiable cert required from peer

and the options for "ssl_protocol" are:

  PROTOCOL_SSLv2
  PROTOCOL_SSLv3
  PROTOCOL_SSLv23
  PROTOCOL_TLSv1

The "cert_filename" is optional for clients, required for servers; the
"key_filename" can be used to specify another file where the private key
for the certificate is stored, otherwise the key is presumed to be in
the cert file along with the certificate (or certicates; the cert file
may contain a chain of certificates).

In use, the application creates an "SSLSocket" instead of a regular
"socket" if it wishes to use SSL.  If it's client-side, when the
"connect" call is made on the socket, the SSL handshake is performed,
and subsequent reads from and writes to the socket are delegated to
the SSLObject that's returned from the handshake.  If it's
server-side, when the "accept" call is made, the socket returned is
another SSLSocket with the SSLObject already set, so that writes and
reads are processed via the SSLObject.  Actually, "recv", "recvinto",
"send" and "sendall" are the methods delegated; "recv_from" and
"send_to" raise exceptions (and I think using any flags on the other
methods should also raise exceptions).

An instance of SSLSocket will have a method called "getpeercert",
which will return either None, if the socket is not connected, or a
dict containing pertinent fields of the peer's certificate; empty if
the peer did not provide a cert or if the CERT_NONE option was
specified, or filled with fields like "subject", "issuer", etc. if it
did and the cert was verified.  (What fields, exactly?  Seems to me
that it should have subject, issuer, good_after, good_until, but what
else?)

The existing use of "socket.ssl" will continue to work, for backwards
compatibility.

This design should allow SSLSocket to be used in all the various
places where a regular socket is currently used, such as
SimpleHTTPServer, with only a few tweaks.  For instance, here's an SSL
version of TCPServer:

class MySSLTCPServer (TCPServer):

    def __init__(self, server_address, RequestHandlerClass, certfile,
	         bind_and_activate=True):
        """Constructor.  May be extended, do not override."""
        BaseServer.__init__(self, server_address, RequestHandlerClass)
        self.socket = socket.SSLSocket(self.address_family,
                                       self.socket_type,
                                       certfile=certfile)
        if bind_and_activate:
            self.server_bind()
            self.server_activate()

Bill



From martin at v.loewis.de  Mon Aug 20 19:05:07 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 20 Aug 2007 19:05:07 +0200
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <46C9C9C3.9090201@v.loewis.de>

> We add to the socket module a subtype of socket.socket,
> socket.SSLSocket.  It has the following constructor:
> 
>   SSLSocket (family=AF_INET, type=SOCK_STREAM, proto=0,
> 	     cert_filename=None, key_filename=None,
> 	     cert_policy=CERT_NONE, ssl_protocol=PROTOCOL_SSLv23)

That's somewhat limiting - you should be able to do connection
upgrades (e.g. SMTP STARTTLS, or HTTP Connection: Upgrade); with
that design, such usages would not be possible, no?

Regards,
Martin

From janssen at parc.com  Mon Aug 20 19:19:25 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 20 Aug 2007 10:19:25 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <46C9C9C3.9090201@v.loewis.de> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
Message-ID: <07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>

> That's somewhat limiting - you should be able to do connection
> upgrades (e.g. SMTP STARTTLS, or HTTP Connection: Upgrade); with
> that design, such usages would not be possible, no?

Yes, you're right.  Of course, STARTTLS is properly regarded as a
terrible hack :-).

The actual functionality exported from _ssl.c is still the "ssl"
wrapper, but with more arguments to control its behavior.  So to do
STARTTLS, server-side, you'd do something like

  mooring = socket.socket()
  mooring.bind()
  mooring.listen()
  [... connection request comes in ...]
  fd = mooring.accept()		# normal socket
  [... read request for TLS upgrade over socket ...]
  sslobj = socket.ssl(fd, ..., server=True)
  fd = socket.SSLSocket(..., ssl_protocol=PROTOCOL_TLSv1, _sock=fd, _sslobj=sslobj)

and continue on with normal use of the socket.  Do you see an easier
way to do it?

Bill

From martin at v.loewis.de  Mon Aug 20 19:33:45 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 20 Aug 2007 19:33:45 +0200
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <46C9D079.1040009@v.loewis.de>

> Yes, you're right.  Of course, STARTTLS is properly regarded as a
> terrible hack :-).

I know you mean that jokingly - but I think it is considered as the
*proper* usage of TLS, with all these "let's use a different well-known
port for TLS over X" being those protocols that are hacks. I believe
it's official IETF policy (although I can't find the RFC right now)
that future protocols should prefer connection upgrades for TLS over
introducing separate protocols.

> The actual functionality exported from _ssl.c is still the "ssl"
> wrapper, but with more arguments to control its behavior.  So to do
> STARTTLS, server-side, you'd do something like
> 
>   mooring = socket.socket()
>   mooring.bind()
>   mooring.listen()
>   [... connection request comes in ...]
>   fd = mooring.accept()		# normal socket
>   [... read request for TLS upgrade over socket ...]
>   sslobj = socket.ssl(fd, ..., server=True)
>   fd = socket.SSLSocket(..., ssl_protocol=PROTOCOL_TLSv1, _sock=fd, _sslobj=sslobj)
> 
> and continue on with normal use of the socket.  Do you see an easier
> way to do it?

If you have use cases where you need to pass _sock, anyway, why not
always require a wrapper object (i.e. not support the family/type/proto
arguments). I think there are also cases where you need to set socket
options on the "raw" socket.

I view TLS as a wrapper around / layer on top of TCP, and so I think the
API should look like, as well.

Whatever you design, it will either be complicated to use or
insufficient in reasonable use cases.

Regards,
Martin

From guido at python.org  Mon Aug 20 20:20:47 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 20 Aug 2007 11:20:47 -0700
Subject: [Python-Dev] Email addresses in PEPs?
In-Reply-To: <18121.47310.218893.540750@montanaro.dyndns.org>
References: <18121.47310.218893.540750@montanaro.dyndns.org>
Message-ID: <ca471dc20708201120o23d527d4lc62e01bc4fc13585@mail.gmail.com>

Ask the people involved if they're okay with "user at host dot com"
obfuscation. That's used in a few places already (e.g. PEP 0).

On 8/20/07, skip at pobox.com <skip at pobox.com> wrote:
> Is there a policy about putting email addresses in PEPs?  I have the names
> and email addresses of a couple platform maintainers to add to PEP 11.
> Having the email addresses there would make it handy to contact said
> maintainers, but I realize people don't always appreciate the extra exposure
> to spammers.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From fijall at gmail.com  Mon Aug 20 20:27:50 2007
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 20 Aug 2007 20:27:50 +0200
Subject: [Python-Dev] Segfault
Message-ID: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>

IMHO this shouldn't segfault:

import thread

while 1:
    f = open("/tmp/dupa", "w")
    thread.start_new_thread(f.close, ())
    f.close()

while it does on cpython 2.5.1, linux box.

May I consider this a bug?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070820/531dbe9b/attachment.htm 

From brett at python.org  Mon Aug 20 21:16:02 2007
From: brett at python.org (Brett Cannon)
Date: Mon, 20 Aug 2007 12:16:02 -0700
Subject: [Python-Dev] Email addresses in PEPs?
In-Reply-To: <ca471dc20708201120o23d527d4lc62e01bc4fc13585@mail.gmail.com>
References: <18121.47310.218893.540750@montanaro.dyndns.org>
	<ca471dc20708201120o23d527d4lc62e01bc4fc13585@mail.gmail.com>
Message-ID: <bbaeab100708201216k213112e0u10b47013ec75837a@mail.gmail.com>

On 8/20/07, Guido van Rossum <guido at python.org> wrote:
> Ask the people involved if they're okay with "user at host dot com"
> obfuscation. That's used in a few places already (e.g. PEP 0).
>

I believe email addresses are automatically obfuscated as part of the
HTML generation process, but one of the PEP editors can correct me if
I am wrong.

-Brett


> On 8/20/07, skip at pobox.com <skip at pobox.com> wrote:
> > Is there a policy about putting email addresses in PEPs?  I have the names
> > and email addresses of a couple platform maintainers to add to PEP 11.
> > Having the email addresses there would make it handy to contact said
> > maintainers, but I realize people don't always appreciate the extra exposure
> > to spammers.
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>

From goodger at python.org  Mon Aug 20 21:32:10 2007
From: goodger at python.org (David Goodger)
Date: Mon, 20 Aug 2007 15:32:10 -0400
Subject: [Python-Dev] [PEPs]  Email addresses in PEPs?
In-Reply-To: <bbaeab100708201216k213112e0u10b47013ec75837a@mail.gmail.com>
References: <18121.47310.218893.540750@montanaro.dyndns.org>
	<ca471dc20708201120o23d527d4lc62e01bc4fc13585@mail.gmail.com>
	<bbaeab100708201216k213112e0u10b47013ec75837a@mail.gmail.com>
Message-ID: <4335d2c40708201232s19b10c69ye44d39351a4da97d@mail.gmail.com>

On 8/20/07, Brett Cannon <brett at python.org> wrote:
> I believe email addresses are automatically obfuscated as part of the
> HTML generation process, but one of the PEP editors can correct me if
> I am wrong.

Yes, email addresses are obfuscated in PEPs.

For example, in PEPs 0 & 12, my address is encoded as
"goodger&#32;&#97;t&#32;python.org" (the "@" is changed to " at " and
further obfuscated from there).  More tricks could be played, but that
would only decrease the usefulness of addresses for legitimate
purposes.

Spam is a fact of life.  People just have to deal with it.

-- 
David Goodger <http://python.net/~goodger>

From janssen at parc.com  Mon Aug 20 21:33:28 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 20 Aug 2007 12:33:28 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <46C9D079.1040009@v.loewis.de> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
	<46C9D079.1040009@v.loewis.de>
Message-ID: <07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>

> I view TLS as a wrapper around / layer on top of TCP, and so I think the
> API should look like, as well.

I think Martin raises a valid point here, which should at least be
discussed more thoroughly.  Should there be an "SSL socket", which is
just like a regular socket?  Does that really provide any additional
functionality to anyone?  Most apps and classes that use TCP sockets
wrap the socket with socket._fileobject() to get "read" and "write",
anyway -- can't they just wrap it with socket.ssl instead?

Perhaps in the sprint, I should just concentrate on widening the
"socket.ssl" interface a bit, and improving the functionality of the
SSLObject a bit.

Suggested improvements:

  *  Allow server-side operation.

  *  Allow specification of particular SSL protocol version.

  *  Allow certificate validation.  This is a bit tricky; typically
     certs are validated against some database of root certificates, so you
     need a whole infrastructure to maintain that database.  Currently, we
     don't have one, so no certs can be validated.  We could add a switch
     to allow auto-validation of self-signed certs pretty easily.  I could
     add a parameter to the SSLObject constructor which would be a filepath
     for a file full of root certs (see SSL_CTX_load_verify_locations(3ssl)).

  *  Add a method to retrieve the other side's certificate info.  What's
     a good format for the "notBefore" and "notAfter" dates?  The simplest
     thing to use is the string formatting OpenSSL provides, which is
     always like "Sep 29 16:38:04 2006 GMT", which can easily be parsed
     by the time.strptime() function if the user wants something else.
     On the other hand, figuring out how to use strptime() is always a
     pain, so providing a convenience function wouldn't be a terrible idea.

  *  Add a shutdown() method to stop using SSL on the underlying socket
     without closing the socket.

  *  Make SSLObject conform to the Raw I/O API in PEP 3116.  This one is
     interesting; what should close() do?  Close the underlying socket?  Or
     just do an SSL shutdown?

Bill


From amauryfa at gmail.com  Mon Aug 20 23:07:52 2007
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Mon, 20 Aug 2007 23:07:52 +0200
Subject: [Python-Dev] Cannot build new documentation
Message-ID: <e27efe130708201407o716a8fc4ha4c1cce2ad883fb5@mail.gmail.com>

Hello,

The new documentation system is really a major improvement: congratulations!

However, I run into a problem while trying to build the new python docs.
I initially used python2.5 on windows, but Linux seems to have the same problem.
Am I really the only one who gets this error?

The offending code is in Doc\tools\sphinx\builder.py, and looks like this:

>>> from __future__ import with_statement
>>> import codecs
>>> with codecs.open('foo.txt', 'w', 'utf-8') as fp:
...     print type(fp), fp
...     fp.write(u"\xb6")
...
<type 'file'> <open file 'foo.txt', mode 'wb' at 0x00C649E8>
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb6' in
position 0: ordinal not in range(128)

Where does the 'ascii' codec come from? I propose the following explanation:

- codecs.open returns a wrapped stream (a codec.StreamReaderWriter).
This object implements some methods (read, write...) and delegates the
others to the underlying file object.
- 'with .. as ..' calls the __enter__ method, and assigns the result to fp.
- but StreamReaderWriter does not define __enter__, so file.__enter__
is called instead
and fp actually references the underlying file!

An obvious workaround is to add the __enter__ method to StreamReaderWriter:
def __enter__(self):
    return self
This is not perfect though, because one may want to wrap say a
socket.makefile instead of a file.

It seems like the delegation pattern does not mix well with context managers...
Is there another solution?
Or did I miss something obvious?

-- 
Amaury Forgeot d'Arc

From martin at v.loewis.de  Tue Aug 21 00:08:38 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 21 Aug 2007 00:08:38 +0200
Subject: [Python-Dev] Cannot build new documentation
In-Reply-To: <e27efe130708201407o716a8fc4ha4c1cce2ad883fb5@mail.gmail.com>
References: <e27efe130708201407o716a8fc4ha4c1cce2ad883fb5@mail.gmail.com>
Message-ID: <46CA10E6.2020106@v.loewis.de>

> However, I run into a problem while trying to build the new python docs.
> I initially used python2.5 on windows, but Linux seems to have the same problem.
> Am I really the only one who gets this error?

Not sure - but it works fine for me.

>>>> from __future__ import with_statement
>>>> import codecs
>>>> with codecs.open('foo.txt', 'w', 'utf-8') as fp:
> ...     print type(fp), fp
> ...     fp.write(u"\xb6")
> ...
> <type 'file'> <open file 'foo.txt', mode 'wb' at 0x00C649E8>

That is surprising. Are you sure codecs.open is what it should be
on your system? I get

<type 'instance'> <open file 'foo.txt', mode 'wb' at 0xb7d89e78>

> Where does the 'ascii' codec come from? I propose the following explanation:
> 
> - codecs.open returns a wrapped stream (a codec.StreamReaderWriter).
> This object implements some methods (read, write...) and delegates the
> others to the underlying file object.
> - 'with .. as ..' calls the __enter__ method, and assigns the result to fp.
> - but StreamReaderWriter does not define __enter__, so file.__enter__
> is called instead
> and fp actually refrences the underlying file!

Good explanation. I'm using Python 2.5.1, and this may be relevant:

------------------------------------------------------------------------
r52518 | georg.brandl | 2006-10-29 09:39:27 +0100 (So, 29 Okt 2006) | 4
lines

Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and
fix all codecs file wrappers to work correctly with the "with"
statement (bug #1586513).
 (backport from rev. 52517)

> It seems like the delegation pattern does not mix well with context managers...
> Is there another solution?
> Or did I miss something obvious?

I think the obvious thing you missed is that the problem got fixed
already. Whether the documentation system should be more defensive and
work with 2.5.0 also is a different question.

Regards,
Martin

From amauryfa at gmail.com  Tue Aug 21 00:26:51 2007
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 21 Aug 2007 00:26:51 +0200
Subject: [Python-Dev]  Cannot build new documentation
In-Reply-To: <e27efe130708201525t73fdf599i939d27062b934b00@mail.gmail.com>
References: <e27efe130708201407o716a8fc4ha4c1cce2ad883fb5@mail.gmail.com>
	<46CA10E6.2020106@v.loewis.de>
	<e27efe130708201525t73fdf599i939d27062b934b00@mail.gmail.com>
Message-ID: <e27efe130708201526y5b3e35a5n45ece48a9e67c859@mail.gmail.com>

Hello,

Martin v. L?wis wrote:
> I think the obvious thing you missed is that the problem got fixed
> already. Whether the documentation system should be more defensive and
> work with 2.5.0 also is a different question.

<red of confusion>You are right. python 2.5.1 corrected this.
May I still dare propose the following patch:

Index: Doc/tools/sphinx-build.py
===================================================================
--- Doc/tools/sphinx-build.py   (revision 57158)
+++ Doc/tools/sphinx-build.py   (working copy)
@@ -11,9 +11,9 @@

 if __name__ == '__main__':

-    if sys.version_info[:3] < (2, 5, 0):
+    if sys.version_info[:3] < (2, 5, 1):
         print >>sys.stderr, """\
-Error: Sphinx needs to be executed with Python 2.5 or newer.
+Error: Sphinx needs to be executed with Python 2.5.1 or newer.
 (If you run this from the Makefile, you can set the PYTHON variable
 to the path of an alternative interpreter executable.)
 """

--
Amaury Forgeot d'Arc

From steve at holdenweb.com  Tue Aug 21 02:17:36 2007
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 20 Aug 2007 20:17:36 -0400
Subject: [Python-Dev] Cygwin: Problem detecting subprocess termination
 after _spawn_posix in distutils?
In-Reply-To: <46B06554.6020501@holdenweb.com>
References: <f8lcd3$9ou$1@sea.gmane.org>	<46AFAD41.9060203@v.loewis.de>	<46AFDF97.3090909@holdenweb.com>	<46B01E90.8090402@v.loewis.de>
	<46B06554.6020501@holdenweb.com>
Message-ID: <46CA2F20.5050002@holdenweb.com>

Steve Holden wrote:
> Martin v. L?wis wrote:
>>> The only environment variables that don't appear in the shell output 
>>> from the env command are INFOPATH, MAKE_MODE and PLAT. I am still flummoxed.
>> At this point, I'd recommend to perform a cygwin update; with Cygwin,
>> these problems often go away with an update.
>>
> I updated Cygwin and did a rebaseall before posting.
> 
>> If that doesn't help, you can ask on the Cygwin list also; to analyse
>> this further, ISTM one will need to debug the internals of cygwin.
>>
> I posted on Cygwin before asking here.
> 
>> One thing you could try is to add -v to the list of gcc options;
>> you can then see whether gcc is progressing correctly.
>>
> I'll do that, though I have reason to believe the gcc *is* terminating 
> and _spawn_posix isn't detecting the end of the process. At the very 
> least we should get another test out of this dreadfully irritating bug.
> 
> Thanks again for looking at this.
> 
A further data point (sorry, I've not had a lot of time to look at 
this). Building Python from the trunk at rev 57076 yielded a system that 
*would* cleanly build the same release of PIL. This is somewhat annoying 
because it appears to put the bug in the past where there's less 
incentive to investigate it. But I may find time to take a look.

Hardly worth it if I'm the only one noticing the issue, though.

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC/Ltd           http://www.holdenweb.com
Skype: holdenweb      http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------


From janssen at parc.com  Tue Aug 21 02:39:39 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 20 Aug 2007 17:39:39 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug20.123330pdt."57996"@synergy1.parc.xerox.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
	<46C9D079.1040009@v.loewis.de>
	<07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug20.173941pdt."57996"@synergy1.parc.xerox.com>

>   *  Allow certificate validation.  This is a bit tricky; typically
>      certs are validated against some database of root certificates, so you
>      need a whole infrastructure to maintain that database.  Currently, we
>      don't have one, so no certs can be validated.  We could add a switch
>      to allow auto-validation of self-signed certs pretty easily.  I could
>      add a parameter to the SSLObject constructor which would be a filepath
>      for a file full of root certs (see SSL_CTX_load_verify_locations(3ssl)).

The simplest way to do verification is to allow the application to
provide a set of root certs that it would like to verify against, and
use the built-in OpenSSL verification procedure.

The OpenSSL CAcerts file format is just a number of certificates
concatenated together, separated by text headers that identify the
boundary:

-----BEGIN CERTIFICATE-----
 ... (CA certificate in base64 encoding) ...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
 ... (another CA certificate in base64 encoding) ...
-----END CERTIFICATE-----
 ...

I suggest we just use that.  Applications which want something fancier
are free to implement something :-).

Bill



From andrew-pythondev at puzzling.org  Tue Aug 21 02:57:19 2007
From: andrew-pythondev at puzzling.org (Andrew Bennetts)
Date: Tue, 21 Aug 2007 10:57:19 +1000
Subject: [Python-Dev] Segfault
In-Reply-To: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
Message-ID: <20070821005719.GB24805@steerpike.home.puzzling.org>

Maciej Fijalkowski wrote:
> IMHO this shouldn't segfault:
> 
> import thread
> 
> while 1:
>     f = open("/tmp/dupa", "w")
>     thread.start_new_thread(f.close, ())
>     f.close()
> 
> while it does on cpython 2.5.1 , linux box.
> 
> May I consider this a bug?

Yes, that's a bug.  Please file it at
http://sourceforge.net/bugs/?group_id=5470.

-Andrew.


From brett at python.org  Tue Aug 21 05:57:17 2007
From: brett at python.org (Brett Cannon)
Date: Mon, 20 Aug 2007 20:57:17 -0700
Subject: [Python-Dev] How to interpret get_code from PEP 302?
Message-ID: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>

PEP 302 ("New Import Hooks") has an optional extensions section so
that tools like py2exe and py2app have an easier time.  Part of the
optional extensions is the method get_code that is to return the code
object for the specified method (if the loader can handle it).

But there is a lack in the definition of how get_code is supposed to
be implemented.  The definition says that the "method should return
the code object associated with the module", which is fine.  But then
it goes on to state that "If the loader doesn't have the code object
but it _does_ have the source code, it should return the compiled
source code".  This throws me as this makes it sound like bytecode
does not need to be used if the loader does not already have a code
object and there is no source to be had; any bytecode can be ignored.

Now I doubt this is how it is supposed to be read.  Does anyone
disagree with that?  If not, I will change the wording to mention that
bytecode must be used if no source is available (and that the magic
number must be verified).

-Brett

From guido at python.org  Tue Aug 21 06:10:16 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 20 Aug 2007 21:10:16 -0700
Subject: [Python-Dev] How to interpret get_code from PEP 302?
In-Reply-To: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>
References: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>
Message-ID: <ca471dc20708202110x4e2a3bd7me79a812699249c8d@mail.gmail.com>

On 8/20/07, Brett Cannon <brett at python.org> wrote:
> PEP 302 ("New Import Hooks") has an optional extensions section so
> that tools like py2exe and py2app have an easier time.  Part of the
> optional extensions is the method get_code that is to return the code
> object for the specified method (if the loader can handle it).
>
> But there is a lack in the definition of how get_code is supposed to
> be implemented.  The definition says that the "method should return
> the code object associated with the module", which is fine.  But then
> it goes on to state that "If the loader doesn't have the code object
> but it _does_ have the source code, it should return the compiled
> source code".  This throws me as this makes it sound like bytecode
> does not need to be used if the loader does not already have a code
> object and there is no source to be had; any bytecode can be ignored.
>
> Now I doubt this is how it is supposed to be read.  Does anyone
> disagree with that?  If not, I will change the wording to mention that
> bytecode must be used if no source is available (and that the magic
> number must be verified).

The only reading that makes sense is that it should always return a
code object (i.e. always byte code) and thet it can either get the
byte code by unmarshaling the equivalent of a .pyc file (after
skipping the first 8 bytes) *or* by calling compile() on the source
code. So, in the specific case of a zip file, it should first look for
a .pyc file, and then for a .py file. This is the opposite of how
things work on the filesystem, where it first looks for a .py, and
then for a .pyc; if both exist, the .pyc is only used if its mtime
matches the timestamp in the .pyc header. (But why am I explaining
this to you? :-) The idea for zip files is that quick loading is
preferred, and that the source is only used if no .pyc is found at all
(or whatever convention is used). This makes it easier to have a zip
that contains both .py and .pyc files -- the .py files are only used
for reference (e.g. one could imagine using them for tracebacks).

But maybe I'm missing a subtlety in your description of the issue?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From g.brandl at gmx.net  Tue Aug 21 07:58:07 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 21 Aug 2007 07:58:07 +0200
Subject: [Python-Dev] Cannot build new documentation
In-Reply-To: <e27efe130708201526y5b3e35a5n45ece48a9e67c859@mail.gmail.com>
References: <e27efe130708201407o716a8fc4ha4c1cce2ad883fb5@mail.gmail.com>	<46CA10E6.2020106@v.loewis.de>	<e27efe130708201525t73fdf599i939d27062b934b00@mail.gmail.com>
	<e27efe130708201526y5b3e35a5n45ece48a9e67c859@mail.gmail.com>
Message-ID: <faduta$7qt$1@sea.gmane.org>

Amaury Forgeot d'Arc schrieb:
> Hello,
> 
> Martin v. L?wis wrote:
>> I think the obvious thing you missed is that the problem got fixed
>> already. Whether the documentation system should be more defensive and
>> work with 2.5.0 also is a different question.
> 
> <red of confusion>You are right. python 2.5.1 corrected this.
> May I still dare propose the following patch:
> 
> Index: Doc/tools/sphinx-build.py
> ===================================================================
> --- Doc/tools/sphinx-build.py   (revision 57158)
> +++ Doc/tools/sphinx-build.py   (working copy)
> @@ -11,9 +11,9 @@
> 
>  if __name__ == '__main__':
> 
> -    if sys.version_info[:3] < (2, 5, 0):
> +    if sys.version_info[:3] < (2, 5, 1):
>          print >>sys.stderr, """\
> -Error: Sphinx needs to be executed with Python 2.5 or newer.
> +Error: Sphinx needs to be executed with Python 2.5.1 or newer.
>  (If you run this from the Makefile, you can set the PYTHON variable
>  to the path of an alternative interpreter executable.)
>  """

Yes, this is reasonable.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From martin at v.loewis.de  Tue Aug 21 08:57:09 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 21 Aug 2007 08:57:09 +0200
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug20.173941pdt."57996"@synergy1.parc.xerox.com>
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>	<46C9C9C3.9090201@v.loewis.de>	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>	<46C9D079.1040009@v.loewis.de>	<07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.173941pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <46CA8CC5.70804@v.loewis.de>

> The simplest way to do verification is to allow the application to
> provide a set of root certs that it would like to verify against, and
> use the built-in OpenSSL verification procedure.

That's good. I don't recall whether you planned for this, however,
it would then be necessary to find out who the authenticated user
is, to do authorization. Getting that as a pair (client dn, issuer dn)
is the interface that springs to mind first.

Regards,
Martin

From thomas at python.org  Tue Aug 21 10:23:51 2007
From: thomas at python.org (Thomas Wouters)
Date: Tue, 21 Aug 2007 10:23:51 +0200
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <f9t2nn$ksg$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org>
Message-ID: <9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>

On 8/14/07, Georg Brandl <g.brandl at gmx.net> wrote:
>
> Now that the converted documentation is fairly bug-free, I want to
> make the switch.


One thing I miss (and I haven't followed the discussions about the new
layout at all, so I don't know if it came up already) is section numbers. I
very often refer people to "Section 4.7 of the tutorial at
docs.python.org/tut" or "Appendix B of the tutorial" or "Section 3.6.2 of
the library reference", which the new layout doesn't exactly permit. I also
think the index and section headers are less clear, more chaotic, without
the numbering. Any chance we can get something like it back? :)

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070821/23e2da63/attachment.htm 

From hrvoje.niksic at avl.com  Tue Aug 21 10:13:48 2007
From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=)
Date: Tue, 21 Aug 2007 10:13:48 +0200
Subject: [Python-Dev] Segfault
In-Reply-To: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
Message-ID: <1187684028.11322.18.camel@localhost>

On Mon, 2007-08-20 at 20:27 +0200, Maciej Fijalkowski wrote:
> import thread
> 
> while 1:
>     f = open("/tmp/dupa", "w")
>     thread.start_new_thread(f.close, ())
>     f.close()

file_close inadvertently allows fclose to be called twice on the same
stdio file.  This patch should fix the problem:

Python-2.5.1/Objects/fileobject.c
--- Python-2.5.1.orig/Objects/fileobject.c	2007-01-23 14:54:30.000000000 +0100
+++ Python-2.5.1/Objects/fileobject.c	2007-08-21 10:04:18.000000000 +0200
@@ -440,13 +440,14 @@
 {
 	int sts = 0;
 	if (f->f_fp != NULL) {
+		FILE *fp = f->f_fp;
+		f->f_fp = NULL;
 		if (f->f_close != NULL) {
 			Py_BEGIN_ALLOW_THREADS
 			errno = 0;
-			sts = (*f->f_close)(f->f_fp);
+			sts = (*f->f_close)(fp);
 			Py_END_ALLOW_THREADS
 		}
-		f->f_fp = NULL;
 	}
 	PyMem_Free(f->f_setbuf);
 	f->f_setbuf = NULL;



From ronaldoussoren at mac.com  Tue Aug 21 10:50:46 2007
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 21 Aug 2007 01:50:46 -0700
Subject: [Python-Dev] Segfault
In-Reply-To: <1187684028.11322.18.camel@localhost>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
Message-ID: <74EDC975-0114-1000-B3BC-48599676C42C-Webmail-10021@mac.com>

Please create a patch for this in the tracker to ensure that this patch doesn't get lost.

Ronald
 
On Tuesday, August 21, 2007, at 10:47AM, "Hrvoje Nik?i?" <hrvoje.niksic at avl.com> wrote:
>On Mon, 2007-08-20 at 20:27 +0200, Maciej Fijalkowski wrote:
>> import thread
>> 
>> while 1:
>>     f = open("/tmp/dupa", "w")
>>     thread.start_new_thread(f.close, ())
>>     f.close()
>
>file_close inadvertently allows fclose to be called twice on the same
>stdio file.  This patch should fix the problem:
>
>Python-2.5.1/Objects/fileobject.c
>--- Python-2.5.1.orig/Objects/fileobject.c	2007-01-23 14:54:30.000000000 +0100
>+++ Python-2.5.1/Objects/fileobject.c	2007-08-21 10:04:18.000000000 +0200
>@@ -440,13 +440,14 @@
> {
> 	int sts = 0;
> 	if (f->f_fp != NULL) {
>+		FILE *fp = f->f_fp;
>+		f->f_fp = NULL;
> 		if (f->f_close != NULL) {
> 			Py_BEGIN_ALLOW_THREADS
> 			errno = 0;
>-			sts = (*f->f_close)(f->f_fp);
>+			sts = (*f->f_close)(fp);
> 			Py_END_ALLOW_THREADS
> 		}
>-		f->f_fp = NULL;
> 	}
> 	PyMem_Free(f->f_setbuf);
> 	f->f_setbuf = NULL;
>
>
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
>
>

From p.f.moore at gmail.com  Tue Aug 21 14:40:51 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 21 Aug 2007 13:40:51 +0100
Subject: [Python-Dev] How to interpret get_code from PEP 302?
In-Reply-To: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>
References: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>
Message-ID: <79990c6b0708210540s6851164didc5413d14e4f8090@mail.gmail.com>

On 21/08/07, Brett Cannon <brett at python.org> wrote:
> PEP 302 ("New Import Hooks") has an optional extensions section so
> that tools like py2exe and py2app have an easier time.  Part of the
> optional extensions is the method get_code that is to return the code
> object for the specified method (if the loader can handle it).
>
> But there is a lack in the definition of how get_code is supposed to
> be implemented.  The definition says that the "method should return
> the code object associated with the module", which is fine.  But then
> it goes on to state that "If the loader doesn't have the code object
> but it _does_ have the source code, it should return the compiled
> source code".  This throws me as this makes it sound like bytecode
> does not need to be used if the loader does not already have a code
> object and there is no source to be had; any bytecode can be ignored.
>
> Now I doubt this is how it is supposed to be read.  Does anyone
> disagree with that?  If not, I will change the wording to mention that
> bytecode must be used if no source is available (and that the magic
> number must be verified).

Hmm, yes, that's muddled. Maybe it made sense to me when I wrote it
:-) (I think it was my wording rather than Just's)

get_code must *always* return the same code object that
loader.load_module is using - whether that be bytecode or compiled
source (and it must respect things like file timestamps where
appropriate just like load_module does). What the sentence you quote
is trying to say is that if there's a need to compile source, the
get_code method must do this on behalf of the caller - it can't return
None and expect the caller to try get_source and compile it manually.
Someone who only wants a code object should never need to call
get_source.

I'm not sure that's any clearer! If you need further clarification,
let me know (either on or off list). I'd appreciate it if you can
clear the PEP's wording up.

Paul.

From facundobatista at gmail.com  Tue Aug 21 14:52:08 2007
From: facundobatista at gmail.com (Facundo Batista)
Date: Tue, 21 Aug 2007 09:52:08 -0300
Subject: [Python-Dev] RFC - proposal for urilib. unified module of
	urlparse, urllib and urllib2
In-Reply-To: <20070819191031.GD3437@gmail.com>
References: <20070819191031.GD3437@gmail.com>
Message-ID: <e04bdf310708210552y68bddde2ya351a34a547f489@mail.gmail.com>

2007/8/19, O.R.Senthil Kumaran <orsenthil at users.sourceforge.net>:

> I am drafting a PEP proposing a module 'urilib', which will be the unified module of urlparse, urllib and urllib2.

Great!!


> a) _all_ functions will include from urlparse,urllib and urllib2.
> b) overlapping functionality between urllib and urllib2 to be ironed out.
> c) RFC3986 and RFC3987 compliant.

Seems good to me. Maybe something from cgi? I always use
cgi.parse_qs() when using urrlib2... Something about this arised in
other thread...


> I would like to know your thoughts on
> - How should we plan to maintain the backward compatiblity?
> - What other features should be kept in mind while designing?

First of all you should draft the PEP, send it here, and put it
somewhere in the web. Then the community could help you to shape it
and get a final proposal.

If too many problems regardgin compatibility arises, you always can
implement it in Py3k...

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From p.f.moore at gmail.com  Tue Aug 21 15:25:34 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 21 Aug 2007 14:25:34 +0100
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>
References: <f9t2nn$ksg$1@sea.gmane.org>
	<9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>
Message-ID: <79990c6b0708210625h7c763a21h619207a8e95d1070@mail.gmail.com>

On 21/08/07, Thomas Wouters <thomas at python.org> wrote:
>
>
> On 8/14/07, Georg Brandl <g.brandl at gmx.net> wrote:
> > Now that the converted documentation is fairly bug-free, I want to
> > make the switch.
>
> One thing I miss (and I haven't followed the discussions about the new
> layout at all, so I don't know if it came up already) is section numbers.

On a partly related note, I am assuming that the Windows Python
installer will come with the documentation in .chm (HTML Help) format,
as previously. It would be nice to see a preview version of this, as
I'm not 100% clear how the new section layout maps into the HTML Help
contents pane (as a specific example, the new "Built In Types" section
of the library reference is a single page - does this imply that the
HTML help won't have any subsections? I often use the HTML Help
contents pane to drill all the way down to (for example) section 3.6.1
"String Methods" and click to get directly to that section.

Paul.

From g.brandl at gmx.net  Tue Aug 21 15:40:00 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 21 Aug 2007 15:40:00 +0200
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>
References: <f9t2nn$ksg$1@sea.gmane.org>
	<9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>
Message-ID: <faepvb$eqm$1@sea.gmane.org>

Thomas Wouters schrieb:
> 
> 
> On 8/14/07, *Georg Brandl* <g.brandl at gmx.net <mailto:g.brandl at gmx.net>>
> wrote:
> 
>     Now that the converted documentation is fairly bug-free, I want to
>     make the switch.
> 
> 
> One thing I miss (and I haven't followed the discussions about the new
> layout at all, so I don't know if it came up already) is section
> numbers. I very often refer people to "Section 4.7 of the tutorial at
> docs.python.org/tut <http://docs.python.org/tut>" or "Appendix B of the
> tutorial" or "Section 3.6.2 of the library reference", which the new
> layout doesn't exactly permit. I also think the index and section
> headers are less clear, more chaotic, without the numbering. Any chance
> we can get something like it back? :)

You're right, it can be helpful. I'll look into it, how much work it
probably would be.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From g.brandl at gmx.net  Tue Aug 21 15:42:57 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 21 Aug 2007 15:42:57 +0200
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <79990c6b0708210625h7c763a21h619207a8e95d1070@mail.gmail.com>
References: <f9t2nn$ksg$1@sea.gmane.org>	<9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>
	<79990c6b0708210625h7c763a21h619207a8e95d1070@mail.gmail.com>
Message-ID: <faeq4s$eqm$2@sea.gmane.org>

Paul Moore schrieb:
> On 21/08/07, Thomas Wouters <thomas at python.org> wrote:
>>
>>
>> On 8/14/07, Georg Brandl <g.brandl at gmx.net> wrote:
>> > Now that the converted documentation is fairly bug-free, I want to
>> > make the switch.
>>
>> One thing I miss (and I haven't followed the discussions about the new
>> layout at all, so I don't know if it came up already) is section numbers.
> 
> On a partly related note, I am assuming that the Windows Python
> installer will come with the documentation in .chm (HTML Help) format,
> as previously. It would be nice to see a preview version of this, as
> I'm not 100% clear how the new section layout maps into the HTML Help
> contents pane (as a specific example, the new "Built In Types" section
> of the library reference is a single page - does this imply that the
> HTML help won't have any subsections? I often use the HTML Help
> contents pane to drill all the way down to (for example) section 3.6.1
> "String Methods" and click to get directly to that section.

I wasn't sure how I had implemented the contents pane, so I've now created
a HTML help version ("make htmlhelp" in python/Doc already does that;
you only need to run HTML Help Workshop over it), and the "String Methods"
indeed have their own contents tree item.

I put the CHM at <http://www.python.org/~gbrandl/pydoc.chm>, if you want
to have a look.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From benji at benjiyork.com  Tue Aug 21 16:03:34 2007
From: benji at benjiyork.com (Benji York)
Date: Tue, 21 Aug 2007 10:03:34 -0400
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <faeq4s$eqm$2@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org>	<9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>	<79990c6b0708210625h7c763a21h619207a8e95d1070@mail.gmail.com>
	<faeq4s$eqm$2@sea.gmane.org>
Message-ID: <46CAF0B6.30506@benjiyork.com>

Georg Brandl wrote:
> I put the CHM at <http://www.python.org/~gbrandl/pydoc.chm>, if you want
> to have a look.

Generally looks good.  I did get this error when opening the CHM:

"""
A Runtime Error has occurred.
Do you with to Debug?

Line: 236
Error: Expected identifier, string or number
"""

Nothing happened when I clicked "Yes".  The error is repeated each time 
a new page is opened.

An additional opinion:  The red, appear-on-hover permalink markers don't 
make much sense in a CHM and are somewhat distracting.  Can those be 
hidden, perhaps via CSS?
-- 
Benji York
http://benjiyork.com

From g.brandl at gmx.net  Tue Aug 21 16:12:17 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 21 Aug 2007 16:12:17 +0200
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <46CAF0B6.30506@benjiyork.com>
References: <f9t2nn$ksg$1@sea.gmane.org>	<9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>	<79990c6b0708210625h7c763a21h619207a8e95d1070@mail.gmail.com>	<faeq4s$eqm$2@sea.gmane.org>
	<46CAF0B6.30506@benjiyork.com>
Message-ID: <faerrq$m3e$1@sea.gmane.org>

Benji York schrieb:
> Georg Brandl wrote:
>> I put the CHM at <http://www.python.org/~gbrandl/pydoc.chm>, if you want
>> to have a look.
> 
> Generally looks good.  I did get this error when opening the CHM:
> 
> """
> A Runtime Error has occurred.
> Do you with to Debug?
> 
> Line: 236
> Error: Expected identifier, string or number
> """
> 
> Nothing happened when I clicked "Yes".  The error is repeated each time 
> a new page is opened.

Having the JS in the CHM makes no sense anyway, I'll just leave it out.

> An additional opinion:  The red, appear-on-hover permalink markers don't 
> make much sense in a CHM and are somewhat distracting.  Can those be 
> hidden, perhaps via CSS?

Sure.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ncoghlan at gmail.com  Tue Aug 21 17:01:11 2007
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 22 Aug 2007 01:01:11 +1000
Subject: [Python-Dev] How to interpret get_code from PEP 302?
In-Reply-To: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>
References: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>
Message-ID: <46CAFE37.8010605@gmail.com>

Brett Cannon wrote:
> PEP 302 ("New Import Hooks") has an optional extensions section so
> that tools like py2exe and py2app have an easier time.  Part of the
> optional extensions is the method get_code that is to return the code
> object for the specified method (if the loader can handle it).
> 
> But there is a lack in the definition of how get_code is supposed to
> be implemented.  The definition says that the "method should return
> the code object associated with the module", which is fine.  But then
> it goes on to state that "If the loader doesn't have the code object
> but it _does_ have the source code, it should return the compiled
> source code".  This throws me as this makes it sound like bytecode
> does not need to be used if the loader does not already have a code
> object and there is no source to be had; any bytecode can be ignored.

It also goes on to say *why* this requirement is there:
"(This is so that our caller doesn't also need to check get_source()
  if all it needs is the code object.)"

In terms of *what* get_code() returns, all it needs to do is accurately 
reflect what load_module() actually runs, but it may not be the exact 
same code object (e.g. if the code object isn't cached anywhere by the 
loader, then it will be recreated anew for each call to get_code() or 
load_module()).

> Now I doubt this is how it is supposed to be read.  Does anyone
> disagree with that?  If not, I will change the wording to mention that
> bytecode must be used if no source is available (and that the magic
> number must be verified).

The magic number belongs to the pyc file, not the code object. It's up 
to the loader to do whatever checking is needed to decide whether or not 
the code object is sane for the current environment (and it makes no 
difference whether this is done for get_code() or for an actual 
load_module() call).

If load_module() would use some cached bytecode (e.g. a pyc file) in 
preference to recompiling from source (e.g. a py file), then get_code() 
should do the same thing. On the other hand, if load_module() has a 
means for detecting stale bytecode and will recompile in that case, then 
the same check is also needed in get_code().

The important thing is for load_module() and get_code() to be consistent 
  - beyond that, there are no requirements regarding how the source and 
bytecode are stored externally (if they're stored at all). For example, 
you could write a perfectly valid loader which ignored .pyo and .pyc 
files entirely, and always recompiled from the source file (it would be 
slower than the normal one, but it would work).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From p.f.moore at gmail.com  Tue Aug 21 17:01:58 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 21 Aug 2007 16:01:58 +0100
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <46CAF0B6.30506@benjiyork.com>
References: <f9t2nn$ksg$1@sea.gmane.org>
	<9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>
	<79990c6b0708210625h7c763a21h619207a8e95d1070@mail.gmail.com>
	<faeq4s$eqm$2@sea.gmane.org> <46CAF0B6.30506@benjiyork.com>
Message-ID: <79990c6b0708210801q6cff70cfpce672235dc9390c8@mail.gmail.com>

On 21/08/07, Benji York <benji at benjiyork.com> wrote:
> Georg Brandl wrote:
> > I put the CHM at <http://www.python.org/~gbrandl/pydoc.chm>, if you want
> > to have a look.
>
> Generally looks good.  I did get this error when opening the CHM:
>
> """
> A Runtime Error has occurred.
> Do you with to Debug?
>
> Line: 236
> Error: Expected identifier, string or number
> """
>
> Nothing happened when I clicked "Yes".  The error is repeated each time
> a new page is opened.

I got that as well, in index.html line 264 character 7. I suspect it's
one of the included js files (which I can't get access to).

> An additional opinion:  The red, appear-on-hover permalink markers don't
> make much sense in a CHM and are somewhat distracting.  Can those be
> hidden, perhaps via CSS?

That sounds reasonable.

But I'd agree with Benji that the new CHM file looks fine! Thanks for
all your work on this.

Paul.

From janssen at parc.com  Tue Aug 21 17:11:28 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 21 Aug 2007 08:11:28 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <46CA8CC5.70804@v.loewis.de> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
	<46C9D079.1040009@v.loewis.de>
	<07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.173941pdt."57996"@synergy1.parc.xerox.com>
	<46CA8CC5.70804@v.loewis.de>
Message-ID: <07Aug21.081137pdt."57996"@synergy1.parc.xerox.com>

> > The simplest way to do verification is to allow the application to
> > provide a set of root certs that it would like to verify against, and
> > use the built-in OpenSSL verification procedure.
> 
> That's good. I don't recall whether you planned for this, however,
> it would then be necessary to find out who the authenticated user
> is, to do authorization. Getting that as a pair (client dn, issuer dn)
> is the interface that springs to mind first.

Yes, that's right.  If the cert verifies, its details are then
available, as a mapping, something like this:

{'notBefore': 'Sep 29 16:38:04 2006 GMT',
 'notAfter': 'Dec  7 16:38:04 2008 GMT',
 'issuer':
    {'organizationalUnitName': u'UpLib',
     'organizationName': u'PARC',
     'commonName': u'wolfe-64.parc.xerox.com',
     'stateOrProvinceName': u'California',
     'countryName': u'US',
     'localityName': u'Palo Alto'},
 'version': 2,
 'subject':
    {'organizationalUnitName': u'UpLib',
     'organizationName': u'PARC',
     'commonName': u'wolfe-64.parc.xerox.com',
     'stateOrProvinceName': u'California',
     'countryName': u'US',
     'localityName': u'Palo Alto'}
 }

This is a self-signed cert, and it's still an open question whether
they should verify, and under what circumstances.  I'm currently
thinking that in the CERT_OPTIONAL regime, they could, but with
CERT_REQUIRED, they shouldn't.

Bill

From janssen at parc.com  Tue Aug 21 17:30:23 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 21 Aug 2007 08:30:23 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug21.081137pdt."57996"@synergy1.parc.xerox.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
	<46C9D079.1040009@v.loewis.de>
	<07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.173941pdt."57996"@synergy1.parc.xerox.com>
	<46CA8CC5.70804@v.loewis.de>
	<07Aug21.081137pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug21.083032pdt."57996"@synergy1.parc.xerox.com>

> This is a self-signed cert, and it's still an open question whether
> they should verify, and under what circumstances.  I'm currently
> thinking that in the CERT_OPTIONAL regime, they could, but with
> CERT_REQUIRED, they shouldn't.

If an application wanted self-signed certs to verify under
CERT_REQUIRED, they could add them to their rootcerts file.

Bill

From nnorwitz at gmail.com  Tue Aug 21 18:14:37 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 21 Aug 2007 09:14:37 -0700
Subject: [Python-Dev] Segfault
In-Reply-To: <1187684028.11322.18.camel@localhost>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
Message-ID: <ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>

On 8/21/07, Hrvoje Nik?i? <hrvoje.niksic at avl.com> wrote:
> On Mon, 2007-08-20 at 20:27 +0200, Maciej Fijalkowski wrote:
> > import thread
> >
> > while 1:
> >     f = open("/tmp/dupa", "w")
> >     thread.start_new_thread(f.close, ())
> >     f.close()
>
> file_close inadvertently allows fclose to be called twice on the same
> stdio file.  This patch should fix the problem:

The patch is insufficient to prevent all types of crashes that occur
when accessing a file from 2 threads (closing in one and doing
whatever in another).  Almost every place that accesses f_fp is a
problem.  For example, try changing one of the f.close to f.tell.  Or
in the main thread do a write.  With all of these changes to the test
above, I was able to crash python with the patch (until I fixed the
uses of f_fp).

For example:

import thread

while 1:
    f = open("/tmp/dupa", "w")
    thread.start_new_thread(f.close, ())
    try:
      f.write('a')
      f.close()
    except: pass

I've attached a patch (against 2.5) that fixes most of the problems,
but there are still a few more.  (Search for Py_BEGIN_ALLOW_THREADS
and check for usage of f_fp within the block.)  I'm not convinced the
attached patch is good enough though.

I think there might already be a bug report about file not being
thread-safe.  (It could have also been socket, but I think Martin
fixed a problem in socket a while ago.)

n

From graham.horler at gmail.com  Tue Aug 21 19:13:59 2007
From: graham.horler at gmail.com (Graham Horler)
Date: Tue, 21 Aug 2007 17:13:59 +0000
Subject: [Python-Dev] More on server-side SSL support
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
	<46C9D079.1040009@v.loewis.de>
	<07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.173941pdt."57996"@synergy1.parc.xerox.com>
	<46CA8CC5.70804@v.loewis.de>
	<07Aug21.081137pdt."57996"@synergy1.parc.xerox.com>
	<07Aug21.083032pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <20070821161359.E6F17D4439@mra02.ch.as12513.net>

+1 for mutual authentication, I would use this.

Can the TLS handshake be made to respect timeouts on sockets, or would
this require changes deep inside OpenSSL?

Graham

From g.brandl at gmx.net  Tue Aug 21 18:53:23 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 21 Aug 2007 18:53:23 +0200
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <79990c6b0708210801q6cff70cfpce672235dc9390c8@mail.gmail.com>
References: <f9t2nn$ksg$1@sea.gmane.org>	<9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>	<79990c6b0708210625h7c763a21h619207a8e95d1070@mail.gmail.com>	<faeq4s$eqm$2@sea.gmane.org>
	<46CAF0B6.30506@benjiyork.com>
	<79990c6b0708210801q6cff70cfpce672235dc9390c8@mail.gmail.com>
Message-ID: <faf59s$pl0$1@sea.gmane.org>

Paul Moore schrieb:
> On 21/08/07, Benji York <benji at benjiyork.com> wrote:
>> Georg Brandl wrote:
>> > I put the CHM at <http://www.python.org/~gbrandl/pydoc.chm>, if you want
>> > to have a look.
>>
>> Generally looks good.  I did get this error when opening the CHM:
>>
>> """
>> A Runtime Error has occurred.
>> Do you with to Debug?
>>
>> Line: 236
>> Error: Expected identifier, string or number
>> """
>>
>> Nothing happened when I clicked "Yes".  The error is repeated each time
>> a new page is opened.
> 
> I got that as well, in index.html line 264 character 7. I suspect it's
> one of the included js files (which I can't get access to).
> 
>> An additional opinion:  The red, appear-on-hover permalink markers don't
>> make much sense in a CHM and are somewhat distracting.  Can those be
>> hidden, perhaps via CSS?
> 
> That sounds reasonable.

Okay, I uploaded a new version without JavaScript and with hidden permalink
markers.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From janssen at parc.com  Tue Aug 21 18:58:40 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 21 Aug 2007 09:58:40 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug20.123330pdt."57996"@synergy1.parc.xerox.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
	<46C9D079.1040009@v.loewis.de>
	<07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug21.095847pdt."57996"@synergy1.parc.xerox.com>

> Should there be an "SSL socket", which is
> just like a regular socket?  Does that really provide any additional
> functionality to anyone?  Most apps and classes that use TCP sockets
> wrap the socket with socket._fileobject() to get "read" and "write",
> anyway -- can't they just wrap it with socket.ssl instead?

The issue here is the pervasive influence of the socket model.

Take the SocketServer module's BaseRequestHandler.  It has a slot
called "request" which is supposed to contain a socket, not a Raw I/O
stream.  Lots of other modules use that framework, including "pydoc",
"SimpleHTTPServer", "BaseHTTPServer", "SimpleXMLRPCServer", and
"CGIHTTPServer".

I think a few appropriate modifications to BaseRequestHandler and
StreamRequestHandler would do the trick here.

More serious is "asyncore", which is all about sockets.  The
"dispatcher" class in that module is riddled with socket assumptions.
And there's lots of code based on it.  Looking at that, I come back
to the idea that socket.ssl() should return an object which is a
proper subtype of socket.socket.

Probably with a few extra methods:

  read(), write() -- to support existing use of the SSLObject
  getpeercert() -- analogue to "getpeeraddr", but returns cert details
  shutdown() -- to close the SSL channel -- would return the underlying socket object

Bill

From martin at v.loewis.de  Tue Aug 21 19:22:55 2007
From: martin at v.loewis.de (=?ISO-8859-2?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 21 Aug 2007 19:22:55 +0200
Subject: [Python-Dev] Segfault
In-Reply-To: <ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
Message-ID: <46CB1F6F.4050909@v.loewis.de>

> I think there might already be a bug report about file not being
> thread-safe.  (It could have also been socket, but I think Martin
> fixed a problem in socket a while ago.)

This was a different problem, though: the socket object contained
an address structure, which it didn't really need and which wasn't
thread-safe. The (simple) fix was to remove the address from the
object, and move it onto the stack.

Regards,
Martin

From janssen at parc.com  Tue Aug 21 19:22:14 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 21 Aug 2007 10:22:14 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <20070821161359.E6F17D4439@mra02.ch.as12513.net> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
	<46C9D079.1040009@v.loewis.de>
	<07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.173941pdt."57996"@synergy1.parc.xerox.com>
	<46CA8CC5.70804@v.loewis.de>
	<07Aug21.081137pdt."57996"@synergy1.parc.xerox.com>
	<07Aug21.083032pdt."57996"@synergy1.parc.xerox.com>
	<20070821161359.E6F17D4439@mra02.ch.as12513.net>
Message-ID: <07Aug21.102222pdt."57996"@synergy1.parc.xerox.com>

> Can the TLS handshake be made to respect timeouts on sockets, or would
> this require changes deep inside OpenSSL?

I'm not sure.  Good test case to try.  I believe it will work.

By the way, interested parties should read

http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html

and think about the options there.  I see the code currently selects

   SSL_OP_ALL

to enable all the various bug workarounds, but selects none of the
modifiers.  I'm not currently planning to change that behavior.

Bill

From brett at python.org  Tue Aug 21 20:22:01 2007
From: brett at python.org (Brett Cannon)
Date: Tue, 21 Aug 2007 11:22:01 -0700
Subject: [Python-Dev] How to interpret get_code from PEP 302?
In-Reply-To: <79990c6b0708210540s6851164didc5413d14e4f8090@mail.gmail.com>
References: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>
	<79990c6b0708210540s6851164didc5413d14e4f8090@mail.gmail.com>
Message-ID: <bbaeab100708211122h184670e2q5dad33646b9026e4@mail.gmail.com>

[Thanks to Guido, Paul, and Nick for replying; going to just reply to
Paul since everyone said the same thing and it's his fault I didn't
understand the wording.  =)]

On 8/21/07, Paul Moore <p.f.moore at gmail.com> wrote:
> On 21/08/07, Brett Cannon <brett at python.org> wrote:
> > PEP 302 ("New Import Hooks") has an optional extensions section so
> > that tools like py2exe and py2app have an easier time.  Part of the
> > optional extensions is the method get_code that is to return the code
> > object for the specified method (if the loader can handle it).
> >
> > But there is a lack in the definition of how get_code is supposed to
> > be implemented.  The definition says that the "method should return
> > the code object associated with the module", which is fine.  But then
> > it goes on to state that "If the loader doesn't have the code object
> > but it _does_ have the source code, it should return the compiled
> > source code".  This throws me as this makes it sound like bytecode
> > does not need to be used if the loader does not already have a code
> > object and there is no source to be had; any bytecode can be ignored.
> >
> > Now I doubt this is how it is supposed to be read.  Does anyone
> > disagree with that?  If not, I will change the wording to mention that
> > bytecode must be used if no source is available (and that the magic
> > number must be verified).
>
> Hmm, yes, that's muddled. Maybe it made sense to me when I wrote it
> :-) (I think it was my wording rather than Just's)
>
> get_code must *always* return the same code object that
> loader.load_module is using - whether that be bytecode or compiled
> source (and it must respect things like file timestamps where
> appropriate just like load_module does).

Damn, I was afraid you were going to say something like that. =)  I
was hoping for an answer that would allow me to use the source if
available and then use bytecode as a backup.  That might not have the
best performance, but it is the simplest to implement as that skips
the bytecode timestamp check.  This way means I need to refactor some
things to do less import stuff and just do code object creation and
abstract everything else that deals with modules directly to another
function; nothing big but it would have been nice to avoid.  =)

This also means pkgutil is possibly non-compliant as it's get_code
implementation does the above suggestion (tries for source to avoid
timestamp check, and uses bytecode as backup).

> What the sentence you quote
> is trying to say is that if there's a need to compile source, the
> get_code method must do this on behalf of the caller - it can't return
> None and expect the caller to try get_source and compile it manually.
> Someone who only wants a code object should never need to call
> get_source.
>
> I'm not sure that's any clearer! If you need further clarification,
> let me know (either on or off list). I'd appreciate it if you can
> clear the PEP's wording up.

Basically reword it as what you said; the code object returned by
get_code needs to be equivalent to the one load_module would use if
requested to load the same module (e.g., if load_module would use the
bytecode, then it must use the bytecode as it might subtly differ
somehow, like with bytecode optimizations or something).  And mention
that writing out new bytecode is not required (but should it be
optional or not allowed at all?).

-Brett

From pje at telecommunity.com  Tue Aug 21 21:03:20 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 21 Aug 2007 15:03:20 -0400
Subject: [Python-Dev] How to interpret get_code from PEP 302?
In-Reply-To: <bbaeab100708211122h184670e2q5dad33646b9026e4@mail.gmail.co
 m>
References: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>
	<79990c6b0708210540s6851164didc5413d14e4f8090@mail.gmail.com>
	<bbaeab100708211122h184670e2q5dad33646b9026e4@mail.gmail.com>
Message-ID: <20070821190056.0CAE13A403D@sparrow.telecommunity.com>

At 11:22 AM 8/21/2007 -0700, Brett Cannon wrote:
>This also means pkgutil is possibly non-compliant as it's get_code
>implementation does the above suggestion (tries for source to avoid
>timestamp check, and uses bytecode as backup).

That's not actually true, it just *looks* that way.  IIRC, mod_type 
will only be PY_SOURCE if there's no compiled version available.

There's a different bug, though, which is that if you call get_code() 
on an ImpLoader when the module hasn't been loaded, you could get 
out-of-date code, because it's currently relying on imp to do the 
timestamp check and recompile.


From janssen at parc.com  Tue Aug 21 21:49:55 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 21 Aug 2007 12:49:55 PDT
Subject: [Python-Dev] backport new server-side SSL to older Pythons?
Message-ID: <07Aug21.124958pdt."57996"@synergy1.parc.xerox.com>

I'd like to be able to backport this server-side SSL support to older
Pythons, like the 2.3.4 in CentOS 4 and the 2.3.5 in OS X 10.4.

So I'd like to move all the SSL stuff out of the "socket" module, and
add a new top-level module called "ssl" (or "networking.ssl", or
whatever the Py3K naming scheme says it should be).  The socket module
will then re-export a function from that module as socket.ssl(), which
will continue to do exactly what it does now.  More advanced users will
call functions in the "ssl" module.

Then I can bundle up the new versions of _ssl.c and ssl.py with a
setup.py file, and provide that as an add-on for older Python
installations.

Does this make sense?

Bill


From guido at python.org  Tue Aug 21 21:59:57 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 21 Aug 2007 12:59:57 -0700
Subject: [Python-Dev] backport new server-side SSL to older Pythons?
In-Reply-To: <-2942577283399265163@unknownmsgid>
References: <-2942577283399265163@unknownmsgid>
Message-ID: <ca471dc20708211259q580868d5v89fa2ccda1c9fa7d@mail.gmail.com>

On 8/21/07, Bill Janssen <janssen at parc.com> wrote:
> I'd like to be able to backport this server-side SSL support to older
> Pythons, like the 2.3.4 in CentOS 4 and the 2.3.5 in OS X 10.4.

That would have to be a private fork or a 3rd party extension module;
python.org is committed to keeping existing releases stable
(feature-wise).

> So I'd like to move all the SSL stuff out of the "socket" module, and
> add a new top-level module called "ssl" (or "networking.ssl", or
> whatever the Py3K naming scheme says it should be).  The socket module
> will then re-export a function from that module as socket.ssl(), which
> will continue to do exactly what it does now.  More advanced users will
> call functions in the "ssl" module.
>
> Then I can bundle up the new versions of _ssl.c and ssl.py with a
> setup.py file, and provide that as an add-on for older Python
> installations.
>
> Does this make sense?

I think that it probably can be done, but beware that older Pythons
(and you're going quite a while back!) may use different APIs for
object creation/deletion, so you may end up having to do some work
still. Also, those older versions may have (client-side) ssl support
in their socket module -- isn't that going to conflict? Finally, some
old Python versions may not like new openssl versions (I don't know if
this is the case, but I wouldn't rule it out without testing).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From p.f.moore at gmail.com  Tue Aug 21 22:12:28 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 21 Aug 2007 21:12:28 +0100
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <faf59s$pl0$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org>
	<9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>
	<79990c6b0708210625h7c763a21h619207a8e95d1070@mail.gmail.com>
	<faeq4s$eqm$2@sea.gmane.org> <46CAF0B6.30506@benjiyork.com>
	<79990c6b0708210801q6cff70cfpce672235dc9390c8@mail.gmail.com>
	<faf59s$pl0$1@sea.gmane.org>
Message-ID: <79990c6b0708211312ieeec9f9q6e79d701bd0bafdf@mail.gmail.com>

On 21/08/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Okay, I uploaded a new version without JavaScript and with hidden permalink
> markers.

Very nice! Works fine now. Thanks for the quick fix.
Paul

From janssen at parc.com  Wed Aug 22 01:47:22 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 21 Aug 2007 16:47:22 PDT
Subject: [Python-Dev] backport new server-side SSL to older Pythons?
In-Reply-To: <ca471dc20708211259q580868d5v89fa2ccda1c9fa7d@mail.gmail.com> 
References: <-2942577283399265163@unknownmsgid>
	<ca471dc20708211259q580868d5v89fa2ccda1c9fa7d@mail.gmail.com>
Message-ID: <07Aug21.164728pdt."57996"@synergy1.parc.xerox.com>

> > I'd like to be able to backport this server-side SSL support to older
> > Pythons, like the 2.3.4 in CentOS 4 and the 2.3.5 in OS X 10.4.
> 
> That would have to be a private fork or a 3rd party extension module;
> python.org is committed to keeping existing releases stable
> (feature-wise).

Yes, that was my thinking -- just something for us holdouts to use.

> I think that it probably can be done, but beware that older Pythons
> (and you're going quite a while back!) may use different APIs for
> object creation/deletion, so you may end up having to do some work
> still.

Really?  2.3 to 2.6?  I've looked through the ChangeLog for _ssl.c,
and it does seem extensive.

 Also, those older versions may have (client-side) ssl support
> in their socket module -- isn't that going to conflict?

I don't think so.  Installing this package would upgrade that support
without modifying the existing functionality.

> Finally, some
> old Python versions may not like new openssl versions (I don't know if
> this is the case, but I wouldn't rule it out without testing).

Could be.  But it's the same OpenSSL that we're using to build the new
Python on that platform, so I think it should work.  It's more an
instance of the latest Python not liking older OpenSSL versions.

Bill

From greg.ewing at canterbury.ac.nz  Wed Aug 22 02:15:37 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 22 Aug 2007 12:15:37 +1200
Subject: [Python-Dev] How to interpret get_code from PEP 302?
In-Reply-To: <79990c6b0708210540s6851164didc5413d14e4f8090@mail.gmail.com>
References: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>
	<79990c6b0708210540s6851164didc5413d14e4f8090@mail.gmail.com>
Message-ID: <46CB8029.8030301@canterbury.ac.nz>

Paul Moore wrote:
> What the sentence you quote
> is trying to say is that if there's a need to compile source, the
> get_code method must do this on behalf of the caller - it can't return
> None and expect the caller to try get_source and compile it manually.

Why not simply say that it must return a code object?
All the rest follows from that.

--
Greg

From hrvoje.niksic at avl.com  Wed Aug 22 10:35:18 2007
From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=)
Date: Wed, 22 Aug 2007 10:35:18 +0200
Subject: [Python-Dev] Segfault
In-Reply-To: <ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
Message-ID: <1187771718.11322.47.camel@localhost>

On Tue, 2007-08-21 at 09:14 -0700, Neal Norwitz wrote:
> The patch is insufficient to prevent all types of crashes that occur
> when accessing a file from 2 threads (closing in one and doing
> whatever in another).

You are right.  I wouldn't go so far to say the file object
thread-unsafe, but it certainly has lurking bugs with calling close
while other threads are running.  BTW your mail doesn't seem to contain
the actual patch.

> Almost every place that accesses f_fp is a problem.

I think we need a reliable mechanism to prevent file_close messing with
f_fp while other operations are being performed.  Since each FILE has an
internal lock associated with it, flockfile could be used to lock out
the sections that access f_fp (falling back to a mutex in PyFileObject
on platforms without flockfile) without affecting other threads that
operate on other FILE's.

It's questionnable whether such an overhaul would pay off, given the
Py3k rewrite of the IO subsystem.



From hrvoje.niksic at avl.com  Wed Aug 22 10:42:47 2007
From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=)
Date: Wed, 22 Aug 2007 10:42:47 +0200
Subject: [Python-Dev] Segfault
In-Reply-To: <1187771718.11322.47.camel@localhost>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
	<1187771718.11322.47.camel@localhost>
Message-ID: <1187772167.11322.50.camel@localhost>

On Wed, 2007-08-22 at 10:35 +0200, Hrvoje Nik?i? wrote:
> I think we need a reliable mechanism to prevent file_close messing with
> f_fp while other operations are being performed.  Since each FILE has an
> internal lock associated with it, flockfile could be used to lock out
> the sections that access f_fp (falling back to a mutex in PyFileObject
> on platforms without flockfile) without affecting other threads that
> operate on other FILE's.

Scrap this suggestion, it won't work as written.  flockfile must not be
called after the FILE is deallocated, which is exactly what fclose does.
The mutex would need to be in PyFileObject itself.



From ncoghlan at gmail.com  Wed Aug 22 12:28:50 2007
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 22 Aug 2007 20:28:50 +1000
Subject: [Python-Dev] How to interpret get_code from PEP 302?
In-Reply-To: <46CB8029.8030301@canterbury.ac.nz>
References: <bbaeab100708202057nb6bdcd7g808d1a52d083f7f0@mail.gmail.com>	<79990c6b0708210540s6851164didc5413d14e4f8090@mail.gmail.com>
	<46CB8029.8030301@canterbury.ac.nz>
Message-ID: <46CC0FE2.3060301@gmail.com>

Greg Ewing wrote:
> Paul Moore wrote:
>> What the sentence you quote
>> is trying to say is that if there's a need to compile source, the
>> get_code method must do this on behalf of the caller - it can't return
>> None and expect the caller to try get_source and compile it manually.
> 
> Why not simply say that it must return a code object?
> All the rest follows from that.

The method is allowed to return None if a code object genuinely isn't 
available (which happens when the module is a builtin or extension module).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From benji at benjiyork.com  Wed Aug 22 15:23:35 2007
From: benji at benjiyork.com (Benji York)
Date: Wed, 22 Aug 2007 09:23:35 -0400
Subject: [Python-Dev] [Python-3000] Documentation switch imminent
In-Reply-To: <faf59s$pl0$1@sea.gmane.org>
References: <f9t2nn$ksg$1@sea.gmane.org>	<9e804ac0708210123k24e6c05cpb321b32e10be476e@mail.gmail.com>	<79990c6b0708210625h7c763a21h619207a8e95d1070@mail.gmail.com>	<faeq4s$eqm$2@sea.gmane.org>	<46CAF0B6.30506@benjiyork.com>	<79990c6b0708210801q6cff70cfpce672235dc9390c8@mail.gmail.com>
	<faf59s$pl0$1@sea.gmane.org>
Message-ID: <46CC38D7.50807@benjiyork.com>

Georg Brandl wrote:
> Okay, I uploaded a new version without JavaScript and with hidden permalink
> markers.

Very nice.  I've compared this version and the 2.4 CHM a little and 
while mostly similar, there are several small improvements I appreciate.

One other minor issue, when viewing the "random" module docs, it looks 
like two or more lines overlap in the footer (scroll all the way down).
-- 
Benji York
http://benjiyork.com

From janssen at parc.com  Wed Aug 22 16:23:18 2007
From: janssen at parc.com (Bill Janssen)
Date: Wed, 22 Aug 2007 07:23:18 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug21.095847pdt."57996"@synergy1.parc.xerox.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
	<46C9D079.1040009@v.loewis.de>
	<07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>
	<07Aug21.095847pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug22.072324pdt."57996"@synergy1.parc.xerox.com>

>   getpeercert() -- analogue to "getpeeraddr", but returns cert details

This would return three kinds of values:

  No certificate received --> None

  Certificate received but not validated --> {}

  Certificate received and validated --> { full details }

Bill

From janssen at parc.com  Wed Aug 22 17:17:30 2007
From: janssen at parc.com (Bill Janssen)
Date: Wed, 22 Aug 2007 08:17:30 PDT
Subject: [Python-Dev] More on server-side SSL support
In-Reply-To: <07Aug20.123330pdt."57996"@synergy1.parc.xerox.com> 
References: <ca471dc20708131025k23a57570jc85c022065ebaff1@mail.gmail.com>
	<07Aug15.182939pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.133416pdt."57996"@synergy1.parc.xerox.com>
	<07Aug19.151552pdt."57996"@synergy1.parc.xerox.com>
	<07Aug20.094618pdt."57996"@synergy1.parc.xerox.com>
	<46C9C9C3.9090201@v.loewis.de>
	<07Aug20.101928pdt."57996"@synergy1.parc.xerox.com>
	<46C9D079.1040009@v.loewis.de>
	<07Aug20.123330pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug22.081734pdt."57996"@synergy1.parc.xerox.com>

For those of you following along at home, there's now a patch at
http://bill.janssen.org/python/ssl-update-diff which applies against
the current trunk.  Working code, though I still need to tweak some
import statements for backwards compatibility.  I've started updating
the test suite, see Lib/test_ssl.py.  I'd appreciate review and
comments.

Note that the suggested use of the newer code is to

     import ssl, socket
     conn = ssl.sslsocket(socket.socket())

Using the older

     import socket
     conn = socket.ssl(socket.socket())

still works for backwards compatibility (same restrictions:
client-side only, no verification of server cert, SSLv23).  However,
the object returned from this call is not the older C-implemented SSL
object, but rather an instance of ssl.sslsocket (which supports the
same read() and write() calls).  Should I return the C SSL object,
instead?

Bill

> > I view TLS as a wrapper around / layer on top of TCP, and so I think the
> > API should look like, as well.
> 
> I think Martin raises a valid point here, which should at least be
> discussed more thoroughly.  Should there be an "SSL socket", which is
> just like a regular socket?  Does that really provide any additional
> functionality to anyone?  Most apps and classes that use TCP sockets
> wrap the socket with socket._fileobject() to get "read" and "write",
> anyway -- can't they just wrap it with socket.ssl instead?
> 
> Perhaps in the sprint, I should just concentrate on widening the
> "socket.ssl" interface a bit, and improving the functionality of the
> SSLObject a bit.
> 
> Suggested improvements:
> 
>   *  Allow server-side operation.
> 
>   *  Allow specification of particular SSL protocol version.
> 
>   *  Allow certificate validation.  This is a bit tricky; typically
>      certs are validated against some database of root certificates, so you
>      need a whole infrastructure to maintain that database.  Currently, we
>      don't have one, so no certs can be validated.  We could add a switch
>      to allow auto-validation of self-signed certs pretty easily.  I could
>      add a parameter to the SSLObject constructor which would be a filepath
>      for a file full of root certs (see SSL_CTX_load_verify_locations(3ssl)).
> 
>   *  Add a method to retrieve the other side's certificate info.  What's
>      a good format for the "notBefore" and "notAfter" dates?  The simplest
>      thing to use is the string formatting OpenSSL provides, which is
>      always like "Sep 29 16:38:04 2006 GMT", which can easily be parsed
>      by the time.strptime() function if the user wants something else.
>      On the other hand, figuring out how to use strptime() is always a
>      pain, so providing a convenience function wouldn't be a terrible idea.
> 
>   *  Add a shutdown() method to stop using SSL on the underlying socket
>      without closing the socket.
> 
>   *  Make SSLObject conform to the Raw I/O API in PEP 3116.  This one is
>      interesting; what should close() do?  Close the underlying socket?  Or
>      just do an SSL shutdown?
> 
> Bill
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/janssen%40parc.com


From skip at pobox.com  Wed Aug 22 18:52:08 2007
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 22 Aug 2007 11:52:08 -0500
Subject: [Python-Dev] AtheOS/Syllable
Message-ID: <18124.27064.798088.920762@montanaro.dyndns.org>

The Syllable folks apparently still want to support Python, so I'm going to
keep that support in there, switching "atheos" for "syllable" where it
occurs.  I have a patch against 2.5.1 from the Syllable folks.  I'll see if
I can make it apply to the trunk and py3k branches in a semi-clean fashion.

Skip


From snaury at gmail.com  Wed Aug 22 23:18:45 2007
From: snaury at gmail.com (Alexey Borzenkov)
Date: Thu, 23 Aug 2007 01:18:45 +0400
Subject: [Python-Dev] Strange behavior of subprocess.Popen._get_handles
	under Windows
Message-ID: <e2480c70708221418l36335cd5g652fa48c0ed1b6ee@mail.gmail.com>

For a long time I was surprised why if I have a script testme.py:

    import subprocess
    subprocess.call("echo Something", shell=True)

and I try to execute it like this:

    python testme.py >testme.txt

I get the output:

    The handle is invalid.

Strange failures randomly happened with different programs, so I
thought maybe this was intended (mis)behavior, and I found that I can
workaround this by explicitly specifying std* handles:

    subprocess.call(...., stdin=sys.stdin, stdout=sys.stdout, stderr=sys.stderr)

I lived with this until I understood that if something substitutes
sys.std* with a proxy (examples: running under IDLE or wxPython, or
when there is a custom logger on stdout/stderr, etc), this will no
longer work:

      File "C:\Programs\Python25\Lib\subprocess.py", line 698, in _get_handles
        p2cread = msvcrt.get_osfhandle(stdin.fileno())
    AttributeError: fileno

Now I finally found that my problem are these two lines in subprocess.py:

                if stdin is None and stdout is None and stderr is None:
                    return (None, None, None, None, None, None)

These lines add an interesting quirk: if I explicitly specify any
single channel (i.e. stdout=1) the problem suddenly disappears, and if
I just comment these lines altogether my problem vanishes completely!
(i.e. output redirection works absolutely well)

Further investigations showed that it seems to be some strange OS
quirk/bug, i.e. this behavior is caused when STARTF_USESTDHANDLES is
not used, and crt's spawnl* behaves incorrectly too:

    os.spawnl(r"C:\WINDOWS\system32\cmd.exe",
r"C:\WINDOWS\system32\cmd.exe /c echo Something")

So the question is should these two lines be removed to workaround this OS bug?

To my thinking, this will not change behavior when any std* arguments
are passed to Popen, and the only case when it kicks in is when no
std* arguments are specified and the resulting side effects are
previously failing cases start working correctly. Or am I wrong here,
are there any side effects of using STARTF_USESTDHANDLES that I'm
missing here?

Best regards,
Alexey.

From mhammond at skippinet.com.au  Thu Aug 23 00:44:06 2007
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Thu, 23 Aug 2007 08:44:06 +1000
Subject: [Python-Dev] Strange behavior of
	subprocess.Popen._get_handles	under Windows
In-Reply-To: <e2480c70708221418l36335cd5g652fa48c0ed1b6ee@mail.gmail.com>
References: <e2480c70708221418l36335cd5g652fa48c0ed1b6ee@mail.gmail.com>
Message-ID: <01e501c7e50d$f4e607c0$deb21740$@com.au>

> Now I finally found that my problem are these two lines in
> subprocess.py:
> 
>                 if stdin is None and stdout is None and stderr is None:
>                     return (None, None, None, None, None, None)
> 
> These lines add an interesting quirk: if I explicitly specify any
> single channel (i.e. stdout=1) the problem suddenly disappears, and if
> I just comment these lines altogether my problem vanishes completely!
> (i.e. output redirection works absolutely well)
> 
> Further investigations showed that it seems to be some strange OS
> quirk/bug,

I'm not quite with you here - what strange OS bug do you think you have
found?  I expect that such a bug would be well documented somewhere, even if
not directly by MS.

> To my thinking, this will not change behavior when any std* arguments
> are passed to Popen, and the only case when it kicks in is when no
> std* arguments are specified and the resulting side effects are
> previously failing cases start working correctly. Or am I wrong here,
> are there any side effects of using STARTF_USESTDHANDLES that I'm
> missing here?

MSDN documents that without that flag, hStdInput will be "the keyboard
buffer" while the output handles will default to "the console window's
buffer" - which sounds significantly different to your expectation that the
existing handles will be used as the default (unless I misunderstand).
Sadly, your mail isn't clear enough for me to be sure about what semantics
you are seeing and exactly what you expect, and I don't have time to
experiment.  Maybe a clear indication of the OS bug you are referring to,
and some complete examples which demonstrate the problems you are having
would help.

Cheers,

Mark


From alexandre at peadrop.com  Thu Aug 23 01:44:02 2007
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Wed, 22 Aug 2007 19:44:02 -0400
Subject: [Python-Dev] Avoiding cascading test failures
Message-ID: <acd65fa20708221644y2ab376bemd4d550ff43fdd91@mail.gmail.com>

When I was fixing tests failing in the py3k branch, I found the number
duplicate failures annoying. Often, a single bug, in an important
method or function, caused a large number of testcase to fail. So, I
thought of a simple mechanism for avoiding such cascading failures.

My solution is to add a notion of dependency to testcases. A typical
usage would look like this:

    @depends('test_getvalue')
    def test_writelines(self):
        ...
        memio.writelines([buf] * 100)
        self.assertEqual(memio.getvalue(), buf * 100)
        ...

Here, running the test is pointless if test_getvalue fails. So by
making test_writelines depends on the success of test_getvalue, we can
ensure that the report won't be polluted with unnecessary failures.

Also, I believe this feature will lead to more orthogonal tests, since
it encourages the user to write smaller test with less dependencies.

I wrote an example implementation (included below) as a proof of
concept. If the idea get enough support, I will implement it and add
it to the unittest module.

-- Alexandre


class CycleError(Exception):
    pass


class TestCase:

    def __init__(self):
        self.graph = {}
        tests = [x for x in dir(self) if x.startswith('test')]
        for testname in tests:
            test = getattr(self, testname)
            if hasattr(test, 'deps'):
                self.graph[testname] = test.deps
            else:
                self.graph[testname] = set()

    def run(self):
        graph = self.graph
        toskip = set()
        msgs = []
        while graph:
            # find tests without any pending dependencies
            source = [test for test, deps in graph.items() if not deps]
            if not source:
                raise CycleError
            for testname in source:
                if testname in toskip:
                    msgs.append("%s... skipped" % testname)
                    resolvedeps(graph, testname)
                    del graph[testname]
                    continue
                test = getattr(self, testname)
                try:
                    test()
                except AssertionError:
                    toskip.update(getrevdeps(graph, testname))
                    msgs.append("%s... failed" % testname)
                except:
                    toskip.update(getrevdeps(graph, testname))
                    msgs.append("%s... error" % testname)
                else:
                    msgs.append("%s... ok" % testname)
                finally:
                    resolvedeps(graph, testname)
                    del graph[testname]
        for msg in sorted(msgs):
            print(msg)


def getrevdeps(graph, testname):
    """Return the reverse depencencies of a test"""
    rdeps = set()
    for x in graph:
        if testname in graph[x]:
            rdeps.add(x)
    if rdeps:
        # propagate depencencies recursively
        for x in rdeps.copy():
            rdeps.update(getrevdeps(graph, x))
    return rdeps

def resolvedeps(graph, testname):
    for test in graph:
        if testname in graph[test]:
            graph[test].remove(testname)

def depends(*args):
    def decorator(test):
        if hasattr(test, 'deps'):
            test.deps.update(args)
        else:
            test.deps = set(args)
        return test
    return decorator


class MyTest(TestCase):

    @depends('test_foo')
    def test_nah(self):
        pass

    @depends('test_bar', 'test_baz')
    def test_foo(self):
        pass

    @depends('test_tin')
    def test_bar(self):
        self.fail()

    def test_baz(self):
        self.error()

    def test_tin(self):
        pass

    def error(self):
        raise ValueError

    def fail(self):
        raise AssertionError

if __name__ == '__main__':
    t = MyTest()
    t.run()

From nnorwitz at gmail.com  Thu Aug 23 06:32:11 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 22 Aug 2007 21:32:11 -0700
Subject: [Python-Dev] Segfault
In-Reply-To: <1187771718.11322.47.camel@localhost>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
	<1187771718.11322.47.camel@localhost>
Message-ID: <ee2a432c0708222132v6fbcf29bx7ea617ba6ea2887d@mail.gmail.com>

On 8/22/07, Hrvoje Nik?i? <hrvoje.niksic at avl.com> wrote:
> On Tue, 2007-08-21 at 09:14 -0700, Neal Norwitz wrote:
> > The patch is insufficient to prevent all types of crashes that occur
> > when accessing a file from 2 threads (closing in one and doing
> > whatever in another).
>
> You are right.  I wouldn't go so far to say the file object
> thread-unsafe, but it certainly has lurking bugs with calling close
> while other threads are running.  BTW your mail doesn't seem to contain
> the actual patch.

Sorry about that.  I thought something was up when gmail didn't
automatically upload it while I was writing the message.  It looks
like it took this time.  (The attached patch isn't complete, but
catches most cases.)

n
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file.patch
Type: text/x-patch
Size: 7810 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20070822/88afac31/attachment-0001.bin 

From snaury at gmail.com  Thu Aug 23 07:41:40 2007
From: snaury at gmail.com (Alexey Borzenkov)
Date: Thu, 23 Aug 2007 09:41:40 +0400
Subject: [Python-Dev] Strange behavior of subprocess.Popen._get_handles
	under Windows
In-Reply-To: <01e501c7e50d$f4e607c0$deb21740$@com.au>
References: <e2480c70708221418l36335cd5g652fa48c0ed1b6ee@mail.gmail.com>
	<01e501c7e50d$f4e607c0$deb21740$@com.au>
Message-ID: <e2480c70708222241m13cf3228jff79b88814a276d0@mail.gmail.com>

On 8/23/07, Mark Hammond <mhammond at skippinet.com.au> wrote:
> > Further investigations showed that it seems to be some strange OS
> > quirk/bug,
> I'm not quite with you here - what strange OS bug do you think you have
> found?  I expect that such a bug would be well documented somewhere, even if
> not directly by MS.
[...]
> MSDN documents that without that flag, hStdInput will be "the keyboard
> buffer" while the output handles will default to "the console window's
> buffer" - which sounds significantly different to your expectation that the
> existing handles will be used as the default (unless I misunderstand).
> Sadly, your mail isn't clear enough for me to be sure about what semantics
> you are seeing and exactly what you expect, and I don't have time to
> experiment.  Maybe a clear indication of the OS bug you are referring to,
> and some complete examples which demonstrate the problems you are having
> would help.

Hmm, sorry if I wasn't clear. Here's a better example:

1.c:
    #include <stdio.h>
    #include <windows.h>

    int main(int argc, char** argv)
    {
        HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
        fprintf(stderr, "hStdOut: %08X\n", hStdOut);
        DWORD dwWritten;
        WriteFile(hStdOut, "Something", 9, &dwWritten, NULL);
        return 0;
    }

1.py:
    import subprocess
    subprocess.call(["1.exe"])

in cmd.exe shell:
    C:\>1.exe
    hStdOut: 00000007
    Something

    C:\>1.exe>1.txt
    hStdOut: 000006E0

    (1.txt file now contains string "Something")

    C:\>1.py
    hStdOut: 00000007
    Something

    C:\>1.py>1.txt
    hStdOut: 00000004

    (1.txt file is now completely empty, i.e. hStdOut is invalid)

If what you say was true and stdout would default to console window's
buffer, then I would at least see "Something" in the console. On the
contrary output just stops working at all.

And what I expect (and always expected) is output redirection to be
inherited by child processes. The best example is when I'm writing a
batch file:

    dosomething.exe /something
    dosomething2.exe /somethingmore

and then execute it as batchfile.cmd>somefile.txt I always get this
behavior, i.e. all standard output (except standard error) is
redirected to somefile.txt. I'm sure on *nixes it's all the same as
well. So what documentation for STARTUPINFO specifies is already very
suspicious and uncommon, while in reality it doesn't even work as
specified!

Best regards,
Alexey.

From snaury at gmail.com  Thu Aug 23 08:21:01 2007
From: snaury at gmail.com (Alexey Borzenkov)
Date: Thu, 23 Aug 2007 10:21:01 +0400
Subject: [Python-Dev] Strange behavior of subprocess.Popen._get_handles
	under Windows
In-Reply-To: <e2480c70708222241m13cf3228jff79b88814a276d0@mail.gmail.com>
References: <e2480c70708221418l36335cd5g652fa48c0ed1b6ee@mail.gmail.com>
	<01e501c7e50d$f4e607c0$deb21740$@com.au>
	<e2480c70708222241m13cf3228jff79b88814a276d0@mail.gmail.com>
Message-ID: <e2480c70708222321t115f7db8h27f8999da662e67b@mail.gmail.com>

>     C:\>1.py
>     hStdOut: 00000007
>     Something
>
>     C:\>1.py>1.txt
>     hStdOut: 00000004
>
>     (1.txt file is now completely empty, i.e. hStdOut is invalid)

Hmm, now I see that maybe that's where I was wrong. When I used
`python 1.py>1.txt' it suddenly started working as expected. :-/ Must
be some strange difference with how HKCR entries are parsed and then
executed...

Best regards,
Alexey.

From mhammond at skippinet.com.au  Thu Aug 23 08:25:18 2007
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Thu, 23 Aug 2007 16:25:18 +1000
Subject: [Python-Dev] Strange behavior of subprocess.Popen._get_handles
	under Windows
In-Reply-To: <e2480c70708222321t115f7db8h27f8999da662e67b@mail.gmail.com>
References: <e2480c70708221418l36335cd5g652fa48c0ed1b6ee@mail.gmail.com>	
	<01e501c7e50d$f4e607c0$deb21740$@com.au>	
	<e2480c70708222241m13cf3228jff79b88814a276d0@mail.gmail.com>
	<e2480c70708222321t115f7db8h27f8999da662e67b@mail.gmail.com>
Message-ID: <022b01c7e54e$637aa3e0$2a6feba0$@com.au>

> >     C:\>1.py
> >     hStdOut: 00000007
> >     Something
> >
> >     C:\>1.py>1.txt
> >     hStdOut: 00000004
> >
> >     (1.txt file is now completely empty, i.e. hStdOut is invalid)
> 
> Hmm, now I see that maybe that's where I was wrong. When I used
> `python 1.py>1.txt' it suddenly started working as expected. :-/ Must
> be some strange difference with how HKCR entries are parsed and then
> executed...

I think it is more about how the shell (ie, cmd.exe by default) (mis-)handles the command-line.

Cheers,

Mark


From hrvoje.niksic at avl.com  Thu Aug 23 09:38:16 2007
From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=)
Date: Thu, 23 Aug 2007 09:38:16 +0200
Subject: [Python-Dev] Segfault
In-Reply-To: <ee2a432c0708222132v6fbcf29bx7ea617ba6ea2887d@mail.gmail.com>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
	<1187771718.11322.47.camel@localhost>
	<ee2a432c0708222132v6fbcf29bx7ea617ba6ea2887d@mail.gmail.com>
Message-ID: <1187854696.11322.67.camel@localhost>

On Wed, 2007-08-22 at 21:32 -0700, Neal Norwitz wrote:
>         Py_BEGIN_ALLOW_THREADS
>         errno = 0;
> -       ret = _portable_fseek(f->f_fp, offset, whence);
> +       if (f->f_fp != NULL)
> +               ret = _portable_fseek(f->f_fp, offset, whence); 

Doesn't this kind of code retain a race condition?  Since this is an
"allow threads" section that runs with the GIL released, file_close
might acquire the GIL and be running in parallel to this code.  If
file_close sets f_fp to NULL after the "if" condition evaluates, but
before the call to _portable_fseek completes, we still get a segfault.

If I understand the Python threading model, the only reliable way to
resolve this is to protect the PyFileObject operations with a mutex.



From martin at v.loewis.de  Thu Aug 23 15:11:14 2007
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Thu, 23 Aug 2007 15:11:14 +0200
Subject: [Python-Dev] SF tracker closed now
Message-ID: <20070823151114.eji9nl6d28gcw8s4@webmail.df.eu>

We are starting with the tracker conversion. Please stop using the bug  
trackers (either SF or roundup) until further notice.

Regards,
Martin



From Martin.vonLoewis at hpi.uni-potsdam.de  Thu Aug 23 15:20:42 2007
From: Martin.vonLoewis at hpi.uni-potsdam.de (=?iso-8859-1?Q?Martin_von_L=F6wis?=)
Date: Thu, 23 Aug 2007 15:20:42 +0200
Subject: [Python-Dev] SF tracker closed now
Message-ID: <5AC0F51A926CD442BC08D55DA7DAB1D40670BB68@3mxma1r.hpi.uni-potsdam.de>

[second copy, since the first one apparently did not go through]

We are starting with the tracker conversion. Please stop using 
the bug trackers (either SF or roundup) until further notice.

Regards,
Martin

From Martin.vonLoewis at hpi.uni-potsdam.de  Thu Aug 23 18:50:24 2007
From: Martin.vonLoewis at hpi.uni-potsdam.de (=?iso-8859-1?Q?Martin_von_L=F6wis?=)
Date: Thu, 23 Aug 2007 18:50:24 +0200
Subject: [Python-Dev] New bug tracker is active now
Message-ID: <5AC0F51A926CD442BC08D55DA7DAB1D40670BB6A@3mxma1r.hpi.uni-potsdam.de>

We have completed the conversion to Roundup as the bug tracker
for Python; the new tracker is now life at

  http://bugs.python.org/

It has all reports from the SourceForge tracker, along with
all messages and files that had been attached.

To edit bug reports, you still need to log in. We have migrated
all account names from SF as-is, but obviously couldn't migrate
the passwords as well. So you need to reset your password, through

  http://bugs.python.org/user?@template=forgotten

Fill your SF account name in the bottom field, wait for an email,
click on the link, wait for a second email, change your password
to something you can remember. All SF project members have developer
privileges in the roundup tracker.

If you find problems with that bug tracker, follow the Report Tracker
Link. This meta-tracker does *not* use the same accounts as the Python
tracker.

I'd like to take the opportunity to thank Erik Forsberg, who really did
the majority of the work involved in switching to roundup (along with
various people helping out).

Regards,
Martin

From nnorwitz at gmail.com  Thu Aug 23 20:57:45 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 23 Aug 2007 11:57:45 -0700
Subject: [Python-Dev] Segfault
In-Reply-To: <1187854696.11322.67.camel@localhost>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
	<1187771718.11322.47.camel@localhost>
	<ee2a432c0708222132v6fbcf29bx7ea617ba6ea2887d@mail.gmail.com>
	<1187854696.11322.67.camel@localhost>
Message-ID: <ee2a432c0708231157j65f5b7e7yd23c0430f9e1c5d3@mail.gmail.com>

On 8/23/07, Hrvoje Nik?i? <hrvoje.niksic at avl.com> wrote:
> On Wed, 2007-08-22 at 21:32 -0700, Neal Norwitz wrote:
> >         Py_BEGIN_ALLOW_THREADS
> >         errno = 0;
> > -       ret = _portable_fseek(f->f_fp, offset, whence);
> > +       if (f->f_fp != NULL)
> > +               ret = _portable_fseek(f->f_fp, offset, whence);
>
> Doesn't this kind of code retain a race condition?  Since this is an

Exactly what I'm thinking.  That's why I said:

    I'm not convinced the attached patch is good enough though.

It definitely reduces the race and might fix the problem.

> "allow threads" section that runs with the GIL released, file_close
> might acquire the GIL and be running in parallel to this code.  If
> file_close sets f_fp to NULL after the "if" condition evaluates, but
> before the call to _portable_fseek completes, we still get a segfault.

However, the setting of f_fp to NULL happens with the GIL set, ie
while only one thread is running.  So I *think* (but I'm not sure) the
condition you mention is not possible.  Either way the fix seems
brittle.  (Also, the patch I attached was not complete.)

Also all this code has gone away in 3.0.

n

From sjoerd at acm.org  Thu Aug 23 21:23:20 2007
From: sjoerd at acm.org (Sjoerd Mullender)
Date: Thu, 23 Aug 2007 21:23:20 +0200
Subject: [Python-Dev] New bug tracker is active now
In-Reply-To: <5AC0F51A926CD442BC08D55DA7DAB1D40670BB6A@3mxma1r.hpi.uni-potsdam.de>
References: <5AC0F51A926CD442BC08D55DA7DAB1D40670BB6A@3mxma1r.hpi.uni-potsdam.de>
Message-ID: <46CDDEA8.3070801@acm.org>

On 2007-08-23 18:50, Martin von L?wis wrote:
> We have completed the conversion to Roundup as the bug tracker
> for Python; the new tracker is now life at
> 
>   http://bugs.python.org/

So far, I have one issue with the new tracker: the timezone.

When editing my details I saw there is a field for my timezone.  The
comment says: "this is a numeric hour offset, the default is UTC", so
I'm assuming it counts in whole hours (fine for me, not so fine for
people in e.g. India, unless it accepts decimal notation).  I have two
issues with this: it's unclear which direction this works (is East
positive or negative?), and I will need to change this number twice
yearly.  In other words, it shouldn't be a number, it should be a name,
like the names used in Linux: Europe/Amsterdam.

Looks nice otherwise.

-- 
Sjoerd Mullender


From martin at v.loewis.de  Thu Aug 23 21:57:16 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 23 Aug 2007 21:57:16 +0200
Subject: [Python-Dev] New bug tracker is active now
In-Reply-To: <46CDDEA8.3070801@acm.org>
References: <5AC0F51A926CD442BC08D55DA7DAB1D40670BB6A@3mxma1r.hpi.uni-potsdam.de>
	<46CDDEA8.3070801@acm.org>
Message-ID: <46CDE69C.8030906@v.loewis.de>

> When editing my details I saw there is a field for my timezone.  The
> comment says: "this is a numeric hour offset, the default is UTC", so
> I'm assuming it counts in whole hours (fine for me, not so fine for
> people in e.g. India, unless it accepts decimal notation).  I have two
> issues with this: it's unclear which direction this works (is East
> positive or negative?), and I will need to change this number twice
> yearly.  In other words, it shouldn't be a number, it should be a name,
> like the names used in Linux: Europe/Amsterdam.

I think this will be fairly difficult to fix. Roundup doesn't have
a timezone database; it just applies the delta to the UTC time. We are
merely using roundup, instead of developing it.

See

http://sourceforge.net/tracker/index.php?func=detail&aid=818711&group_id=31577&atid=402791

Regards,
Martin

From brett at python.org  Thu Aug 23 22:23:40 2007
From: brett at python.org (Brett Cannon)
Date: Thu, 23 Aug 2007 13:23:40 -0700
Subject: [Python-Dev] New bug tracker is active now
In-Reply-To: <5AC0F51A926CD442BC08D55DA7DAB1D40670BB6A@3mxma1r.hpi.uni-potsdam.de>
References: <5AC0F51A926CD442BC08D55DA7DAB1D40670BB6A@3mxma1r.hpi.uni-potsdam.de>
Message-ID: <bbaeab100708231323n6f82a0e5y4843691e4d079420@mail.gmail.com>

On 8/23/07, Martin von L?wis <Martin.vonLoewis at hpi.uni-potsdam.de> wrote:
> We have completed the conversion to Roundup as the bug tracker
> for Python; the new tracker is now life at
>
>   http://bugs.python.org/
>
> It has all reports from the SourceForge tracker, along with
> all messages and files that had been attached.
>
> To edit bug reports, you still need to log in. We have migrated
> all account names from SF as-is, but obviously couldn't migrate
> the passwords as well. So you need to reset your password, through
>
>   http://bugs.python.org/user?@template=forgotten
>
> Fill your SF account name in the bottom field, wait for an email,
> click on the link, wait for a second email, change your password
> to something you can remember. All SF project members have developer
> privileges in the roundup tracker.
>

Just so people know, if you are unhappy with your SF username, you can
change it after you get your password reset.  Just go to your details
page and change your username to whatever you want.  Personally, I
think they should match your svn.python.org account name but I don't
know if anyone else wants that as a suggestion or policy.

-Brett

From brett at python.org  Thu Aug 23 22:38:36 2007
From: brett at python.org (Brett Cannon)
Date: Thu, 23 Aug 2007 13:38:36 -0700
Subject: [Python-Dev] PEP 302 optional extensions only for loaders?
Message-ID: <bbaeab100708231338w656af3f6w3a5b1a842169d4bf@mail.gmail.com>

When you read PEP 302 and the section on the optional extensions, it
mentions that they "are highly recommended for general purpose
importers".  But then the PEP says very shortly thereafter that
"loader objects" are to have the extensions.

So which is it, importers or loaders?  The reason I ask is because if
one can assume that the extensions are only called on what find_module
returns, and thus extensions are only called on loaders, then some
caching can be done based on what find_module had to do.  In my case I
am reimplementing zipimport and if I can assume find_module was called
before is_package I can just cache in the object what module names
were found to be a package and then have is_package just be a lookup
in a set instead of having to read the zip file every time or store
all of the ZipInfo files for the zipfile.

-Brett

From kbk at shore.net  Thu Aug 23 23:53:14 2007
From: kbk at shore.net (Kurt B. Kaiser)
Date: Thu, 23 Aug 2007 17:53:14 -0400 (EDT)
Subject: [Python-Dev] Final SF Python Patch/Bug Summary
Message-ID: <200708232153.l7NLrEUg021672@hampton.thirdcreek.com>

New / Reopened Patches
______________________

minidom pretty xml output improvement  (2007-08-19)
       http://python.org/sf/1777134  opened by  Teajay

removeTest() method patch for unittest.TestSuite  (2007-08-21)
       http://python.org/sf/1778410  opened by  Mark Edgington

robotparser.py fixes  (2007-08-21)
       http://python.org/sf/1778443  opened by  Indy

Remove redundancies inside class logging.Logger  (2007-08-22)
       http://python.org/sf/1779550  opened by  Carsten Grohmann

Replace AtheOS w/ Syllable  (2007-08-22)
       http://python.org/sf/1779613  opened by  Skip Montanaro

Make python build with gcc-4.2 on OS X 10.4.9  (2007-08-23)
       http://python.org/sf/1779871  opened by  Jeffrey Yasskin

Patches Closed
______________

distutils.util.get_platform() return value on 64bit Windows  (2007-07-27)
       http://python.org/sf/1761786  closed by  mhammond

MSVC6/7 issues with bdist_wininst and --target-version  (2004-07-01)
       http://python.org/sf/983164  closed by  mhammond

Override flags set by IOBase in io.StringIO  (2007-08-15)
       http://python.org/sf/1774828  closed by  avassalotti

Convert str to bytes in io.BytesIO.__init__  (2007-08-15)
       http://python.org/sf/1774833  closed by  avassalotti

BufReader, TextReader for PEP 3116 "New I/O"  (2007-06-04)
       http://python.org/sf/1731036  closed by  gvanrossum

utf-32 codecs  (2007-08-16)
       http://python.org/sf/1775604  closed by  gvanrossum

Make it possible to use SVK to develop Python  (2007-08-14)
       http://python.org/sf/1774414  closed by  collinwinter

New / Reopened Bugs
___________________

memoryview('test') is causing a segfault  (2007-08-19)
CLOSED http://python.org/sf/1777057  reopened by  tiran

Please warn about a subtle trap  (2007-08-19)
CLOSED http://python.org/sf/1777160  opened by  Greg Kochanski

Confusing typography Python Ref 5.9  (2007-08-19)
CLOSED http://python.org/sf/1777168  opened by  Greg Kochanski

IDLE Freezes After Running Scripts  (2007-08-19)
       http://python.org/sf/1777398  opened by  Ross Peoples

Python's strftime dislikes years before 1900  (2007-08-20)
       http://python.org/sf/1777412  opened by  Benno Rice

glob doesn't return unicode with unicode parameter  (2007-08-20)
       http://python.org/sf/1777458  opened by  Grzegorz Adam Hankiewicz

ctypes on Solaris  (2007-08-20)
       http://python.org/sf/1777530  opened by  Aki

rounding inconsisntency using string formatting  (2007-08-20)
CLOSED http://python.org/sf/1778207  opened by  Jim Hurlburt

Segfault.  (2007-08-21)
       http://python.org/sf/1778376  opened by  Maciek Fijalkowski

PyThreadState_SetAsyncExc and the main thread  (2007-08-22)
       http://python.org/sf/1779233  opened by  Rotem

urlparse.urljoin does not obey current uri rfc (rfc 3986)  (2007-08-22)
       http://python.org/sf/1779700  opened by  John Adams

Bugs Closed
___________

memoryview('test') is causing a segfault  (2007-08-18)
       http://python.org/sf/1777057  closed by  nnorwitz

memoryview('test') is causing a segfault  (2007-08-18)
       http://python.org/sf/1777057  closed by  nnorwitz

Please warn about a subtle trap  (2007-08-19)
       http://python.org/sf/1777160  closed by  gbrandl

Confusing typography Python Ref 5.9  (2007-08-19)
       http://python.org/sf/1777168  closed by  gbrandl

struni: str() doesn't call __str__() of subclasses of str  (2007-07-11)
       http://python.org/sf/1751598  closed by  tiran

rounding inconsisntency using string formatting  (2007-08-21)
       http://python.org/sf/1778207  closed by  loewis

mapitags.PROP_TAG() doesn't account for new longs  (2004-12-31)
       http://python.org/sf/1093389  closed by  mhammond

Overenthusiastic check in Swap?  (2004-04-01)
       http://python.org/sf/926910  closed by  mhammond


From p.f.moore at gmail.com  Thu Aug 23 23:54:31 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 23 Aug 2007 22:54:31 +0100
Subject: [Python-Dev] PEP 302 optional extensions only for loaders?
In-Reply-To: <bbaeab100708231338w656af3f6w3a5b1a842169d4bf@mail.gmail.com>
References: <bbaeab100708231338w656af3f6w3a5b1a842169d4bf@mail.gmail.com>
Message-ID: <79990c6b0708231454l2260dfbn38f1081283f0129f@mail.gmail.com>

On 23/08/07, Brett Cannon <brett at python.org> wrote:
> When you read PEP 302 and the section on the optional extensions, it
> mentions that they "are highly recommended for general purpose
> importers".  But then the PEP says very shortly thereafter that
> "loader objects" are to have the extensions.
>
> So which is it, importers or loaders?

Loaders. "general purpose importers" was sloppy terminology, meant to
mean "general purpose things that implement the new importing
protocol, as specified here".

Loaders are what implement the extensions, as made clear by the examples.

Apologies for the sloppy wording in the PEP.
Paul.

From brett at python.org  Fri Aug 24 00:16:10 2007
From: brett at python.org (Brett Cannon)
Date: Thu, 23 Aug 2007 15:16:10 -0700
Subject: [Python-Dev] Final SF Python Patch/Bug Summary
In-Reply-To: <200708232153.l7NLrEUg021672@hampton.thirdcreek.com>
References: <200708232153.l7NLrEUg021672@hampton.thirdcreek.com>
Message-ID: <bbaeab100708231516j2bac27ach5417b4a79b952a75@mail.gmail.com>

Thanks for having done these, Kurt!

-Brett

On 8/23/07, Kurt B. Kaiser <kbk at shore.net> wrote:
> New / Reopened Patches
> ______________________
>
> minidom pretty xml output improvement  (2007-08-19)
>        http://python.org/sf/1777134  opened by  Teajay
>
> removeTest() method patch for unittest.TestSuite  (2007-08-21)
>        http://python.org/sf/1778410  opened by  Mark Edgington
>
> robotparser.py fixes  (2007-08-21)
>        http://python.org/sf/1778443  opened by  Indy
>
> Remove redundancies inside class logging.Logger  (2007-08-22)
>        http://python.org/sf/1779550  opened by  Carsten Grohmann
>
> Replace AtheOS w/ Syllable  (2007-08-22)
>        http://python.org/sf/1779613  opened by  Skip Montanaro
>
> Make python build with gcc-4.2 on OS X 10.4.9  (2007-08-23)
>        http://python.org/sf/1779871  opened by  Jeffrey Yasskin
>
> Patches Closed
> ______________
>
> distutils.util.get_platform() return value on 64bit Windows  (2007-07-27)
>        http://python.org/sf/1761786  closed by  mhammond
>
> MSVC6/7 issues with bdist_wininst and --target-version  (2004-07-01)
>        http://python.org/sf/983164  closed by  mhammond
>
> Override flags set by IOBase in io.StringIO  (2007-08-15)
>        http://python.org/sf/1774828  closed by  avassalotti
>
> Convert str to bytes in io.BytesIO.__init__  (2007-08-15)
>        http://python.org/sf/1774833  closed by  avassalotti
>
> BufReader, TextReader for PEP 3116 "New I/O"  (2007-06-04)
>        http://python.org/sf/1731036  closed by  gvanrossum
>
> utf-32 codecs  (2007-08-16)
>        http://python.org/sf/1775604  closed by  gvanrossum
>
> Make it possible to use SVK to develop Python  (2007-08-14)
>        http://python.org/sf/1774414  closed by  collinwinter
>
> New / Reopened Bugs
> ___________________
>
> memoryview('test') is causing a segfault  (2007-08-19)
> CLOSED http://python.org/sf/1777057  reopened by  tiran
>
> Please warn about a subtle trap  (2007-08-19)
> CLOSED http://python.org/sf/1777160  opened by  Greg Kochanski
>
> Confusing typography Python Ref 5.9  (2007-08-19)
> CLOSED http://python.org/sf/1777168  opened by  Greg Kochanski
>
> IDLE Freezes After Running Scripts  (2007-08-19)
>        http://python.org/sf/1777398  opened by  Ross Peoples
>
> Python's strftime dislikes years before 1900  (2007-08-20)
>        http://python.org/sf/1777412  opened by  Benno Rice
>
> glob doesn't return unicode with unicode parameter  (2007-08-20)
>        http://python.org/sf/1777458  opened by  Grzegorz Adam Hankiewicz
>
> ctypes on Solaris  (2007-08-20)
>        http://python.org/sf/1777530  opened by  Aki
>
> rounding inconsisntency using string formatting  (2007-08-20)
> CLOSED http://python.org/sf/1778207  opened by  Jim Hurlburt
>
> Segfault.  (2007-08-21)
>        http://python.org/sf/1778376  opened by  Maciek Fijalkowski
>
> PyThreadState_SetAsyncExc and the main thread  (2007-08-22)
>        http://python.org/sf/1779233  opened by  Rotem
>
> urlparse.urljoin does not obey current uri rfc (rfc 3986)  (2007-08-22)
>        http://python.org/sf/1779700  opened by  John Adams
>
> Bugs Closed
> ___________
>
> memoryview('test') is causing a segfault  (2007-08-18)
>        http://python.org/sf/1777057  closed by  nnorwitz
>
> memoryview('test') is causing a segfault  (2007-08-18)
>        http://python.org/sf/1777057  closed by  nnorwitz
>
> Please warn about a subtle trap  (2007-08-19)
>        http://python.org/sf/1777160  closed by  gbrandl
>
> Confusing typography Python Ref 5.9  (2007-08-19)
>        http://python.org/sf/1777168  closed by  gbrandl
>
> struni: str() doesn't call __str__() of subclasses of str  (2007-07-11)
>        http://python.org/sf/1751598  closed by  tiran
>
> rounding inconsisntency using string formatting  (2007-08-21)
>        http://python.org/sf/1778207  closed by  loewis
>
> mapitags.PROP_TAG() doesn't account for new longs  (2004-12-31)
>        http://python.org/sf/1093389  closed by  mhammond
>
> Overenthusiastic check in Swap?  (2004-04-01)
>        http://python.org/sf/926910  closed by  mhammond
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>

From guido at python.org  Fri Aug 24 07:03:57 2007
From: guido at python.org (Guido van Rossum)
Date: Thu, 23 Aug 2007 22:03:57 -0700
Subject: [Python-Dev] uuid creation not thread-safe?
In-Reply-To: <1185671069.839769.274620@z28g2000prd.googlegroups.com>
References: <ca471dc20707201052p68883fc5l3efd8ecc5cfd497f@mail.gmail.com>
	<1185671069.839769.274620@z28g2000prd.googlegroups.com>
Message-ID: <ca471dc20708232203h5e918e3aw9af156da3b0fbb9b@mail.gmail.com>

This was now fixed in 3.0. Somebody might want to backport.

On 7/28/07, lcaamano <lcaamano at gmail.com> wrote:
>
> On Jul 20, 1:52 pm, "Guido van Rossum" <gu... at python.org> wrote:
> > I discovered what appears to be a thread-unsafety inuuid.py. This is
> > in the trunk as well as in 3.x; I'm using the trunk here for easy
> > reference. There's some code around like 395:
> >
> >     import ctypes, ctypes.util
> >     _buffer = ctypes.create_string_buffer(16)
> >
> > This creates a *global* buffer which is used as the output parameter
> > to later calls to _uuid_generate_random() and _uuid_generate_time().
> > For example, around line 481, in uuid1():
> >
> >         _uuid_generate_time(_buffer)
> >         returnUUID(bytes=_buffer.raw)
> >
> > Clearly if two threads do this simultaneously they are overwriting
> > _buffer in unpredictable order. There are a few other occurrences of
> > this too.
> >
> > I find it somewhat disturbing that what seems a fairly innocent
> > function that doesn't *appear* to have global state is nevertheless
> > not thread-safe. Would it be wise to fix this, e.g. by allocating a
> > fresh output buffer inside uuid1() and other callers?
> >
>
>
> I didn't find any reply to this, which is odd, so forgive me if it's
> old news.
>
> I agree with you that it's not thread safe and that a local buffer in
> the stack should fix it.
>
> Just for reference, the thread-safe uuid extension we've been using
> since python 2.1, which I don't recall where we borrow it from, uses a
> local buffer in the stack.  It looks like this:
>
> -----begin uuid.c--------------
>
> static char uuid__doc__ [] =
> "DCE compatible Universally Unique Identifier module";
>
> #include "Python.h"
> #include <uuid/uuid.h>
>
> static char uuidgen__doc__ [] =
> "Create a new DCE compatible UUID value";
>
> static PyObject *
> uuidgen(void)
> {
> uuid_t out;
> char buf[48];
>
>     uuid_generate(out);
>     uuid_unparse(out, buf);
>     return PyString_FromString(buf);
> }
>
> static PyMethodDef uuid_methods[] = {
>     {"uuidgen", uuidgen, 0, uuidgen__doc__},
>     {NULL,      NULL}        /* Sentinel */
> };
>
> DL_EXPORT(void)
> inituuid(void)
> {
>     Py_InitModule4("uuid",
>                uuid_methods,
>                uuid__doc__,
>                (PyObject *)NULL,
>                PYTHON_API_VERSION);
> }
>
> -----end uuid.c--------------
>
>
> It also seems that using uuid_generate()/uuid_unparse() should be
> faster than using uuid_generate_random() and then creating a python
> object to call its __str__ method.  If so, it would be nice if the
> uuid.py module also provided equivalent fast versions that returned
> strings instead of objects.
>
>
> --
> Luis P Caamano
> Atlanta, GA, USA
>
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From forsberg at efod.se  Thu Aug 23 22:06:01 2007
From: forsberg at efod.se (Erik Forsberg)
Date: Thu, 23 Aug 2007 22:06:01 +0200
Subject: [Python-Dev] New bug tracker is active now
References: <5AC0F51A926CD442BC08D55DA7DAB1D40670BB6A@3mxma1r.hpi.uni-potsdam.de>
	<46CDDEA8.3070801@acm.org> <46CDE69C.8030906@v.loewis.de>
Message-ID: <87lkc28192.fsf@uterus.efod.se>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"Martin v. L?wis" <martin at v.loewis.de> writes:

>> When editing my details I saw there is a field for my timezone.  The
>> comment says: "this is a numeric hour offset, the default is UTC", so
>> I'm assuming it counts in whole hours (fine for me, not so fine for
>> people in e.g. India, unless it accepts decimal notation).  I have two
>> issues with this: it's unclear which direction this works (is East
>> positive or negative?), and I will need to change this number twice
>> yearly.  In other words, it shouldn't be a number, it should be a name,
>> like the names used in Linux: Europe/Amsterdam.
>
> I think this will be fairly difficult to fix. Roundup doesn't have
> a timezone database; it just applies the delta to the UTC time. We are
> merely using roundup, instead of developing it.

Actually, there is some kind of symbolic time zone support via the
pytz module. If the pytz module is installed, you can provide the time
zone as a symbolic name. I don't know if it follows the DST changes.

I would suggest that someone enter a feature request in the meta
tracker, http://psf.upfronthosting.co.za/roundup/meta/, and we can
take a look.
>
> See
>
> http://sourceforge.net/tracker/index.php?func=detail&aid=818711&group_id=31577&atid=402791

Might be outdated. Could be a case of "sourceforge tracker is so hard
to use that you really don't want to use it". Yeah, it's kind of
ironic that roundup isn't running a roundup tracker..

Regards,
\EF
- -- 
Erik Forsberg                 http://efod.se
GPG/PGP Key: 1024D/0BAC89D9
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8+ <http://mailcrypt.sourceforge.net/>

iD8DBQFGzeilrJurFAusidkRAuxIAJ4xcm5m2/1u2X3zkdi99eAi2Ll+LwCgxoxc
MJ4mO+8my4mbnDv5L6eRRo0=
=bZog
-----END PGP SIGNATURE-----


From hrvoje.niksic at avl.com  Fri Aug 24 09:55:22 2007
From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=)
Date: Fri, 24 Aug 2007 09:55:22 +0200
Subject: [Python-Dev] Segfault
In-Reply-To: <ee2a432c0708231157j65f5b7e7yd23c0430f9e1c5d3@mail.gmail.com>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
	<1187771718.11322.47.camel@localhost>
	<ee2a432c0708222132v6fbcf29bx7ea617ba6ea2887d@mail.gmail.com>
	<1187854696.11322.67.camel@localhost>
	<ee2a432c0708231157j65f5b7e7yd23c0430f9e1c5d3@mail.gmail.com>
Message-ID: <1187942122.11322.71.camel@localhost>

On Thu, 2007-08-23 at 11:57 -0700, Neal Norwitz wrote:
> > "allow threads" section that runs with the GIL released, file_close
> > might acquire the GIL and be running in parallel to this code.  If
> > file_close sets f_fp to NULL after the "if" condition evaluates, but
> > before the call to _portable_fseek completes, we still get a segfault.
> 
> However, the setting of f_fp to NULL happens with the GIL set, ie
> while only one thread is running.

I thought other "IO-bound" threads (the sections that "allow threads")
could also run in parallel.  Otherwise what is the point of
BEGIN_ALLOW_THREADS?



From nayuhz at gmail.com  Fri Aug 24 13:51:55 2007
From: nayuhz at gmail.com (eSX)
Date: Fri, 24 Aug 2007 19:51:55 +0800
Subject: [Python-Dev] any problem with py's RC?
Message-ID: <8ae5ea120708240451m624dd81pd55f3736d29028b1@mail.gmail.com>

hi all,
Here is 3 PYs, we found there is a big memory wasting, specially the program
c.py, we found the memory will increase very soon.
any body knows why? any RC's problem? thanks

-- 
eSX
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070824/22db45dc/attachment.htm 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: b.py
Url: http://mail.python.org/pipermail/python-dev/attachments/20070824/22db45dc/attachment.txt 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: c.py
Url: http://mail.python.org/pipermail/python-dev/attachments/20070824/22db45dc/attachment-0001.txt 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: haha.py
Url: http://mail.python.org/pipermail/python-dev/attachments/20070824/22db45dc/attachment-0002.txt 

From aahz at pythoncraft.com  Fri Aug 24 15:00:42 2007
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 24 Aug 2007 06:00:42 -0700
Subject: [Python-Dev] any problem with py's RC?
In-Reply-To: <8ae5ea120708240451m624dd81pd55f3736d29028b1@mail.gmail.com>
References: <8ae5ea120708240451m624dd81pd55f3736d29028b1@mail.gmail.com>
Message-ID: <20070824130042.GA18456@panix.com>

On Fri, Aug 24, 2007, eSX wrote:
>
> Here is 3 PYs, we found there is a big memory wasting, specially the
> program c.py, we found the memory will increase very soon. any body
> knows why? any RC's problem? thanks

Please post your question to comp.lang.python; python-dev is not a
suitable place for questions about how Python works.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"If you don't know what your program is supposed to do, you'd better not
start writing it."  --Dijkstra

From nnorwitz at gmail.com  Fri Aug 24 18:01:25 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Fri, 24 Aug 2007 09:01:25 -0700
Subject: [Python-Dev] Segfault
In-Reply-To: <1187942122.11322.71.camel@localhost>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
	<1187771718.11322.47.camel@localhost>
	<ee2a432c0708222132v6fbcf29bx7ea617ba6ea2887d@mail.gmail.com>
	<1187854696.11322.67.camel@localhost>
	<ee2a432c0708231157j65f5b7e7yd23c0430f9e1c5d3@mail.gmail.com>
	<1187942122.11322.71.camel@localhost>
Message-ID: <ee2a432c0708240901x369c9886odcc96856e5aaaa5f@mail.gmail.com>

On 8/24/07, Hrvoje Nik?i? <hrvoje.niksic at avl.com> wrote:
> On Thu, 2007-08-23 at 11:57 -0700, Neal Norwitz wrote:
> > > "allow threads" section that runs with the GIL released, file_close
> > > might acquire the GIL and be running in parallel to this code.  If
> > > file_close sets f_fp to NULL after the "if" condition evaluates, but
> > > before the call to _portable_fseek completes, we still get a segfault.
> >
> > However, the setting of f_fp to NULL happens with the GIL set, ie
> > while only one thread is running.
>
> I thought other "IO-bound" threads (the sections that "allow threads")
> could also run in parallel.  Otherwise what is the point of
> BEGIN_ALLOW_THREADS?

Right.  I looked at this with Jeffrey Yasskin and agree that a lock is
needed for f_fp.  The fix is ugly because it's needed around all
accesses to f_fp and there are a ton of them.  Some are with the GIL
held and others when it isn't.

I would be fine with the much simpler approach in the patch I sent
(assuming the patch is finished).  It doesn't completely address the
problem, but does make the race condition much, much smaller.  Part of
the reason I'm ok with this is that in 3.0, all this code has already
been removed.

n

From jmatejek at suse.cz  Fri Aug 24 19:36:41 2007
From: jmatejek at suse.cz (Jan Matejek)
Date: Fri, 24 Aug 2007 19:36:41 +0200
Subject: [Python-Dev] tarfile and directory traversal vulnerability
Message-ID: <46CF1729.5020308@suse.cz>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

hi,
once upon a time there was a known vulnerability in tar (CVE-2001-1267,
[1]), and while tar is now long fixed, python's tarfile module is
affected too.

The vulnerability goes basically like this: If you tar a file named
"../../../../../etc/passwd" and then make the admin untar it,
/etc/passwd gets overwritten.
Another variety of this bug is a symlink one: if tar contains files like:
./aaaa-directory -> /etc
./aaaa-directory/passwd
then the "aaaa-directory" symlink would be created first and /etc/passwd
will be overwritten once again.

I was wondering how to fix it.
The symlink problem obviously applies only to extractall() method and is
easily fixed by delaying external (or possibly all) symlink creation,
similar to how directory attributes are delayed now.
I've attached a draft of the patch, if you like it, i'll polish it.

The traversal problem is harder, and it applies to extract() method as well.
For extractall() alone, i would use something like:

if tarinfo.name.startswith('../'):
    self.extract(tarinfo, path)
else:
    warnings.warn("non-local file skipped: %s" % tarinfo.name,
RuntimeWarning, stacklevel=1)

For extract(), i am not sure. Maybe it should throw exception when it
encounters such file, and have a special option to extract such files
anyway. Or maybe it should be left alone altogether.

Any suggestions are welcome.

regards
jan matejek

[1] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2001-1267
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4-svn0 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFGzxcpjBrWA+AvBr8RAlduAKCk0iiSoBF+wA9xgXmDlpWsECZ7KgCfQORg
lZ85inT1FGwhGqBfxJvCGGU=
=TiWx
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: python-2.5.1-tarfile-traversal.patch
Type: text/x-patch
Size: 1127 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20070824/5605f4ef/attachment.bin 

From martin at v.loewis.de  Sat Aug 25 00:38:14 2007
From: martin at v.loewis.de (=?ISO-8859-2?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 25 Aug 2007 00:38:14 +0200
Subject: [Python-Dev] tarfile and directory traversal vulnerability
In-Reply-To: <46CF1729.5020308@suse.cz>
References: <46CF1729.5020308@suse.cz>
Message-ID: <46CF5DD6.9080701@v.loewis.de>

> The vulnerability goes basically like this: If you tar a file named
> "../../../../../etc/passwd" and then make the admin untar it,
> /etc/passwd gets overwritten.
> Another variety of this bug is a symlink one: if tar contains files like:
> ./aaaa-directory -> /etc
> ./aaaa-directory/passwd
> then the "aaaa-directory" symlink would be created first and /etc/passwd
> will be overwritten once again.

I must admit I fail to see the bug. If root untars a file, and that tar
file contains an instruction to overwrite /etc/passwd, why is an error
to execute that instruction? Shouldn't root just be more careful when
untaring files?

> if tarinfo.name.startswith('../'):
>     self.extract(tarinfo, path)
> else:
>     warnings.warn("non-local file skipped: %s" % tarinfo.name,
> RuntimeWarning, stacklevel=1)

Ok. You seem to be claiming that the tarfile is incorrect in some
sense. Can you please point to some spec that says this is an incorrect
tarfile?

In any case, if you fix what you consider broken, you should do
it exactly the same way as GNU tar does it (assuming you consider
GNU tar fixed).

Regards,
Martin

From lars at gustaebel.de  Sat Aug 25 12:13:12 2007
From: lars at gustaebel.de (Lars =?iso-8859-15?Q?Gust=E4bel?=)
Date: Sat, 25 Aug 2007 12:13:12 +0200
Subject: [Python-Dev] tarfile and directory traversal vulnerability
In-Reply-To: <46CF1729.5020308@suse.cz>
References: <46CF1729.5020308@suse.cz>
Message-ID: <20070825101312.GA31481@core.g33x.de>

On Fri, Aug 24, 2007 at 07:36:41PM +0200, Jan Matejek wrote:
> once upon a time there was a known vulnerability in tar (CVE-2001-1267,
> [1]), and while tar is now long fixed, python's tarfile module is
> affected too.
> 
> The vulnerability goes basically like this: If you tar a file named
> "../../../../../etc/passwd" and then make the admin untar it,
> /etc/passwd gets overwritten.
> Another variety of this bug is a symlink one: if tar contains files like:
> ./aaaa-directory -> /etc
> ./aaaa-directory/passwd
> then the "aaaa-directory" symlink would be created first and /etc/passwd
> will be overwritten once again.

tarfile currently contains no sanity checks at all. The easiest
way to attack /etc/passwd would be to give tarfile a tar created
with `tar -cPf foo.tar /etc/passwd'.

> I was wondering how to fix it.
> The symlink problem obviously applies only to extractall() method and is
> easily fixed by delaying external (or possibly all) symlink creation,
> similar to how directory attributes are delayed now.
> I've attached a draft of the patch, if you like it, i'll polish it.

Suppose we have:
foo -> /etc
foo/passwd

If creation of the foo symlink is delayed, foo/passwd will be
extracted in a directory foo which will be created implicitly.
If we create the foo symlink afterwards it will fail because foo
already exists. The best way would be to completely ignore
members and link targets that are absolute or outside the
archive's scope.

> The traversal problem is harder, and it applies to extract() method as well.
> For extractall() alone, i would use something like:
> 
> if tarinfo.name.startswith('../'):
>     self.extract(tarinfo, path)
> else:
>     warnings.warn("non-local file skipped: %s" % tarinfo.name,
> RuntimeWarning, stacklevel=1)
> 
> For extract(), i am not sure. Maybe it should throw exception when it
> encounters such file, and have a special option to extract such files
> anyway. [...]

Yes, I think that is the right way to do it.

-- 
Lars Gust?bel
lars at gustaebel.de

A chicken is an egg's way of producing more eggs.
(Anonymous)

From janssen at parc.com  Sat Aug 25 20:06:40 2007
From: janssen at parc.com (Bill Janssen)
Date: Sat, 25 Aug 2007 11:06:40 PDT
Subject: [Python-Dev] new bug tracker broken?
Message-ID: <07Aug25.110646pdt."57996"@synergy1.parc.xerox.com>

I've been trying to reset my password on the new RoundUp tracker,
and it isn't working.  Then I tried to register for a new account,
and I got mail from

   metatracker at psf.upfronthosting.co.za

to confirm.  When I did, I got access to a small 27-issue tracker,
which seems to be the bug-tracker for the bug-tracker.  Still can't
get into the Python bug-tracker.

Bill

From nnorwitz at gmail.com  Sat Aug 25 23:30:30 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sat, 25 Aug 2007 14:30:30 -0700
Subject: [Python-Dev] new bug tracker broken?
In-Reply-To: <1504448987588039863@unknownmsgid>
References: <1504448987588039863@unknownmsgid>
Message-ID: <ee2a432c0708251430n393a10ecg919fa3a0db51a177@mail.gmail.com>

It looks like you found the meta tracker rather than the python one.
To reset your password for the main python tracker:

  http://bugs.python.org/user?@template=forgotten

To report bugs, go here:
  http://bugs.python.org/

To register a new account, go here:
  http://bugs.python.org/user?@template=register

Note, I'm not sure if you have an account, you might need to register.

To report problems about the bug tracker (ie, not python problems), go here:
 http://psf.upfronthosting.co.za/roundup/meta/

n
--

On 8/25/07, Bill Janssen <janssen at parc.com> wrote:
> I've been trying to reset my password on the new RoundUp tracker,
> and it isn't working.  Then I tried to register for a new account,
> and I got mail from
>
>    metatracker at psf.upfronthosting.co.za
>
> to confirm.  When I did, I got access to a small 27-issue tracker,
> which seems to be the bug-tracker for the bug-tracker.  Still can't
> get into the Python bug-tracker.
>
> Bill
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From greg at krypto.org  Sun Aug 26 00:59:21 2007
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 25 Aug 2007 15:59:21 -0700
Subject: [Python-Dev] Avoiding cascading test failures
In-Reply-To: <acd65fa20708221644y2ab376bemd4d550ff43fdd91@mail.gmail.com>
References: <acd65fa20708221644y2ab376bemd4d550ff43fdd91@mail.gmail.com>
Message-ID: <20070825225921.GS24059@electricrain.com>

On Wed, Aug 22, 2007 at 07:44:02PM -0400, Alexandre Vassalotti wrote:
> When I was fixing tests failing in the py3k branch, I found the number
> duplicate failures annoying. Often, a single bug, in an important
> method or function, caused a large number of testcase to fail. So, I
> thought of a simple mechanism for avoiding such cascading failures.
> 
> My solution is to add a notion of dependency to testcases. A typical
> usage would look like this:
> 
>     @depends('test_getvalue')
>     def test_writelines(self):
>         ...
>         memio.writelines([buf] * 100)
>         self.assertEqual(memio.getvalue(), buf * 100)
>         ...
> 
> Here, running the test is pointless if test_getvalue fails. So by
> making test_writelines depends on the success of test_getvalue, we can
> ensure that the report won't be polluted with unnecessary failures.
> 
> Also, I believe this feature will lead to more orthogonal tests, since
> it encourages the user to write smaller test with less dependencies.
> 
> I wrote an example implementation (included below) as a proof of
> concept. If the idea get enough support, I will implement it and add
> it to the unittest module.
> 
> -- Alexandre

I like this idea.

Be sure to have an option to ignore dependancies and run all tests.
Also when skipping tests because a depedancy failed have unittest
print out an indication that a test was skipped due to a dependancy
rather than silently running fewer tests.  Otherwise it could be
deceptive and appear that only one test was affected.

Greg


From greg at krypto.org  Sun Aug 26 00:45:07 2007
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 25 Aug 2007 15:45:07 -0700
Subject: [Python-Dev] New bug tracker is active now
In-Reply-To: <87lkc28192.fsf@uterus.efod.se>
References: <5AC0F51A926CD442BC08D55DA7DAB1D40670BB6A@3mxma1r.hpi.uni-potsdam.de>
	<46CDDEA8.3070801@acm.org> <46CDE69C.8030906@v.loewis.de>
	<87lkc28192.fsf@uterus.efod.se>
Message-ID: <20070825224507.GR24059@electricrain.com>

On Thu, Aug 23, 2007 at 10:06:01PM +0200, Erik Forsberg wrote:
> "Martin v. L?wis" <martin at v.loewis.de> writes:
> 
> >> When editing my details I saw there is a field for my timezone.  The
> >> comment says: "this is a numeric hour offset, the default is UTC", so
> >> I'm assuming it counts in whole hours (fine for me, not so fine for
> >> people in e.g. India, unless it accepts decimal notation).  I have two
> >> issues with this: it's unclear which direction this works (is East
> >> positive or negative?), and I will need to change this number twice
> >> yearly.  In other words, it shouldn't be a number, it should be a name,
> >> like the names used in Linux: Europe/Amsterdam.
> >
> > I think this will be fairly difficult to fix. Roundup doesn't have
> > a timezone database; it just applies the delta to the UTC time. We are
> > merely using roundup, instead of developing it.
> 
> Actually, there is some kind of symbolic time zone support via the
> pytz module. If the pytz module is installed, you can provide the time
> zone as a symbolic name. I don't know if it follows the DST changes.
> 
> I would suggest that someone enter a feature request in the meta
> tracker, http://psf.upfronthosting.co.za/roundup/meta/, and we can
> take a look.

I entered a bug to see if they'll fix it:

 http://psf.upfronthosting.co.za/roundup/meta/msg712

Proper timzeone support via pytz has been in roundup since the october
2006 1.2.0 release according to the changelog.

-greg


From skip at pobox.com  Sun Aug 26 01:26:04 2007
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 25 Aug 2007 18:26:04 -0500
Subject: [Python-Dev] new bug tracker broken?
In-Reply-To: <ee2a432c0708251430n393a10ecg919fa3a0db51a177@mail.gmail.com>
References: <1504448987588039863@unknownmsgid>
	<ee2a432c0708251430n393a10ecg919fa3a0db51a177@mail.gmail.com>
Message-ID: <18128.47756.179550.624700@montanaro.dyndns.org>


    Neal> Note, I'm not sure if you have an account, you might need to
    Neal> register.

But try your SF id first.

Skip

From alexandre at peadrop.com  Sun Aug 26 02:23:02 2007
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Sat, 25 Aug 2007 20:23:02 -0400
Subject: [Python-Dev] Avoiding cascading test failures
In-Reply-To: <20070825225921.GS24059@electricrain.com>
References: <acd65fa20708221644y2ab376bemd4d550ff43fdd91@mail.gmail.com>
	<20070825225921.GS24059@electricrain.com>
Message-ID: <acd65fa20708251723p5fa697b9p22e9ea9e0cc7e47@mail.gmail.com>

On 8/25/07, Gregory P. Smith <greg at krypto.org> wrote:
> I like this idea.

Yay! Now, I ain't the only one. ;)

> Be sure to have an option to ignore dependancies and run all tests.

Yes, I planned to add a such option.

> Also when skipping tests because a depedancy failed have unittest
> print out an indication that a test was skipped due to a dependancy
> rather than silently running fewer tests.  Otherwise it could be
> deceptive and appear that only one test was affected.

However, that was never planned.

I added the ignore_dependencies option. Also, I fixed the sub-optimal
dependency resolution algorithm that was in my original example
implementation.

-- Alexandre

--- dep.py.old  2007-08-25 19:54:27.000000000 -0400
+++ dep.py      2007-08-25 20:02:55.000000000 -0400
@@ -2,8 +2,9 @@
 class CycleError(Exception):
     pass

+class TestGraph:

-class TestCase:
+    ignore_dependencies = False

     def __init__(self):
         self.graph = {}
@@ -19,16 +20,16 @@
         graph = self.graph
         toskip = set()
         msgs = []
-        while graph:
+        if self.ignore_dependencies:
+            for test in graph:
+                graph[test].clear()
             # find tests without any pending dependencies
-            source = [test for test, deps in graph.items() if not deps]
-            if not source:
-                raise CycleError
-            for testname in source:
+        queue = [test for test, deps in graph.items() if not deps]
+        while queue:
+            testname = queue.pop()
                 if testname in toskip:
                     msgs.append("%s... skipped" % testname)
-                    resolvedeps(graph, testname)
-                    del graph[testname]
+                queue.extend(resolve(graph, testname))
                     continue
                 test = getattr(self, testname)
                 try:
@@ -42,8 +43,9 @@
                 else:
                     msgs.append("%s... ok" % testname)
                 finally:
-                    resolvedeps(graph, testname)
-                    del graph[testname]
+                queue.extend(resolve(graph, testname))
+        if graph:
+            raise CycleError
         for msg in sorted(msgs):
             print(msg)

@@ -60,10 +62,15 @@
             rdeps.update(getrevdeps(graph, x))
     return rdeps

- def resolvedeps(graph, testname):
+def resolve(graph, testname):
+    toqueue = []
     for test in graph:
         if testname in graph[test]:
             graph[test].remove(testname)
+            if not graph[test]:
+                toqueue.append(test)
+    del graph[testname]
+    return toqueue

 def depends(*args):
     def decorator(test):
@@ -75,7 +82,9 @@
     return decorator


-class MyTest(TestCase):
+class MyTest(TestGraph):
+
+    ignore_dependencies = True

    @depends('test_foo')
    def test_nah(self):
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dep.py
Type: text/x-python
Size: 2884 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20070825/df266f1e/attachment.py 

From janssen at parc.com  Sun Aug 26 04:47:24 2007
From: janssen at parc.com (Bill Janssen)
Date: Sat, 25 Aug 2007 19:47:24 PDT
Subject: [Python-Dev] issue 1024 contains documentation patch for SSL work
Message-ID: <07Aug25.194733pdt."57996"@synergy1.parc.xerox.com>

I've created the documentation patch for the new SSL module.

It's attached to issue 1024.

Bill

From janssen at parc.com  Sun Aug 26 05:13:47 2007
From: janssen at parc.com (Bill Janssen)
Date: Sat, 25 Aug 2007 20:13:47 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
Message-ID: <07Aug25.201354pdt."57996"@synergy1.parc.xerox.com>

I've gone through the other open SSL issues.  Looks like some can be
closed with the adoption of 1018 and 1024:

1027394	4 months ago	socket.ssl should explain that it is a 2/3 connection
889813	4 months ago	making the version of SSL configurable when creating sockets
1583946	9 months ago	SSL "issuer" and "server" names cannot be parsed
783188	46 months ago	support for server side transactions in _ssl

Others are about various standard libraries that interact with SSL 
in various ways.  I'm working on another patch that converts all the
standard library modules over to use the new ssl module, and I'll look
at the rest of the SSL-related bugs as part of that work.

Bill

From nnorwitz at gmail.com  Sun Aug 26 05:27:09 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sat, 25 Aug 2007 20:27:09 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <166458247748783241@unknownmsgid>
References: <166458247748783241@unknownmsgid>
Message-ID: <ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>

Bill,

Could you also look into this problem:

Traceback (most recent call last):
  File "/home/pybot/buildarea/trunk.klose-debian-ia64/build/Lib/threading.py",
line 486, in __bootstrap_inner
    self.run()
  File "/home/pybot/buildarea/trunk.klose-debian-ia64/build/Lib/test/test_ssl.py",
line 144, in run
    cert_reqs=self.server.certreqs)
  File "/home/pybot/buildarea/trunk.klose-debian-ia64/build/Lib/ssl.py",
line 98, in __init__
    cert_reqs, ssl_version, ca_certs)
sslerror: _ssl.c:271: SSL_CTX_use_PrivateKey_file error

This occurs on at least 3 of the buildbots (ubuntu and debian on ia64,
ppc, and hppa).  Here's one example:

http://python.org/dev/buildbot/all/ia64%20Ubuntu%20trunk%20trunk/builds/832/step-test/0

This also looks like it's not working on windows, but there is no info
from here:
http://python.org/dev/buildbot/all/x86%20XP-3%20trunk/builds/164/step-test/0

Other than:
test_ssl
The system cannot find the path specified.

Which happens after it hangs for 1200 seconds.

Thanks,
n
--
On 8/25/07, Bill Janssen <janssen at parc.com> wrote:
> I've gone through the other open SSL issues.  Looks like some can be
> closed with the adoption of 1018 and 1024:
>
> 1027394 4 months ago    socket.ssl should explain that it is a 2/3 connection
> 889813  4 months ago    making the version of SSL configurable when creating sockets
> 1583946 9 months ago    SSL "issuer" and "server" names cannot be parsed
> 783188  46 months ago   support for server side transactions in _ssl
>
> Others are about various standard libraries that interact with SSL
> in various ways.  I'm working on another patch that converts all the
> standard library modules over to use the new ssl module, and I'll look
> at the rest of the SSL-related bugs as part of that work.
>
> Bill
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From janssen at parc.com  Sun Aug 26 19:28:16 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 26 Aug 2007 10:28:16 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
Message-ID: <07Aug26.102820pdt."57996"@synergy1.parc.xerox.com>

> This occurs on at least 3 of the buildbots (ubuntu and debian on ia64,
> ppc, and hppa).  Here's one example:

Unfortunately, I don't have Ubuntu or Debian machines.  But I'd bet
it's a variation in the specific version of OpenSSL being used.  I
just tested on Fedora Core 7, though, and test_ssl runs fine there.
What version of OpenSSL is being used on the buildbots?  Can I log
into one of them and look at the configuration?

> This also looks like it's not working on windows, but there is no info
> from here:
> http://python.org/dev/buildbot/all/x86%20XP-3%20trunk/builds/164/step-test/0
> 
> Other than:
> test_ssl
> The system cannot find the path specified.

This must be the call to os.system in test_ssl.py:create_cert_files().
It's very UNIX-y.  Can any bi-platform folks suggest a good
alternative to

    os.system(
        "openssl req -batch -new -x509 -days 10 -nodes -config %s "
        "-keyout \"%s\" -out \"%s\" > /dev/null < /dev/null 2>&1" %
        (conffile, crtfile, crtfile))

that would be more Windows-friendly?

Bill

From janssen at parc.com  Sun Aug 26 19:55:43 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 26 Aug 2007 10:55:43 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
Message-ID: <07Aug26.105545pdt."57996"@synergy1.parc.xerox.com>

> This occurs on at least 3 of the buildbots (ubuntu and debian on ia64,
> ppc, and hppa).  Here's one example:
> 
> http://python.org/dev/buildbot/all/ia64%20Ubuntu%20trunk%20trunk/builds/832/step-test/0

If I'm reading this right, it's passing tests on "amd64 gentoo trunk",
"x86 gentoo trunk", "g4 osx.4 trunk" (no surprise there).

And looking at the community buildbots, it works on "x86 Redhat 9",
"x86 Debian unstable", "amd64 Ubuntu gutsy", "G5 OS X", and so on.
I've tested it myself on FC 7 and it works.

And looking at the "ppc Debian unstable" case, test_socket is also
failing there, so the test_ssl failure is not a big surprise.

I'm not familiar with what's in Debian "trunk" or Ubuntu "trunk"; any
idea what version of OpenSSL they have in them?

But I think this exposes a more generic bug in test_ssl.py, which is
that the server thread doesn't die when one of these failures occurs.
It probably should.  I'll make a patch -- but I don't have a system
that this fails on, how will I test it?

Bill

From nnorwitz at gmail.com  Sun Aug 26 20:39:57 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sun, 26 Aug 2007 11:39:57 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <7051374911840721061@unknownmsgid>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
Message-ID: <ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>

On 8/26/07, Bill Janssen <janssen at parc.com> wrote:
> > This occurs on at least 3 of the buildbots (ubuntu and debian on ia64,
> > ppc, and hppa).  Here's one example:
> >
> > http://python.org/dev/buildbot/all/ia64%20Ubuntu%20trunk%20trunk/builds/832/step-test/0
>
> If I'm reading this right, it's passing tests on "amd64 gentoo trunk",
> "x86 gentoo trunk", "g4 osx.4 trunk" (no surprise there).
>
> And looking at the community buildbots, it works on "x86 Redhat 9",
> "x86 Debian unstable", "amd64 Ubuntu gutsy", "G5 OS X", and so on.
> I've tested it myself on FC 7 and it works.
>
> And looking at the "ppc Debian unstable" case, test_socket is also
> failing there, so the test_ssl failure is not a big surprise.
>
> I'm not familiar with what's in Debian "trunk" or Ubuntu "trunk"; any
> idea what version of OpenSSL they have in them?

For ia64 http://python.org/dev/buildbot/all/ia64%20Ubuntu%20trunk%20trunk/builds/833/step-test/0

pybot at noisy:~$ openssl version
OpenSSL 0.9.8b 04 May 2006

I have access to some of the machines but not all of them.  All of
these run inside a chroot which might be causing a problem.  I
remember some issues with getting socket stuff to work right on them.
But that was over a year ago and I don't remember the details now.
:-( svn/google probably knows if you want to trawl through checkins.
I'm not sure that will help much though.

On this machines I was able to successfully make a key.  That's why
I'm thinking it might be some strange socket issue.

> But I think this exposes a more generic bug in test_ssl.py, which is
> that the server thread doesn't die when one of these failures occurs.
> It probably should.  I'll make a patch -- but I don't have a system
> that this fails on, how will I test it?

Yeah, I know this is difficult.  Hopefully someone with WIndows will
step up to help.  We can at least make the test more robust and verify
the files exist and are non-zero in size.  I will do that now.  At
least the it shouldn't cause the test to time out.

n

From janssen at parc.com  Sun Aug 26 20:51:14 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 26 Aug 2007 11:51:14 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <07Aug26.105545pdt."57996"@synergy1.parc.xerox.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<07Aug26.105545pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug26.115116pdt."57996"@synergy1.parc.xerox.com>

> But I think this exposes a more generic bug in test_ssl.py, which is
> that the server thread doesn't die when one of these failures occurs.
> It probably should.  I'll make a patch -- but I don't have a system
> that this fails on, how will I test it?

Here's a patch which makes test_ssl a better player in the buildbots
environment.  I deep-ended on "try-except-else" clauses.

Should I post this as an issue to the Tracker?

Bill

Index: Lib/ssl.py
===================================================================
--- Lib/ssl.py	(revision 57506)
+++ Lib/ssl.py	(working copy)
@@ -100,12 +100,13 @@
             # see if it's connected
             try:
                 socket.getpeername(self)
-                # yes
+            except:
+                # no, no connection yet
+                self._sslobj = None
+            else:
+                # yes, create the SSL object
                 self._sslobj = _ssl.sslwrap(self._sock, 0, keyfile, certfile,
                                             cert_reqs, ssl_version, ca_certs)
-            except:
-                # no
-                self._sslobj = None
         self.keyfile = keyfile
         self.certfile = certfile
         self.cert_reqs = cert_reqs
Index: Lib/test/test_ssl.py
===================================================================
--- Lib/test/test_ssl.py	(revision 57506)
+++ Lib/test/test_ssl.py	(working copy)
@@ -91,38 +91,66 @@
     def testTLSecho (self):
 
         s1 = socket.socket()
-        s1.connect(('127.0.0.1', 10024))
-        c1 = ssl.sslsocket(s1, ssl_version=ssl.PROTOCOL_TLSv1)
-        indata = "FOO\n"
-        c1.write(indata)
-        outdata = c1.read()
-        if outdata != indata.lower():
-            sys.stderr.write("bad data <<%s>> received\n" % data)
-        c1.close()
+        try:
+            s1.connect(('127.0.0.1', 10024))
+        except:
+            sys.stdout.write("connection failure:\n" + string.join(
+                traceback.format_exception(*sys.exc_info())))
+            raise test_support.TestFailed("Can't connect to test server")
+        else:
+            try:
+                c1 = ssl.sslsocket(s1, ssl_version=ssl.PROTOCOL_TLSv1)
+            except:
+                sys.stdout.write("SSL handshake failure:\n" + string.join(
+                    traceback.format_exception(*sys.exc_info())))
+                raise test_support.TestFailed("Can't SSL-handshake with test server")
+            else:
+                if not c1:
+                    raise test_support.TestFailed("Can't SSL-handshake with test server")
+                indata = "FOO\n"
+                c1.write(indata)
+                outdata = c1.read()
+                if outdata != indata.lower():
+                    raise test_support.TestFailed("bad data <<%s>> received; expected <<%s>>\n" % (data, indata.lower()))
+                c1.close()
 
     def testReadCert(self):
 
         s2 = socket.socket()
-        s2.connect(('127.0.0.1', 10024))
-        c2 = ssl.sslsocket(s2, ssl_version=ssl.PROTOCOL_TLSv1,
-                           cert_reqs=ssl.CERT_REQUIRED, ca_certs=CERTFILE)
-        cert = c2.getpeercert()
-        if not cert:
-            raise test_support.TestFailed("Can't get peer certificate.")
-        if not cert.has_key('subject'):
-            raise test_support.TestFailed(
-                "No subject field in certificate: %s." %
-                pprint.pformat(cert))
-        if not (cert['subject'].has_key('organizationName')):
-            raise test_support.TestFailed(
-                "No 'organizationName' field in certificate subject: %s." %
-                pprint.pformat(cert))
-        if (cert['subject']['organizationName'] !=
-              "Python Software Foundation"):
-            raise test_support.TestFailed(
-                "Invalid 'organizationName' field in certificate subject; "
-                "should be 'Python Software Foundation'.");
-        c2.close()
+        try:
+            s2.connect(('127.0.0.1', 10024))
+        except:
+            sys.stdout.write("connection failure:\n" + string.join(
+                traceback.format_exception(*sys.exc_info())))
+            raise test_support.TestFailed("Can't connect to test server")
+        else:
+            try:
+                c2 = ssl.sslsocket(s2, ssl_version=ssl.PROTOCOL_TLSv1,
+                                   cert_reqs=ssl.CERT_REQUIRED, ca_certs=CERTFILE)
+            except:
+                sys.stdout.write("SSL handshake failure:\n" + string.join(
+                    traceback.format_exception(*sys.exc_info())))
+                raise test_support.TestFailed("Can't SSL-handshake with test server")
+            else:
+                if not c2:
+                    raise test_support.TestFailed("Can't SSL-handshake with test server")
+                cert = c2.getpeercert()
+                if not cert:
+                    raise test_support.TestFailed("Can't get peer certificate.")
+                if not cert.has_key('subject'):
+                    raise test_support.TestFailed(
+                        "No subject field in certificate: %s." %
+                        pprint.pformat(cert))
+                if not (cert['subject'].has_key('organizationName')):
+                    raise test_support.TestFailed(
+                        "No 'organizationName' field in certificate subject: %s." %
+                        pprint.pformat(cert))
+                if (cert['subject']['organizationName'] !=
+                      "Python Software Foundation"):
+                    raise test_support.TestFailed(
+                        "Invalid 'organizationName' field in certificate subject; "
+                        "should be 'Python Software Foundation'.");
+                c2.close()
 
 
 class threadedEchoServer(threading.Thread):
@@ -138,10 +166,22 @@
 
         def run (self):
             self.running = True
-            sslconn = ssl.sslsocket(self.sock, server_side=True,
-                                    certfile=self.server.certificate,
-                                    ssl_version=self.server.protocol,
-                                    cert_reqs=self.server.certreqs)
+            try:
+                sslconn = ssl.sslsocket(self.sock, server_side=True,
+                                        certfile=self.server.certificate,
+                                        ssl_version=self.server.protocol,
+                                        cert_reqs=self.server.certreqs)
+            except:
+                # here, we want to stop the server, because this shouldn't
+                # happen in the context of our test case
+                sys.stdout.write("Test server failure:\n" + string.join(
+                    traceback.format_exception(*sys.exc_info())))
+                self.running = False
+                # normally, we'd just stop here, but for the test
+                # harness, we want to stop the server
+                self.server.stop()
+                return
+
             while self.running:
                 try:
                     msg = sslconn.read()
@@ -154,15 +194,18 @@
                         self.server.stop()
                         self.running = False
                     else:
-                        # print "server:", msg.strip().lower()
+                        sys.stdout.write("\nserver: %s\n" % msg.strip().lower())
                         sslconn.write(msg.lower())
                 except ssl.sslerror:
-                    sys.stderr.write(string.join(
+                    sys.stdout.write("Test server failure:\n" + string.join(
                         traceback.format_exception(*sys.exc_info())))
                     sslconn.close()
                     self.running = False
+                    # normally, we'd just stop here, but for the test
+                    # harness, we want to stop the server
+                    self.server.stop()
                 except:
-                    sys.stderr.write(string.join(
+                    sys.stdout.write(string.join(
                         traceback.format_exception(*sys.exc_info())))
 
     def __init__(self, port, certificate, ssl_version=None,
@@ -192,21 +235,21 @@
         while self.active:
             try:
                 newconn, connaddr = self.sock.accept()
-                # sys.stderr.write('new connection from ' + str(connaddr))
+                sys.stdout.write('\nserver:  new connection from ' + str(connaddr) + '\n')
                 handler = self.connectionHandler(self, newconn)
                 handler.start()
             except socket.timeout:
                 pass
             except KeyboardInterrupt:
-                self.active = False
+                self.stop()
             except:
-                sys.stderr.write(string.join(
+                sys.stdout.write("Test server failure:\n" + string.join(
                     traceback.format_exception(*sys.exc_info())))
 
     def stop (self):
         self.active = False
+        self.sock.close()
 
-
 CERTFILE_CONFIG_TEMPLATE = """
 # create RSA certs - Server
 

From guido at python.org  Sun Aug 26 21:35:26 2007
From: guido at python.org (Guido van Rossum)
Date: Sun, 26 Aug 2007 12:35:26 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <5531275898429170325@unknownmsgid>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<5531275898429170325@unknownmsgid>
Message-ID: <ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>

No need, I've submitted this for you. Fingers crossed.

On 8/26/07, Bill Janssen <janssen at parc.com> wrote:
> > But I think this exposes a more generic bug in test_ssl.py, which is
> > that the server thread doesn't die when one of these failures occurs.
> > It probably should.  I'll make a patch -- but I don't have a system
> > that this fails on, how will I test it?
>
> Here's a patch which makes test_ssl a better player in the buildbots
> environment.  I deep-ended on "try-except-else" clauses.
>
> Should I post this as an issue to the Tracker?
>
> Bill
>
> Index: Lib/ssl.py
> ===================================================================
> --- Lib/ssl.py  (revision 57506)
> +++ Lib/ssl.py  (working copy)
> @@ -100,12 +100,13 @@
>              # see if it's connected
>              try:
>                  socket.getpeername(self)
> -                # yes
> +            except:
> +                # no, no connection yet
> +                self._sslobj = None
> +            else:
> +                # yes, create the SSL object
>                  self._sslobj = _ssl.sslwrap(self._sock, 0, keyfile, certfile,
>                                              cert_reqs, ssl_version, ca_certs)
> -            except:
> -                # no
> -                self._sslobj = None
>          self.keyfile = keyfile
>          self.certfile = certfile
>          self.cert_reqs = cert_reqs
> Index: Lib/test/test_ssl.py
> ===================================================================
> --- Lib/test/test_ssl.py        (revision 57506)
> +++ Lib/test/test_ssl.py        (working copy)
> @@ -91,38 +91,66 @@
>      def testTLSecho (self):
>
>          s1 = socket.socket()
> -        s1.connect(('127.0.0.1', 10024))
> -        c1 = ssl.sslsocket(s1, ssl_version=ssl.PROTOCOL_TLSv1)
> -        indata = "FOO\n"
> -        c1.write(indata)
> -        outdata = c1.read()
> -        if outdata != indata.lower():
> -            sys.stderr.write("bad data <<%s>> received\n" % data)
> -        c1.close()
> +        try:
> +            s1.connect(('127.0.0.1', 10024))
> +        except:
> +            sys.stdout.write("connection failure:\n" + string.join(
> +                traceback.format_exception(*sys.exc_info())))
> +            raise test_support.TestFailed("Can't connect to test server")
> +        else:
> +            try:
> +                c1 = ssl.sslsocket(s1, ssl_version=ssl.PROTOCOL_TLSv1)
> +            except:
> +                sys.stdout.write("SSL handshake failure:\n" + string.join(
> +                    traceback.format_exception(*sys.exc_info())))
> +                raise test_support.TestFailed("Can't SSL-handshake with test server")
> +            else:
> +                if not c1:
> +                    raise test_support.TestFailed("Can't SSL-handshake with test server")
> +                indata = "FOO\n"
> +                c1.write(indata)
> +                outdata = c1.read()
> +                if outdata != indata.lower():
> +                    raise test_support.TestFailed("bad data <<%s>> received; expected <<%s>>\n" % (data, indata.lower()))
> +                c1.close()
>
>      def testReadCert(self):
>
>          s2 = socket.socket()
> -        s2.connect(('127.0.0.1', 10024))
> -        c2 = ssl.sslsocket(s2, ssl_version=ssl.PROTOCOL_TLSv1,
> -                           cert_reqs=ssl.CERT_REQUIRED, ca_certs=CERTFILE)
> -        cert = c2.getpeercert()
> -        if not cert:
> -            raise test_support.TestFailed("Can't get peer certificate.")
> -        if not cert.has_key('subject'):
> -            raise test_support.TestFailed(
> -                "No subject field in certificate: %s." %
> -                pprint.pformat(cert))
> -        if not (cert['subject'].has_key('organizationName')):
> -            raise test_support.TestFailed(
> -                "No 'organizationName' field in certificate subject: %s." %
> -                pprint.pformat(cert))
> -        if (cert['subject']['organizationName'] !=
> -              "Python Software Foundation"):
> -            raise test_support.TestFailed(
> -                "Invalid 'organizationName' field in certificate subject; "
> -                "should be 'Python Software Foundation'.");
> -        c2.close()
> +        try:
> +            s2.connect(('127.0.0.1', 10024))
> +        except:
> +            sys.stdout.write("connection failure:\n" + string.join(
> +                traceback.format_exception(*sys.exc_info())))
> +            raise test_support.TestFailed("Can't connect to test server")
> +        else:
> +            try:
> +                c2 = ssl.sslsocket(s2, ssl_version=ssl.PROTOCOL_TLSv1,
> +                                   cert_reqs=ssl.CERT_REQUIRED, ca_certs=CERTFILE)
> +            except:
> +                sys.stdout.write("SSL handshake failure:\n" + string.join(
> +                    traceback.format_exception(*sys.exc_info())))
> +                raise test_support.TestFailed("Can't SSL-handshake with test server")
> +            else:
> +                if not c2:
> +                    raise test_support.TestFailed("Can't SSL-handshake with test server")
> +                cert = c2.getpeercert()
> +                if not cert:
> +                    raise test_support.TestFailed("Can't get peer certificate.")
> +                if not cert.has_key('subject'):
> +                    raise test_support.TestFailed(
> +                        "No subject field in certificate: %s." %
> +                        pprint.pformat(cert))
> +                if not (cert['subject'].has_key('organizationName')):
> +                    raise test_support.TestFailed(
> +                        "No 'organizationName' field in certificate subject: %s." %
> +                        pprint.pformat(cert))
> +                if (cert['subject']['organizationName'] !=
> +                      "Python Software Foundation"):
> +                    raise test_support.TestFailed(
> +                        "Invalid 'organizationName' field in certificate subject; "
> +                        "should be 'Python Software Foundation'.");
> +                c2.close()
>
>
>  class threadedEchoServer(threading.Thread):
> @@ -138,10 +166,22 @@
>
>          def run (self):
>              self.running = True
> -            sslconn = ssl.sslsocket(self.sock, server_side=True,
> -                                    certfile=self.server.certificate,
> -                                    ssl_version=self.server.protocol,
> -                                    cert_reqs=self.server.certreqs)
> +            try:
> +                sslconn = ssl.sslsocket(self.sock, server_side=True,
> +                                        certfile=self.server.certificate,
> +                                        ssl_version=self.server.protocol,
> +                                        cert_reqs=self.server.certreqs)
> +            except:
> +                # here, we want to stop the server, because this shouldn't
> +                # happen in the context of our test case
> +                sys.stdout.write("Test server failure:\n" + string.join(
> +                    traceback.format_exception(*sys.exc_info())))
> +                self.running = False
> +                # normally, we'd just stop here, but for the test
> +                # harness, we want to stop the server
> +                self.server.stop()
> +                return
> +
>              while self.running:
>                  try:
>                      msg = sslconn.read()
> @@ -154,15 +194,18 @@
>                          self.server.stop()
>                          self.running = False
>                      else:
> -                        # print "server:", msg.strip().lower()
> +                        sys.stdout.write("\nserver: %s\n" % msg.strip().lower())
>                          sslconn.write(msg.lower())
>                  except ssl.sslerror:
> -                    sys.stderr.write(string.join(
> +                    sys.stdout.write("Test server failure:\n" + string.join(
>                          traceback.format_exception(*sys.exc_info())))
>                      sslconn.close()
>                      self.running = False
> +                    # normally, we'd just stop here, but for the test
> +                    # harness, we want to stop the server
> +                    self.server.stop()
>                  except:
> -                    sys.stderr.write(string.join(
> +                    sys.stdout.write(string.join(
>                          traceback.format_exception(*sys.exc_info())))
>
>      def __init__(self, port, certificate, ssl_version=None,
> @@ -192,21 +235,21 @@
>          while self.active:
>              try:
>                  newconn, connaddr = self.sock.accept()
> -                # sys.stderr.write('new connection from ' + str(connaddr))
> +                sys.stdout.write('\nserver:  new connection from ' + str(connaddr) + '\n')
>                  handler = self.connectionHandler(self, newconn)
>                  handler.start()
>              except socket.timeout:
>                  pass
>              except KeyboardInterrupt:
> -                self.active = False
> +                self.stop()
>              except:
> -                sys.stderr.write(string.join(
> +                sys.stdout.write("Test server failure:\n" + string.join(
>                      traceback.format_exception(*sys.exc_info())))
>
>      def stop (self):
>          self.active = False
> +        self.sock.close()
>
> -
>  CERTFILE_CONFIG_TEMPLATE = """
>  # create RSA certs - Server
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From janssen at parc.com  Sun Aug 26 22:19:52 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 26 Aug 2007 13:19:52 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
Message-ID: <07Aug26.132001pdt."57996"@synergy1.parc.xerox.com>

> Yeah, I know this is difficult.  Hopefully someone with WIndows will
> step up to help.  We can at least make the test more robust and verify
> the files exist and are non-zero in size.  I will do that now.  At
> least the it shouldn't cause the test to time out.

Yes, the patch I sent out should fix that.

I'd like to know, for the machines where the test is failing, what's
in the generated cert file -- should be an RSA PRIVATE KEY and a
CERTIFICATE -- and what order they occur in.

Bill

From janssen at parc.com  Mon Aug 27 00:13:57 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 26 Aug 2007 15:13:57 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<5531275898429170325@unknownmsgid>
	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>
Message-ID: <07Aug26.151406pdt."57996"@synergy1.parc.xerox.com>

Well, as long as you have your ears on, here's another patch to test_ssl.

1)  Fixes the bug that two class names are initial-lower-case.

2)  Replaces the poll waiting for the server to become ready with
    a threading.Event signal.

Bill

Index: Lib/test/test_ssl.py
===================================================================
--- Lib/test/test_ssl.py	(revision 57521)
+++ Lib/test/test_ssl.py	(working copy)
@@ -153,9 +153,9 @@
                 c2.close()
 
 
-class threadedEchoServer(threading.Thread):
+class ThreadedEchoServer(threading.Thread):
 
-    class connectionHandler(threading.Thread):
+    class ConnectionHandler(threading.Thread):
 
         def __init__(self, server, connsock):
             self.server = server
@@ -219,6 +219,7 @@
         self.certreqs = certreqs
         self.cacerts = cacerts
         self.sock = socket.socket()
+        self.flag = None
         if hasattr(socket, 'SO_REUSEADDR'):
             self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
         if hasattr(socket, 'SO_REUSEPORT'):
@@ -228,15 +229,22 @@
         threading.Thread.__init__(self)
         self.setDaemon(False)
 
+    def start (self, flag=None):
+        self.flag = flag
+        threading.Thread.start(self)
+
     def run (self):
         self.sock.settimeout(0.5)
         self.sock.listen(5)
         self.active = True
+        if self.flag:
+            # signal an event
+            self.flag.set()
         while self.active:
             try:
                 newconn, connaddr = self.sock.accept()
                 #sys.stdout.write('\nserver:  new connection from ' + str(connaddr) + '\n')
-                handler = self.connectionHandler(self, newconn)
+                handler = self.ConnectionHandler(self, newconn)
                 handler.start()
             except socket.timeout:
                 pass
@@ -337,9 +345,11 @@
 
     server = None
     if test_support.is_resource_enabled('network'):
-        server = threadedEchoServer(10024, CERTFILE)
-        server.start()
-        time.sleep(1)
+        server = ThreadedEchoServer(10024, CERTFILE)
+        flag = threading.Event()
+        server.start(flag)
+        # wait for it to start
+        flag.wait()
         tests.append(ConnectedTests)
 
     thread_info = test_support.threading_setup()

From nnorwitz at gmail.com  Mon Aug 27 00:20:21 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sun, 26 Aug 2007 15:20:21 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <-4187035365711058012@unknownmsgid>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<5531275898429170325@unknownmsgid>
	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>
	<-4187035365711058012@unknownmsgid>
Message-ID: <ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com>

Thanks.  I applied it. -- n

On 8/26/07, Bill Janssen <janssen at parc.com> wrote:
> Well, as long as you have your ears on, here's another patch to test_ssl.
>
> 1)  Fixes the bug that two class names are initial-lower-case.
>
> 2)  Replaces the poll waiting for the server to become ready with
>     a threading.Event signal.
>
> Bill
>
> Index: Lib/test/test_ssl.py
> ===================================================================
> --- Lib/test/test_ssl.py        (revision 57521)
> +++ Lib/test/test_ssl.py        (working copy)
> @@ -153,9 +153,9 @@
>                  c2.close()
>
>
> -class threadedEchoServer(threading.Thread):
> +class ThreadedEchoServer(threading.Thread):
>
> -    class connectionHandler(threading.Thread):
> +    class ConnectionHandler(threading.Thread):
>
>          def __init__(self, server, connsock):
>              self.server = server
> @@ -219,6 +219,7 @@
>          self.certreqs = certreqs
>          self.cacerts = cacerts
>          self.sock = socket.socket()
> +        self.flag = None
>          if hasattr(socket, 'SO_REUSEADDR'):
>              self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>          if hasattr(socket, 'SO_REUSEPORT'):
> @@ -228,15 +229,22 @@
>          threading.Thread.__init__(self)
>          self.setDaemon(False)
>
> +    def start (self, flag=None):
> +        self.flag = flag
> +        threading.Thread.start(self)
> +
>      def run (self):
>          self.sock.settimeout(0.5)
>          self.sock.listen(5)
>          self.active = True
> +        if self.flag:
> +            # signal an event
> +            self.flag.set()
>          while self.active:
>              try:
>                  newconn, connaddr = self.sock.accept()
>                  #sys.stdout.write('\nserver:  new connection from ' + str(connaddr) + '\n')
> -                handler = self.connectionHandler(self, newconn)
> +                handler = self.ConnectionHandler(self, newconn)
>                  handler.start()
>              except socket.timeout:
>                  pass
> @@ -337,9 +345,11 @@
>
>      server = None
>      if test_support.is_resource_enabled('network'):
> -        server = threadedEchoServer(10024, CERTFILE)
> -        server.start()
> -        time.sleep(1)
> +        server = ThreadedEchoServer(10024, CERTFILE)
> +        flag = threading.Event()
> +        server.start(flag)
> +        # wait for it to start
> +        flag.wait()
>          tests.append(ConnectedTests)
>
>      thread_info = test_support.threading_setup()
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From janssen at parc.com  Mon Aug 27 00:47:02 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 26 Aug 2007 15:47:02 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
Message-ID: <07Aug26.154712pdt."57996"@synergy1.parc.xerox.com>

> Traceback (most recent call last):
>   File "/home/pybot/buildarea/trunk.klose-debian-ia64/build/Lib/threading.py",
> line 486, in __bootstrap_inner
>     self.run()
>   File "/home/pybot/buildarea/trunk.klose-debian-ia64/build/Lib/test/test_ssl.py",
> line 144, in run
>     cert_reqs=self.server.certreqs)
>   File "/home/pybot/buildarea/trunk.klose-debian-ia64/build/Lib/ssl.py",
> line 98, in __init__
>     cert_reqs, ssl_version, ca_certs)
> sslerror: _ssl.c:271: SSL_CTX_use_PrivateKey_file error

Neal, I'm looking at why we're getting such opaque error messages, and
I see that _ssl.c uses a static PyErrorObject that gets created when
the module is initialized:

	PySSLErrorObject = PyErr_NewException("socket.sslerror",
					      PySocketModule.error,
					      NULL);

Is this good style?  It should be thread-safe, as the GIL is held while
the error is being returned to the calling code, but still...

Some of the code sets the error string in this directly before
returning NULL, and other pieces of the code call PySSL_SetError,
which creates the error string.  I think some of the places which set
the string directly probably shouldn't; instead, they should call
PySSL_SetError to cons up the error name directly from the err code.
However, PySSL_SetError only works after the construction of an ssl
object, which means it can't be used there...  I'll take a longer look
at it and see if there's a reasonable fix.

Bill

From janssen at parc.com  Mon Aug 27 02:01:23 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 26 Aug 2007 17:01:23 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<5531275898429170325@unknownmsgid>
	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>
	<-4187035365711058012@unknownmsgid>
	<ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com>
Message-ID: <07Aug26.170127pdt."57996"@synergy1.parc.xerox.com>

Now it looks as if both the Debian and Ubuntu failures are failing
because they can't create a certificate, just like the Windows test.
I'll go out on a limb here and guess that it's because "openssl" isn't
on the path of the user running the tests.

That would also account for the other stack traces, if the keyfile
or certfile didn't actually contain a key or a cert.

Bill

From janssen at parc.com  Mon Aug 27 02:40:12 2007
From: janssen at parc.com (Bill Janssen)
Date: Sun, 26 Aug 2007 17:40:12 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<5531275898429170325@unknownmsgid>
	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>
	<-4187035365711058012@unknownmsgid>
	<ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com>
Message-ID: <07Aug26.174022pdt."57996"@synergy1.parc.xerox.com>

Here's a patch that will turn the buildbots green, by not trying the
connected tests if the certificate can't be created.  It also shows
the created cert if run in verbose mode.

We still need a working os.system command for Windows.

Bill


Index: Lib/test/test_ssl.py
===================================================================
--- Lib/test/test_ssl.py	(revision 57534)
+++ Lib/test/test_ssl.py	(working copy)
@@ -194,7 +194,8 @@
                         self.server.stop()
                         self.running = False
                     else:
-                        #sys.stdout.write("\nserver: %s\n" % msg.strip().lower())
+                        if test_support.verbose:
+                            sys.stdout.write("\nserver: %s\n" % msg.strip().lower())
                         sslconn.write(msg.lower())
                 except ssl.sslerror:
                     sys.stdout.write("Test server failure:\n" + string.join(
@@ -243,7 +244,8 @@
         while self.active:
             try:
                 newconn, connaddr = self.sock.accept()
-                #sys.stdout.write('\nserver:  new connection from ' + str(connaddr) + '\n')
+                if test_support.verbose:
+                    sys.stdout.write('\nserver:  new connection from ' + str(connaddr) + '\n')
                 handler = self.ConnectionHandler(self, newconn)
                 handler.start()
             except socket.timeout:
@@ -322,19 +324,21 @@
         (conffile, crtfile, crtfile))
     # now we have a self-signed server cert in crtfile
     os.unlink(conffile)
-    if error or not os.path.exists(crtfile) or os.path.getsize(crtfile) == 0:
-        raise test_support.TestFailed(
-            "Unable to create certificate for test %d." % error)
-    return d, crtfile
+    if (error or
+        (not os.path.exists(crtfile)) or
+        (os.path.getsize(crtfile) == 0)):
+        if test_support.verbose:
+            sys.stdout.write("\nUnable to create certificate for test.  "
+                             "Error status %d.\n" % (error >> 8))
+        shutil.rmtree(d)
+        return None, None
+    else:
+        if test_support.verbose:
+            sys.stdout.write(open(crtfile, 'r').read() + '\n')
+        return d, crtfile
 
-    # XXX(nnorwitz): should this code be removed now?
-    #sf_certfile = os.path.join(d, "sourceforge-imap.pem")
-    #sf_cert = ssl.fetch_server_certificate('pop.gmail.com', 995)
-    #open(sf_certfile, 'w').write(sf_cert)
-    #return d, crtfile, sf_certfile
-    # sys.stderr.write(open(crtfile, 'r').read() + '\n')
 
-def test_main():
+def test_main(verbose=False):
     if skip_expected:
         raise test_support.TestSkipped("socket module has no ssl support")
 
@@ -344,13 +348,16 @@
     tests = [BasicTests]
 
     server = None
-    if test_support.is_resource_enabled('network'):
+    if CERTFILE and test_support.is_resource_enabled('network'):
         server = ThreadedEchoServer(10024, CERTFILE)
         flag = threading.Event()
         server.start(flag)
         # wait for it to start
         flag.wait()
         tests.append(ConnectedTests)
+    else:
+        if test_support.verbose:
+            sys.stdout.write("\nSkipping test_ssl ConnectedTests; couldn't create a certificate.\n")
 
     thread_info = test_support.threading_setup()
 
@@ -362,7 +369,8 @@
             # wait for it to stop
             server.join()
 
-    shutil.rmtree(tdir)
+    if tdir and os.path.isdir(tdir):
+        shutil.rmtree(tdir)
     test_support.threading_cleanup(*thread_info)
 
 if __name__ == "__main__":


From greg at krypto.org  Mon Aug 27 05:56:01 2007
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 26 Aug 2007 20:56:01 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <4484419364585880902@unknownmsgid>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<5531275898429170325@unknownmsgid>
	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>
	<-4187035365711058012@unknownmsgid>
	<ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com>
	<4484419364585880902@unknownmsgid>
Message-ID: <52dc1c820708262056o3ceb03ffr93d8ba2dd633f79e@mail.gmail.com>

apt-get install openssl  will fix that on those systems.  on windows you're
unlikely to ever have an openssl binary present and available to execute.

On 8/26/07, Bill Janssen <janssen at parc.com> wrote:
>
> Now it looks as if both the Debian and Ubuntu failures are failing
> because they can't create a certificate, just like the Windows test.
> I'll go out on a limb here and guess that it's because "openssl" isn't
> on the path of the user running the tests.
>
> That would also account for the other stack traces, if the keyfile
> or certfile didn't actually contain a key or a cert.
>
> Bill
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070826/24f07848/attachment.htm 

From hrvoje.niksic at avl.com  Mon Aug 27 12:12:19 2007
From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=)
Date: Mon, 27 Aug 2007 12:12:19 +0200
Subject: [Python-Dev] Segfault
In-Reply-To: <ee2a432c0708240901x369c9886odcc96856e5aaaa5f@mail.gmail.com>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
	<1187771718.11322.47.camel@localhost>
	<ee2a432c0708222132v6fbcf29bx7ea617ba6ea2887d@mail.gmail.com>
	<1187854696.11322.67.camel@localhost>
	<ee2a432c0708231157j65f5b7e7yd23c0430f9e1c5d3@mail.gmail.com>
	<1187942122.11322.71.camel@localhost>
	<ee2a432c0708240901x369c9886odcc96856e5aaaa5f@mail.gmail.com>
Message-ID: <1188209539.11322.97.camel@localhost>

On Fri, 2007-08-24 at 09:01 -0700, Neal Norwitz wrote:
> Right.  I looked at this with Jeffrey Yasskin and agree that a lock is
> needed for f_fp.  The fix is ugly because it's needed around all
> accesses to f_fp and there are a ton of them.  Some are with the GIL
> held and others when it isn't.

The way I see it, when the GIL is held, we're in the clear because
f->f_fp can only disappear with the GIL held.  The troublesome cases are
those ones where f_fp is accessed inside an "allow threads" section.

To fix this, it shouldn't be necessary to change every individual place
that accesses f_fp.  It is sufficient to protect the sections that
specifically allow threads, changing the semantics from "allow threads"
to "allow threads, locking out the ones operating on the same
fileobject".

For example, instead of Py_BEGIN_ALLOW_THREADS, the fileobject code
could use a macro such as:

#define PyFile_BEGIN_PROTECT(f)          \
  PyThread_acquire_lock(f->f_lock, 1);   \
  if (!f->f_fp)                          \
    ;                                    \
  else {

#define PyFile_END_PROTECT(f)            \
  }                                      \
  PyThread_release_lock(f->f_lock);

That change might be somewhat easier than changing each individual line
that accesses f->f_fp.

> I would be fine with the much simpler approach in the patch I sent
> (assuming the patch is finished).  It doesn't completely address the
> problem, but does make the race condition much, much smaller.  Part of
> the reason I'm ok with this is that in 3.0, all this code has already
> been removed.

That is a good point.  Does your patch make the race crash go away
entirely in the tests like the one the OP posted?



From fijall at gmail.com  Mon Aug 27 15:23:05 2007
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 27 Aug 2007 15:23:05 +0200
Subject: [Python-Dev] Segfault
In-Reply-To: <1188209539.11322.97.camel@localhost>
References: <693bc9ab0708201127x657aa101mb8696d6ddd52e753@mail.gmail.com>
	<1187684028.11322.18.camel@localhost>
	<ee2a432c0708210914s5b27f411i5a9f82d0112741ea@mail.gmail.com>
	<1187771718.11322.47.camel@localhost>
	<ee2a432c0708222132v6fbcf29bx7ea617ba6ea2887d@mail.gmail.com>
	<1187854696.11322.67.camel@localhost>
	<ee2a432c0708231157j65f5b7e7yd23c0430f9e1c5d3@mail.gmail.com>
	<1187942122.11322.71.camel@localhost>
	<ee2a432c0708240901x369c9886odcc96856e5aaaa5f@mail.gmail.com>
	<1188209539.11322.97.camel@localhost>
Message-ID: <693bc9ab0708270623ya77a943g256a0fa8357720ab@mail.gmail.com>

Honestly, the argument that this code is already gone in 3.0 is not very
valid. 2.x version would be probably used for many years.

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070827/549dde33/attachment.htm 

From greg at krypto.org  Mon Aug 27 18:09:18 2007
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 27 Aug 2007 09:09:18 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <-3623426425687752225@unknownmsgid>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<5531275898429170325@unknownmsgid>
	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>
	<-4187035365711058012@unknownmsgid>
	<ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com>
	<4484419364585880902@unknownmsgid>
	<52dc1c820708262056o3ceb03ffr93d8ba2dd633f79e@mail.gmail.com>
	<-3623426425687752225@unknownmsgid>
Message-ID: <52dc1c820708270909x588bbd72s3c7dd1dcd9f55f3f@mail.gmail.com>

nope, not on many package based distributions.  libssl0.9.8, libssl-dev and
openssl are all separate packages (with appropriate dependencies).
/usr/bin/openssl comes from the openssl package.

Regardless, building a fixed test certificate and checking it in sounds like
the better option.  Then the openssl command in the test code can be turned
into a comment describing how the test data was pregenerated.

On 8/27/07, Bill Janssen <janssen at parc.com> wrote:
>
> > apt-get install openssl  will fix that on those systems.  on windows
> you're
> > unlikely to ever have an openssl binary present and available to
> execute.
>
> Well, if you have OpenSSL in the first place, you'll have the binary,
> won't you?  But I agree it's unlikely to be on your path.  As for Ubuntu
> and Debian, I checked the packaging, and they both put the "openssl"
> binary
> in /usr/bin, so it's unlikely to be a path problem.
>
> We could just build a fixed certificate and check it in, as the
> test_socket_ssl
> test does.  That way we wouldn't have to futz with trying to run openssl.
>
> Bill
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070827/9e57d76d/attachment.htm 

From janssen at parc.com  Mon Aug 27 17:56:01 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 27 Aug 2007 08:56:01 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <52dc1c820708262056o3ceb03ffr93d8ba2dd633f79e@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<5531275898429170325@unknownmsgid>
	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>
	<-4187035365711058012@unknownmsgid>
	<ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com>
	<4484419364585880902@unknownmsgid>
	<52dc1c820708262056o3ceb03ffr93d8ba2dd633f79e@mail.gmail.com>
Message-ID: <07Aug27.085607pdt."57996"@synergy1.parc.xerox.com>

> apt-get install openssl  will fix that on those systems.  on windows you're
> unlikely to ever have an openssl binary present and available to execute.

Well, if you have OpenSSL in the first place, you'll have the binary,
won't you?  But I agree it's unlikely to be on your path.  As for Ubuntu
and Debian, I checked the packaging, and they both put the "openssl" binary
in /usr/bin, so it's unlikely to be a path problem.

We could just build a fixed certificate and check it in, as the test_socket_ssl
test does.  That way we wouldn't have to futz with trying to run openssl.

Bill

From janssen at parc.com  Mon Aug 27 18:59:10 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 27 Aug 2007 09:59:10 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <52dc1c820708270909x588bbd72s3c7dd1dcd9f55f3f@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<5531275898429170325@unknownmsgid>
	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>
	<-4187035365711058012@unknownmsgid>
	<ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com>
	<4484419364585880902@unknownmsgid>
	<52dc1c820708262056o3ceb03ffr93d8ba2dd633f79e@mail.gmail.com>
	<-3623426425687752225@unknownmsgid>
	<52dc1c820708270909x588bbd72s3c7dd1dcd9f55f3f@mail.gmail.com>
Message-ID: <07Aug27.095917pdt."57996"@synergy1.parc.xerox.com>

> Regardless, building a fixed test certificate and checking it in sounds like
> the better option.  Then the openssl command in the test code can be turned
> into a comment describing how the test data was pregenerated.

Here's a patch that does that.

Bill

Index: Lib/test/keycert.pem
===================================================================
--- Lib/test/keycert.pem	(revision 0)
+++ Lib/test/keycert.pem	(revision 0)
@@ -0,0 +1,32 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIICXwIBAAKBgQC8ddrhm+LutBvjYcQlnH21PPIseJ1JVG2HMmN2CmZk2YukO+9L
+opdJhTvbGfEj0DQs1IE8M+kTUyOmuKfVrFMKwtVeCJphrAnhoz7TYOuLBSqt7lVH
+fhi/VwovESJlaBOp+WMnfhcduPEYHYx/6cnVapIkZnLt30zu2um+DzA9jQIDAQAB
+AoGBAK0FZpaKj6WnJZN0RqhhK+ggtBWwBnc0U/ozgKz2j1s3fsShYeiGtW6CK5nU
+D1dZ5wzhbGThI7LiOXDvRucc9n7vUgi0alqPQ/PFodPxAN/eEYkmXQ7W2k7zwsDA
+IUK0KUhktQbLu8qF/m8qM86ba9y9/9YkXuQbZ3COl5ahTZrhAkEA301P08RKv3KM
+oXnGU2UHTuJ1MAD2hOrPxjD4/wxA/39EWG9bZczbJyggB4RHu0I3NOSFjAm3HQm0
+ANOu5QK9owJBANgOeLfNNcF4pp+UikRFqxk5hULqRAWzVxVrWe85FlPm0VVmHbb/
+loif7mqjU8o1jTd/LM7RD9f2usZyE2psaw8CQQCNLhkpX3KO5kKJmS9N7JMZSc4j
+oog58yeYO8BBqKKzpug0LXuQultYv2K4veaIO04iL9VLe5z9S/Q1jaCHBBuXAkEA
+z8gjGoi1AOp6PBBLZNsncCvcV/0aC+1se4HxTNo2+duKSDnbq+ljqOM+E7odU+Nq
+ewvIWOG//e8fssd0mq3HywJBAJ8l/c8GVmrpFTx8r/nZ2Pyyjt3dH1widooDXYSV
+q6Gbf41Llo5sYAtmxdndTLASuHKecacTgZVhy0FryZpLKrU=
+-----END RSA PRIVATE KEY-----
+-----BEGIN CERTIFICATE-----
+MIICpzCCAhCgAwIBAgIJAP+qStv1cIGNMA0GCSqGSIb3DQEBBQUAMIGJMQswCQYD
+VQQGEwJVUzERMA8GA1UECBMIRGVsYXdhcmUxEzARBgNVBAcTCldpbG1pbmd0b24x
+IzAhBgNVBAoTGlB5dGhvbiBTb2Z0d2FyZSBGb3VuZGF0aW9uMQwwCgYDVQQLEwNT
+U0wxHzAdBgNVBAMTFnNvbWVtYWNoaW5lLnB5dGhvbi5vcmcwHhcNMDcwODI3MTY1
+NDUwWhcNMTMwMjE2MTY1NDUwWjCBiTELMAkGA1UEBhMCVVMxETAPBgNVBAgTCERl
+bGF3YXJlMRMwEQYDVQQHEwpXaWxtaW5ndG9uMSMwIQYDVQQKExpQeXRob24gU29m
+dHdhcmUgRm91bmRhdGlvbjEMMAoGA1UECxMDU1NMMR8wHQYDVQQDExZzb21lbWFj
+aGluZS5weXRob24ub3JnMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC8ddrh
+m+LutBvjYcQlnH21PPIseJ1JVG2HMmN2CmZk2YukO+9LopdJhTvbGfEj0DQs1IE8
+M+kTUyOmuKfVrFMKwtVeCJphrAnhoz7TYOuLBSqt7lVHfhi/VwovESJlaBOp+WMn
+fhcduPEYHYx/6cnVapIkZnLt30zu2um+DzA9jQIDAQABoxUwEzARBglghkgBhvhC
+AQEEBAMCBkAwDQYJKoZIhvcNAQEFBQADgYEAF4Q5BVqmCOLv1n8je/Jw9K669VXb
+08hyGzQhkemEBYQd6fzQ9A/1ZzHkJKb1P6yreOLSEh4KcxYPyrLRC1ll8nr5OlCx
+CMhKkTnR6qBsdNV0XtdU2+N25hqW+Ma4ZeqsN/iiJVCGNOZGnvQuvCAGWF8+J/f/
+iHkC6gGdBJhogs4=
+-----END CERTIFICATE-----
Index: Lib/test/test_ssl.py
===================================================================
--- Lib/test/test_ssl.py	(revision 57559)
+++ Lib/test/test_ssl.py	(working copy)
@@ -22,7 +22,6 @@
     skip_expected = True
 
 CERTFILE = None
-GMAIL_POP_CERTFILE = None
 
 
 def handle_error(prefix):
@@ -298,12 +297,15 @@
 nsCertType = server
 """
 
-def create_cert_files():
+def create_cert_files(hostname=None):
 
+    """This is the routine that was run to create the certificate
+    and private key contained in keycert.pem."""
+
     import tempfile, socket, os
     d = tempfile.mkdtemp()
     # now create a configuration file for the CA signing cert
-    fqdn = socket.getfqdn()
+    fqdn = hostname or socket.getfqdn()
     crtfile = os.path.join(d, "cert.pem")
     conffile = os.path.join(d, "ca.conf")
     fp = open(conffile, "w")
@@ -316,7 +318,7 @@
               })
     fp.close()
     error = os.system(
-        "openssl req -batch -new -x509 -days 10 -nodes -config %s "
+        "openssl req -batch -new -x509 -days 2000 -nodes -config %s "
         "-keyout \"%s\" -out \"%s\" > /dev/null < /dev/null 2>&1" %
         (conffile, crtfile, crtfile))
     # now we have a self-signed server cert in crtfile
@@ -324,7 +326,8 @@
     if (os.WEXITSTATUS(error) or
         not os.path.exists(crtfile) or os.path.getsize(crtfile) == 0):
         if test_support.verbose:
-            sys.stdout.write("Unable to create certificate for test %d\n" % error)
+            sys.stdout.write("Unable to create certificate for test, "
+                             + "error status %d\n" % (error >> 8))
         crtfile = None
     elif test_support.verbose:
         sys.stdout.write(open(crtfile, 'r').read() + '\n')
@@ -336,7 +339,8 @@
         raise test_support.TestSkipped("socket module has no ssl support")
 
     global CERTFILE
-    tdir, CERTFILE = create_cert_files()
+    CERTFILE = os.path.join(os.path.dirname(__file__) or os.curdir,
+                            "keycert.pem")
     if not CERTFILE:
         sys.__stdout__.write("Skipping test_ssl ConnectedTests; "
                              "couldn't create a certificate.\n")
@@ -362,8 +366,6 @@
             # wait for it to stop
             server.join()
 
-    if tdir and os.path.isdir(tdir):
-        shutil.rmtree(tdir)
     test_support.threading_cleanup(*thread_info)
 
 if __name__ == "__main__":

From guido at python.org  Mon Aug 27 19:20:17 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 27 Aug 2007 10:20:17 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <1390075235631527364@unknownmsgid>
References: <166458247748783241@unknownmsgid>
	<5531275898429170325@unknownmsgid>
	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>
	<-4187035365711058012@unknownmsgid>
	<ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com>
	<4484419364585880902@unknownmsgid>
	<52dc1c820708262056o3ceb03ffr93d8ba2dd633f79e@mail.gmail.com>
	<-3623426425687752225@unknownmsgid>
	<52dc1c820708270909x588bbd72s3c7dd1dcd9f55f3f@mail.gmail.com>
	<1390075235631527364@unknownmsgid>
Message-ID: <ca471dc20708271020vebba77fxf03c31ff7868097a@mail.gmail.com>

Committed revision 57561.


On 8/27/07, Bill Janssen <janssen at parc.com> wrote:
> > Regardless, building a fixed test certificate and checking it in sounds like
> > the better option.  Then the openssl command in the test code can be turned
> > into a comment describing how the test data was pregenerated.
>
> Here's a patch that does that.
>
> Bill
>
> Index: Lib/test/keycert.pem
> ===================================================================
> --- Lib/test/keycert.pem        (revision 0)
> +++ Lib/test/keycert.pem        (revision 0)
> @@ -0,0 +1,32 @@
> +-----BEGIN RSA PRIVATE KEY-----
> +MIICXwIBAAKBgQC8ddrhm+LutBvjYcQlnH21PPIseJ1JVG2HMmN2CmZk2YukO+9L
> +opdJhTvbGfEj0DQs1IE8M+kTUyOmuKfVrFMKwtVeCJphrAnhoz7TYOuLBSqt7lVH
> +fhi/VwovESJlaBOp+WMnfhcduPEYHYx/6cnVapIkZnLt30zu2um+DzA9jQIDAQAB
> +AoGBAK0FZpaKj6WnJZN0RqhhK+ggtBWwBnc0U/ozgKz2j1s3fsShYeiGtW6CK5nU
> +D1dZ5wzhbGThI7LiOXDvRucc9n7vUgi0alqPQ/PFodPxAN/eEYkmXQ7W2k7zwsDA
> +IUK0KUhktQbLu8qF/m8qM86ba9y9/9YkXuQbZ3COl5ahTZrhAkEA301P08RKv3KM
> +oXnGU2UHTuJ1MAD2hOrPxjD4/wxA/39EWG9bZczbJyggB4RHu0I3NOSFjAm3HQm0
> +ANOu5QK9owJBANgOeLfNNcF4pp+UikRFqxk5hULqRAWzVxVrWe85FlPm0VVmHbb/
> +loif7mqjU8o1jTd/LM7RD9f2usZyE2psaw8CQQCNLhkpX3KO5kKJmS9N7JMZSc4j
> +oog58yeYO8BBqKKzpug0LXuQultYv2K4veaIO04iL9VLe5z9S/Q1jaCHBBuXAkEA
> +z8gjGoi1AOp6PBBLZNsncCvcV/0aC+1se4HxTNo2+duKSDnbq+ljqOM+E7odU+Nq
> +ewvIWOG//e8fssd0mq3HywJBAJ8l/c8GVmrpFTx8r/nZ2Pyyjt3dH1widooDXYSV
> +q6Gbf41Llo5sYAtmxdndTLASuHKecacTgZVhy0FryZpLKrU=
> +-----END RSA PRIVATE KEY-----
> +-----BEGIN CERTIFICATE-----
> +MIICpzCCAhCgAwIBAgIJAP+qStv1cIGNMA0GCSqGSIb3DQEBBQUAMIGJMQswCQYD
> +VQQGEwJVUzERMA8GA1UECBMIRGVsYXdhcmUxEzARBgNVBAcTCldpbG1pbmd0b24x
> +IzAhBgNVBAoTGlB5dGhvbiBTb2Z0d2FyZSBGb3VuZGF0aW9uMQwwCgYDVQQLEwNT
> +U0wxHzAdBgNVBAMTFnNvbWVtYWNoaW5lLnB5dGhvbi5vcmcwHhcNMDcwODI3MTY1
> +NDUwWhcNMTMwMjE2MTY1NDUwWjCBiTELMAkGA1UEBhMCVVMxETAPBgNVBAgTCERl
> +bGF3YXJlMRMwEQYDVQQHEwpXaWxtaW5ndG9uMSMwIQYDVQQKExpQeXRob24gU29m
> +dHdhcmUgRm91bmRhdGlvbjEMMAoGA1UECxMDU1NMMR8wHQYDVQQDExZzb21lbWFj
> +aGluZS5weXRob24ub3JnMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC8ddrh
> +m+LutBvjYcQlnH21PPIseJ1JVG2HMmN2CmZk2YukO+9LopdJhTvbGfEj0DQs1IE8
> +M+kTUyOmuKfVrFMKwtVeCJphrAnhoz7TYOuLBSqt7lVHfhi/VwovESJlaBOp+WMn
> +fhcduPEYHYx/6cnVapIkZnLt30zu2um+DzA9jQIDAQABoxUwEzARBglghkgBhvhC
> +AQEEBAMCBkAwDQYJKoZIhvcNAQEFBQADgYEAF4Q5BVqmCOLv1n8je/Jw9K669VXb
> +08hyGzQhkemEBYQd6fzQ9A/1ZzHkJKb1P6yreOLSEh4KcxYPyrLRC1ll8nr5OlCx
> +CMhKkTnR6qBsdNV0XtdU2+N25hqW+Ma4ZeqsN/iiJVCGNOZGnvQuvCAGWF8+J/f/
> +iHkC6gGdBJhogs4=
> +-----END CERTIFICATE-----
> Index: Lib/test/test_ssl.py
> ===================================================================
> --- Lib/test/test_ssl.py        (revision 57559)
> +++ Lib/test/test_ssl.py        (working copy)
> @@ -22,7 +22,6 @@
>      skip_expected = True
>
>  CERTFILE = None
> -GMAIL_POP_CERTFILE = None
>
>
>  def handle_error(prefix):
> @@ -298,12 +297,15 @@
>  nsCertType = server
>  """
>
> -def create_cert_files():
> +def create_cert_files(hostname=None):
>
> +    """This is the routine that was run to create the certificate
> +    and private key contained in keycert.pem."""
> +
>      import tempfile, socket, os
>      d = tempfile.mkdtemp()
>      # now create a configuration file for the CA signing cert
> -    fqdn = socket.getfqdn()
> +    fqdn = hostname or socket.getfqdn()
>      crtfile = os.path.join(d, "cert.pem")
>      conffile = os.path.join(d, "ca.conf")
>      fp = open(conffile, "w")
> @@ -316,7 +318,7 @@
>                })
>      fp.close()
>      error = os.system(
> -        "openssl req -batch -new -x509 -days 10 -nodes -config %s "
> +        "openssl req -batch -new -x509 -days 2000 -nodes -config %s "
>          "-keyout \"%s\" -out \"%s\" > /dev/null < /dev/null 2>&1" %
>          (conffile, crtfile, crtfile))
>      # now we have a self-signed server cert in crtfile
> @@ -324,7 +326,8 @@
>      if (os.WEXITSTATUS(error) or
>          not os.path.exists(crtfile) or os.path.getsize(crtfile) == 0):
>          if test_support.verbose:
> -            sys.stdout.write("Unable to create certificate for test %d\n" % error)
> +            sys.stdout.write("Unable to create certificate for test, "
> +                             + "error status %d\n" % (error >> 8))
>          crtfile = None
>      elif test_support.verbose:
>          sys.stdout.write(open(crtfile, 'r').read() + '\n')
> @@ -336,7 +339,8 @@
>          raise test_support.TestSkipped("socket module has no ssl support")
>
>      global CERTFILE
> -    tdir, CERTFILE = create_cert_files()
> +    CERTFILE = os.path.join(os.path.dirname(__file__) or os.curdir,
> +                            "keycert.pem")
>      if not CERTFILE:
>          sys.__stdout__.write("Skipping test_ssl ConnectedTests; "
>                               "couldn't create a certificate.\n")
> @@ -362,8 +366,6 @@
>              # wait for it to stop
>              server.join()
>
> -    if tdir and os.path.isdir(tdir):
> -        shutil.rmtree(tdir)
>      test_support.threading_cleanup(*thread_info)
>
>  if __name__ == "__main__":
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From jmatejek at suse.cz  Mon Aug 27 19:35:32 2007
From: jmatejek at suse.cz (Jan Matejek)
Date: Mon, 27 Aug 2007 19:35:32 +0200
Subject: [Python-Dev] tarfile and directory traversal vulnerability
In-Reply-To: <46CF5DD6.9080701@v.loewis.de>
References: <46CF1729.5020308@suse.cz> <46CF5DD6.9080701@v.loewis.de>
Message-ID: <46D30B64.3060903@suse.cz>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin v. L?wis wrote:
> I must admit I fail to see the bug. If root untars a file, and that tar
> file contains an instruction to overwrite /etc/passwd, why is an error
> to execute that instruction? Shouldn't root just be more careful when
> untaring files?

GNU tar is not supposed to place files outside its working directory,
unless explicitly specified otherwise. So this is considered a security
vulnerability.

AFAIK there is no specified behavior and other tars might act
differently. But i think GNU tar behaves correctly in this regard.

Furthermore, extract() and extractall() documentation says "Extract
(...) from the archive to the *current working directory* or directory
[path]."
So current behavior is actually inconsistent with the documentation.

>> if tarinfo.name.startswith('../'):
>>     self.extract(tarinfo, path)
>> else:
>>     warnings.warn("non-local file skipped: %s" % tarinfo.name,
>> RuntimeWarning, stacklevel=1)
> 
> Ok. You seem to be claiming that the tarfile is incorrect in some
> sense. Can you please point to some spec that says this is an incorrect
> tarfile?

No, the tar file itself is correct, according to POSIX. You can put
anything into a tar. Point is, you should be able to untar any file
'safely'.

> In any case, if you fix what you consider broken, you should do
> it exactly the same way as GNU tar does it (assuming you consider
> GNU tar fixed).

I can do that.
I would propose an optional parameter for extract() and extractall(),
absolutePaths, defaulting to False. When encountering a non-local file,
it would strip the leading slash or the path up to the last '../'
sequence (that is what GNU tar does) and extract the file locally.
Setting absolutePaths to True would restore current behavior (no checks).

regards,
jan matejek
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4-svn0 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFG0wtkjBrWA+AvBr8RAmmnAKCtpYYoFZYaNwba2WW11NtRuCyqhwCePkFw
9M2pKHtu0O62fAYfb8NTm3A=
=yfVK
-----END PGP SIGNATURE-----

From jmatejek at suse.cz  Mon Aug 27 19:40:36 2007
From: jmatejek at suse.cz (Jan Matejek)
Date: Mon, 27 Aug 2007 19:40:36 +0200
Subject: [Python-Dev] tarfile and directory traversal vulnerability
In-Reply-To: <20070825101312.GA31481@core.g33x.de>
References: <46CF1729.5020308@suse.cz> <20070825101312.GA31481@core.g33x.de>
Message-ID: <46D30C94.8020204@suse.cz>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Lars Gust?bel wrote:
> Suppose we have:
> foo -> /etc
> foo/passwd
> 
> If creation of the foo symlink is delayed, foo/passwd will be
> extracted in a directory foo which will be created implicitly.
> If we create the foo symlink afterwards it will fail because foo
> already exists. The best way would be to completely ignore
> members and link targets that are absolute or outside the
> archive's scope.

GNU tar doesn't descend into symlinked directories when extracting, such
archive fails anyway:

# tar xvf foo.tar
foo
foo/passwd
tar: foo/passwd: Cannot open: Not a directory
tar: Error exit delayed from previous errors

I think that is the simplest solution, but i'm not sure how to best
implement that in extractall().
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4-svn0 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFG0wyUjBrWA+AvBr8RAjkJAKCJS+hkV1HYL9egOsyeTE5vj44r5ACeNmt7
HquYw+ON+5qVNoC778OtQRE=
=9Kx/
-----END PGP SIGNATURE-----

From janssen at parc.com  Mon Aug 27 20:09:41 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 27 Aug 2007 11:09:41 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <07Aug26.154712pdt."57996"@synergy1.parc.xerox.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<07Aug26.154712pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug27.110942pdt."57996"@synergy1.parc.xerox.com>

> Some of the code sets the error string in this directly before
> returning NULL, and other pieces of the code call PySSL_SetError,
> which creates the error string.  I think some of the places which set
> the string directly probably shouldn't; instead, they should call
> PySSL_SetError to cons up the error name directly from the err code.
> However, PySSL_SetError only works after the construction of an ssl
> object, which means it can't be used there...  I'll take a longer look
> at it and see if there's a reasonable fix.

Here's a patch which addresses this.  It also fixes the indentation in
PySSL_SetError, bringing it into line with PEP 7, fixes a compile warning
about one of the OpenSSL macros, and makes the namespace a bit more
consistent.  I've tested it on FC 7 and OS X 10.4.

% ./python ./Lib/test/regrtest.py -R :1: -u all test_ssl
test_ssl
beginning 6 repetitions
123456
......
1 test OK.
[29244 refs]
%

Bill


Index: Modules/_ssl.c
===================================================================
--- Modules/_ssl.c	(revision 57564)
+++ Modules/_ssl.c	(working copy)
@@ -122,71 +122,74 @@
 	char buf[2048];
 	char *errstr;
 	int err;
-	enum py_ssl_error p;
+	enum py_ssl_error p = PY_SSL_ERROR_NONE;
 
 	assert(ret <= 0);
 
-	err = SSL_get_error(obj->ssl, ret);
+	if ((obj != NULL) && (obj->ssl != NULL)) {
+		err = SSL_get_error(obj->ssl, ret);
 
-	switch (err) {
-	case SSL_ERROR_ZERO_RETURN:
-		errstr = "TLS/SSL connection has been closed";
-		p = PY_SSL_ERROR_ZERO_RETURN;
-		break;
-	case SSL_ERROR_WANT_READ:
-		errstr = "The operation did not complete (read)";
-		p = PY_SSL_ERROR_WANT_READ;
-		break;
-	case SSL_ERROR_WANT_WRITE:
-		p = PY_SSL_ERROR_WANT_WRITE;
-		errstr = "The operation did not complete (write)";
-		break;
-	case SSL_ERROR_WANT_X509_LOOKUP:
-		p = PY_SSL_ERROR_WANT_X509_LOOKUP;
-		errstr = "The operation did not complete (X509 lookup)";
-		break;
-	case SSL_ERROR_WANT_CONNECT:
-		p = PY_SSL_ERROR_WANT_CONNECT;
-		errstr = "The operation did not complete (connect)";
-		break;
-	case SSL_ERROR_SYSCALL:
-	{
-		unsigned long e = ERR_get_error();
-		if (e == 0) {
-			if (ret == 0 || !obj->Socket) {
-				p = PY_SSL_ERROR_EOF;
-				errstr = "EOF occurred in violation of protocol";
-			} else if (ret == -1) {
-				/* the underlying BIO reported an I/O error */
-				return obj->Socket->errorhandler();
-			} else {  /* possible? */
+		switch (err) {
+		case SSL_ERROR_ZERO_RETURN:
+			errstr = "TLS/SSL connection has been closed";
+			p = PY_SSL_ERROR_ZERO_RETURN;
+			break;
+		case SSL_ERROR_WANT_READ:
+			errstr = "The operation did not complete (read)";
+			p = PY_SSL_ERROR_WANT_READ;
+			break;
+		case SSL_ERROR_WANT_WRITE:
+			p = PY_SSL_ERROR_WANT_WRITE;
+			errstr = "The operation did not complete (write)";
+			break;
+		case SSL_ERROR_WANT_X509_LOOKUP:
+			p = PY_SSL_ERROR_WANT_X509_LOOKUP;
+			errstr = "The operation did not complete (X509 lookup)";
+			break;
+		case SSL_ERROR_WANT_CONNECT:
+			p = PY_SSL_ERROR_WANT_CONNECT;
+			errstr = "The operation did not complete (connect)";
+			break;
+		case SSL_ERROR_SYSCALL:
+		{
+			unsigned long e = ERR_get_error();
+			if (e == 0) {
+				if (ret == 0 || !obj->Socket) {
+					p = PY_SSL_ERROR_EOF;
+					errstr = "EOF occurred in violation of protocol";
+				} else if (ret == -1) {
+					/* the underlying BIO reported an I/O error */
+					return obj->Socket->errorhandler();
+				} else { /* possible? */
+					p = PY_SSL_ERROR_SYSCALL;
+					errstr = "Some I/O error occurred";
+				}
+			} else {
 				p = PY_SSL_ERROR_SYSCALL;
-				errstr = "Some I/O error occurred";
+				/* XXX Protected by global interpreter lock */
+				errstr = ERR_error_string(e, NULL);
 			}
-		} else {
-			p = PY_SSL_ERROR_SYSCALL;
-			/* XXX Protected by global interpreter lock */
-			errstr = ERR_error_string(e, NULL);
+			break;
 		}
-		break;
-	}
-	case SSL_ERROR_SSL:
-	{
-		unsigned long e = ERR_get_error();
-		p = PY_SSL_ERROR_SSL;
-		if (e != 0)
-			/* XXX Protected by global interpreter lock */
-			errstr = ERR_error_string(e, NULL);
-		else { /* possible? */
-			errstr = "A failure in the SSL library occurred";
+		case SSL_ERROR_SSL:
+		{
+			unsigned long e = ERR_get_error();
+			p = PY_SSL_ERROR_SSL;
+			if (e != 0)
+				/* XXX Protected by global interpreter lock */
+				errstr = ERR_error_string(e, NULL);
+			else {	/* possible? */
+				errstr = "A failure in the SSL library occurred";
+			}
+			break;
 		}
-		break;
+		default:
+			p = PY_SSL_ERROR_INVALID_ERROR_CODE;
+			errstr = "Invalid error code";
+		}
+	} else {
+		errstr = ERR_error_string(ERR_peek_last_error(), NULL);
 	}
-	default:
-		p = PY_SSL_ERROR_INVALID_ERROR_CODE;
-		errstr = "Invalid error code";
-	}
-
 	PyOS_snprintf(buf, sizeof(buf), "_ssl.c:%d: %s", lineno, errstr);
 	v = Py_BuildValue("(is)", p, buf);
 	if (v != NULL) {
@@ -256,8 +259,8 @@
 			ret = SSL_CTX_load_verify_locations(self->ctx,
 							    cacerts_file, NULL);
 			Py_END_ALLOW_THREADS
-			if (ret < 1) {
-				errstr = ERRSTR("SSL_CTX_load_verify_locations");
+			if (ret != 1) {
+				PySSL_SetError(NULL, 0, __FILE__, __LINE__);
 				goto fail;
 			}
 		}
@@ -267,8 +270,8 @@
 		ret = SSL_CTX_use_PrivateKey_file(self->ctx, key_file,
 						  SSL_FILETYPE_PEM);
 		Py_END_ALLOW_THREADS
-		if (ret < 1) {
-			errstr = ERRSTR("SSL_CTX_use_PrivateKey_file error");
+		if (ret != 1) {
+			PySSL_SetError(NULL, 0, __FILE__, __LINE__);
 			goto fail;
 		}
 
@@ -276,8 +279,8 @@
 		ret = SSL_CTX_use_certificate_chain_file(self->ctx,
 						       cert_file);
 		Py_END_ALLOW_THREADS
-		if (ret < 1) {
-			errstr = ERRSTR("SSL_CTX_use_certificate_chain_file error") ;
+		if (ret != 1) {
+			PySSL_SetError(NULL, 0, __FILE__, __LINE__);
 			goto fail;
 		}
 		SSL_CTX_set_options(self->ctx, SSL_OP_ALL); /* ssl compatibility */
@@ -375,7 +378,7 @@
 }
 
 static PyObject *
-PySocket_ssl(PyObject *self, PyObject *args)
+PySSL_sslwrap(PyObject *self, PyObject *args)
 {
 	PySocketSockObject *Sock;
 	int server_side = 0;
@@ -431,6 +434,9 @@
 	PyObject *pd = PyDict_New();
 	int index_counter;
 
+	if (pd == NULL)
+		return NULL;
+
 	for (index_counter = 0;
 	     index_counter < X509_NAME_entry_count(xname);
 	     index_counter++)
@@ -548,7 +554,7 @@
 	}
 	Py_DECREF(pnotBefore);
 
-	BIO_reset(biobuf);
+	(void) BIO_reset(biobuf);
 	notAfter = X509_get_notAfter(self->peer_cert);
 	ASN1_TIME_print(biobuf, notAfter);
 	len = BIO_gets(biobuf, buf, sizeof(buf)-1);
@@ -857,7 +863,7 @@
 
 static PyTypeObject PySSL_Type = {
 	PyVarObject_HEAD_INIT(NULL, 0)
-	"socket.SSL",			/*tp_name*/
+	"ssl.SSLContext",		/*tp_name*/
 	sizeof(PySSLObject),		/*tp_basicsize*/
 	0,				/*tp_itemsize*/
 	/* methods */
@@ -932,7 +938,7 @@
 "RAND_egd(path) -> bytes\n\
 \n\
 Queries the entropy gather daemon (EGD) on socket path.  Returns number\n\
-of bytes read.  Raises socket.sslerror if connection to EGD fails or\n\
+of bytes read.  Raises ssl.sslerror if connection to EGD fails or\n\
 if it does provide enough data to seed PRNG.");
 
 #endif
@@ -940,7 +946,7 @@
 /* List of functions exported by this module. */
 
 static PyMethodDef PySSL_methods[] = {
-	{"sslwrap",             PySocket_ssl,
+	{"sslwrap",             PySSL_sslwrap,
          METH_VARARGS, ssl_doc},
 #ifdef HAVE_OPENSSL_RAND
 	{"RAND_add",            PySSL_RAND_add, METH_VARARGS,
@@ -979,7 +985,7 @@
 	SSLeay_add_ssl_algorithms();
 
 	/* Add symbols to module dict */
-	PySSLErrorObject = PyErr_NewException("socket.sslerror",
+	PySSLErrorObject = PyErr_NewException("ssl.sslerror",
 					      PySocketModule.error,
 					      NULL);
 	if (PySSLErrorObject == NULL)


From guido at python.org  Mon Aug 27 20:42:35 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 27 Aug 2007 11:42:35 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <-6964906228903917740@unknownmsgid>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<-6964906228903917740@unknownmsgid>
Message-ID: <ca471dc20708271142j7ebf38dbt4d2b4273cf19cf09@mail.gmail.com>

Committed revision 57568.

Anything else I can check in for you?

On 8/27/07, Bill Janssen <janssen at parc.com> wrote:
> > Some of the code sets the error string in this directly before
> > returning NULL, and other pieces of the code call PySSL_SetError,
> > which creates the error string.  I think some of the places which set
> > the string directly probably shouldn't; instead, they should call
> > PySSL_SetError to cons up the error name directly from the err code.
> > However, PySSL_SetError only works after the construction of an ssl
> > object, which means it can't be used there...  I'll take a longer look
> > at it and see if there's a reasonable fix.
>
> Here's a patch which addresses this.  It also fixes the indentation in
> PySSL_SetError, bringing it into line with PEP 7, fixes a compile warning
> about one of the OpenSSL macros, and makes the namespace a bit more
> consistent.  I've tested it on FC 7 and OS X 10.4.
>
> % ./python ./Lib/test/regrtest.py -R :1: -u all test_ssl
> test_ssl
> beginning 6 repetitions
> 123456
> ......
> 1 test OK.
> [29244 refs]
> %
>
> Bill
>
>
> Index: Modules/_ssl.c
> ===================================================================
> --- Modules/_ssl.c      (revision 57564)
> +++ Modules/_ssl.c      (working copy)
> @@ -122,71 +122,74 @@
>         char buf[2048];
>         char *errstr;
>         int err;
> -       enum py_ssl_error p;
> +       enum py_ssl_error p = PY_SSL_ERROR_NONE;
>
>         assert(ret <= 0);
>
> -       err = SSL_get_error(obj->ssl, ret);
> +       if ((obj != NULL) && (obj->ssl != NULL)) {
> +               err = SSL_get_error(obj->ssl, ret);
>
> -       switch (err) {
> -       case SSL_ERROR_ZERO_RETURN:
> -               errstr = "TLS/SSL connection has been closed";
> -               p = PY_SSL_ERROR_ZERO_RETURN;
> -               break;
> -       case SSL_ERROR_WANT_READ:
> -               errstr = "The operation did not complete (read)";
> -               p = PY_SSL_ERROR_WANT_READ;
> -               break;
> -       case SSL_ERROR_WANT_WRITE:
> -               p = PY_SSL_ERROR_WANT_WRITE;
> -               errstr = "The operation did not complete (write)";
> -               break;
> -       case SSL_ERROR_WANT_X509_LOOKUP:
> -               p = PY_SSL_ERROR_WANT_X509_LOOKUP;
> -               errstr = "The operation did not complete (X509 lookup)";
> -               break;
> -       case SSL_ERROR_WANT_CONNECT:
> -               p = PY_SSL_ERROR_WANT_CONNECT;
> -               errstr = "The operation did not complete (connect)";
> -               break;
> -       case SSL_ERROR_SYSCALL:
> -       {
> -               unsigned long e = ERR_get_error();
> -               if (e == 0) {
> -                       if (ret == 0 || !obj->Socket) {
> -                               p = PY_SSL_ERROR_EOF;
> -                               errstr = "EOF occurred in violation of protocol";
> -                       } else if (ret == -1) {
> -                               /* the underlying BIO reported an I/O error */
> -                               return obj->Socket->errorhandler();
> -                       } else {  /* possible? */
> +               switch (err) {
> +               case SSL_ERROR_ZERO_RETURN:
> +                       errstr = "TLS/SSL connection has been closed";
> +                       p = PY_SSL_ERROR_ZERO_RETURN;
> +                       break;
> +               case SSL_ERROR_WANT_READ:
> +                       errstr = "The operation did not complete (read)";
> +                       p = PY_SSL_ERROR_WANT_READ;
> +                       break;
> +               case SSL_ERROR_WANT_WRITE:
> +                       p = PY_SSL_ERROR_WANT_WRITE;
> +                       errstr = "The operation did not complete (write)";
> +                       break;
> +               case SSL_ERROR_WANT_X509_LOOKUP:
> +                       p = PY_SSL_ERROR_WANT_X509_LOOKUP;
> +                       errstr = "The operation did not complete (X509 lookup)";
> +                       break;
> +               case SSL_ERROR_WANT_CONNECT:
> +                       p = PY_SSL_ERROR_WANT_CONNECT;
> +                       errstr = "The operation did not complete (connect)";
> +                       break;
> +               case SSL_ERROR_SYSCALL:
> +               {
> +                       unsigned long e = ERR_get_error();
> +                       if (e == 0) {
> +                               if (ret == 0 || !obj->Socket) {
> +                                       p = PY_SSL_ERROR_EOF;
> +                                       errstr = "EOF occurred in violation of protocol";
> +                               } else if (ret == -1) {
> +                                       /* the underlying BIO reported an I/O error */
> +                                       return obj->Socket->errorhandler();
> +                               } else { /* possible? */
> +                                       p = PY_SSL_ERROR_SYSCALL;
> +                                       errstr = "Some I/O error occurred";
> +                               }
> +                       } else {
>                                 p = PY_SSL_ERROR_SYSCALL;
> -                               errstr = "Some I/O error occurred";
> +                               /* XXX Protected by global interpreter lock */
> +                               errstr = ERR_error_string(e, NULL);
>                         }
> -               } else {
> -                       p = PY_SSL_ERROR_SYSCALL;
> -                       /* XXX Protected by global interpreter lock */
> -                       errstr = ERR_error_string(e, NULL);
> +                       break;
>                 }
> -               break;
> -       }
> -       case SSL_ERROR_SSL:
> -       {
> -               unsigned long e = ERR_get_error();
> -               p = PY_SSL_ERROR_SSL;
> -               if (e != 0)
> -                       /* XXX Protected by global interpreter lock */
> -                       errstr = ERR_error_string(e, NULL);
> -               else { /* possible? */
> -                       errstr = "A failure in the SSL library occurred";
> +               case SSL_ERROR_SSL:
> +               {
> +                       unsigned long e = ERR_get_error();
> +                       p = PY_SSL_ERROR_SSL;
> +                       if (e != 0)
> +                               /* XXX Protected by global interpreter lock */
> +                               errstr = ERR_error_string(e, NULL);
> +                       else {  /* possible? */
> +                               errstr = "A failure in the SSL library occurred";
> +                       }
> +                       break;
>                 }
> -               break;
> +               default:
> +                       p = PY_SSL_ERROR_INVALID_ERROR_CODE;
> +                       errstr = "Invalid error code";
> +               }
> +       } else {
> +               errstr = ERR_error_string(ERR_peek_last_error(), NULL);
>         }
> -       default:
> -               p = PY_SSL_ERROR_INVALID_ERROR_CODE;
> -               errstr = "Invalid error code";
> -       }
> -
>         PyOS_snprintf(buf, sizeof(buf), "_ssl.c:%d: %s", lineno, errstr);
>         v = Py_BuildValue("(is)", p, buf);
>         if (v != NULL) {
> @@ -256,8 +259,8 @@
>                         ret = SSL_CTX_load_verify_locations(self->ctx,
>                                                             cacerts_file, NULL);
>                         Py_END_ALLOW_THREADS
> -                       if (ret < 1) {
> -                               errstr = ERRSTR("SSL_CTX_load_verify_locations");
> +                       if (ret != 1) {
> +                               PySSL_SetError(NULL, 0, __FILE__, __LINE__);
>                                 goto fail;
>                         }
>                 }
> @@ -267,8 +270,8 @@
>                 ret = SSL_CTX_use_PrivateKey_file(self->ctx, key_file,
>                                                   SSL_FILETYPE_PEM);
>                 Py_END_ALLOW_THREADS
> -               if (ret < 1) {
> -                       errstr = ERRSTR("SSL_CTX_use_PrivateKey_file error");
> +               if (ret != 1) {
> +                       PySSL_SetError(NULL, 0, __FILE__, __LINE__);
>                         goto fail;
>                 }
>
> @@ -276,8 +279,8 @@
>                 ret = SSL_CTX_use_certificate_chain_file(self->ctx,
>                                                        cert_file);
>                 Py_END_ALLOW_THREADS
> -               if (ret < 1) {
> -                       errstr = ERRSTR("SSL_CTX_use_certificate_chain_file error") ;
> +               if (ret != 1) {
> +                       PySSL_SetError(NULL, 0, __FILE__, __LINE__);
>                         goto fail;
>                 }
>                 SSL_CTX_set_options(self->ctx, SSL_OP_ALL); /* ssl compatibility */
> @@ -375,7 +378,7 @@
>  }
>
>  static PyObject *
> -PySocket_ssl(PyObject *self, PyObject *args)
> +PySSL_sslwrap(PyObject *self, PyObject *args)
>  {
>         PySocketSockObject *Sock;
>         int server_side = 0;
> @@ -431,6 +434,9 @@
>         PyObject *pd = PyDict_New();
>         int index_counter;
>
> +       if (pd == NULL)
> +               return NULL;
> +
>         for (index_counter = 0;
>              index_counter < X509_NAME_entry_count(xname);
>              index_counter++)
> @@ -548,7 +554,7 @@
>         }
>         Py_DECREF(pnotBefore);
>
> -       BIO_reset(biobuf);
> +       (void) BIO_reset(biobuf);
>         notAfter = X509_get_notAfter(self->peer_cert);
>         ASN1_TIME_print(biobuf, notAfter);
>         len = BIO_gets(biobuf, buf, sizeof(buf)-1);
> @@ -857,7 +863,7 @@
>
>  static PyTypeObject PySSL_Type = {
>         PyVarObject_HEAD_INIT(NULL, 0)
> -       "socket.SSL",                   /*tp_name*/
> +       "ssl.SSLContext",               /*tp_name*/
>         sizeof(PySSLObject),            /*tp_basicsize*/
>         0,                              /*tp_itemsize*/
>         /* methods */
> @@ -932,7 +938,7 @@
>  "RAND_egd(path) -> bytes\n\
>  \n\
>  Queries the entropy gather daemon (EGD) on socket path.  Returns number\n\
> -of bytes read.  Raises socket.sslerror if connection to EGD fails or\n\
> +of bytes read.  Raises ssl.sslerror if connection to EGD fails or\n\
>  if it does provide enough data to seed PRNG.");
>
>  #endif
> @@ -940,7 +946,7 @@
>  /* List of functions exported by this module. */
>
>  static PyMethodDef PySSL_methods[] = {
> -       {"sslwrap",             PySocket_ssl,
> +       {"sslwrap",             PySSL_sslwrap,
>           METH_VARARGS, ssl_doc},
>  #ifdef HAVE_OPENSSL_RAND
>         {"RAND_add",            PySSL_RAND_add, METH_VARARGS,
> @@ -979,7 +985,7 @@
>         SSLeay_add_ssl_algorithms();
>
>         /* Add symbols to module dict */
> -       PySSLErrorObject = PyErr_NewException("socket.sslerror",
> +       PySSLErrorObject = PyErr_NewException("ssl.sslerror",
>                                               PySocketModule.error,
>                                               NULL);
>         if (PySSLErrorObject == NULL)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From janssen at parc.com  Mon Aug 27 21:04:19 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 27 Aug 2007 12:04:19 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ca471dc20708271142j7ebf38dbt4d2b4273cf19cf09@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<-6964906228903917740@unknownmsgid>
	<ca471dc20708271142j7ebf38dbt4d2b4273cf19cf09@mail.gmail.com>
Message-ID: <07Aug27.120423pdt."57996"@synergy1.parc.xerox.com>

> Anything else I can check in for you?

Ho, ho!

No, I'm done for now.  I think everything is working; let's see what
the buildbots say.  I've got to do some job work now, and I promised
Barry I'd help with the email work, but I'll get back to this with a
port to the Py3K branch.  I'd also like to expand the test_ssl suite
to cover more edge cases, particularly failure modes.  Maybe I'll do
that first.

Actually... if you're in a check-in mood, there is the documentation
patch (issue 1024), which explains about certificates...

Bill


From lars at gustaebel.de  Mon Aug 27 21:59:50 2007
From: lars at gustaebel.de (Lars =?iso-8859-15?Q?Gust=E4bel?=)
Date: Mon, 27 Aug 2007 21:59:50 +0200
Subject: [Python-Dev] tarfile and directory traversal vulnerability
In-Reply-To: <46D30C94.8020204@suse.cz>
References: <46CF1729.5020308@suse.cz> <20070825101312.GA31481@core.g33x.de>
	<46D30C94.8020204@suse.cz>
Message-ID: <20070827195950.GA8546@core.g33x.de>

On Mon, Aug 27, 2007 at 07:40:36PM +0200, Jan Matejek wrote:
> Lars Gust?bel wrote:
> > Suppose we have:
> > foo -> /etc
> > foo/passwd
> > 
> > If creation of the foo symlink is delayed, foo/passwd will be
> > extracted in a directory foo which will be created implicitly.
> > If we create the foo symlink afterwards it will fail because foo
> > already exists. The best way would be to completely ignore
> > members and link targets that are absolute or outside the
> > archive's scope.
> 
> GNU tar doesn't descend into symlinked directories when extracting, such
> archive fails anyway:
> 
> # tar xvf foo.tar
> foo
> foo/passwd
> tar: foo/passwd: Cannot open: Not a directory
> tar: Error exit delayed from previous errors
> 
> I think that is the simplest solution, but i'm not sure how to best
> implement that in extractall().

GNU tar creates a placeholder file for every hard or symbolic
link during the extract process and in a second step replaces
them with links.
I don't think that this is a good choice for a library. The
problem is that it leads to delayed and (from the user's POV)
unrelated errors. I prefer the solution that archive members
with pathnames that either start with a "/" or a "../" raise an
exception by default and can be extracted only by direct
request.

I am currently working on a patch. Should we move this
discussion over to the bugtracker?

-- 
Lars Gust?bel
lars at gustaebel.de

Linux is like a wigwam - no Gates, no Windows, Apache inside.

From janssen at parc.com  Tue Aug 28 02:33:18 2007
From: janssen at parc.com (Bill Janssen)
Date: Mon, 27 Aug 2007 17:33:18 PDT
Subject: [Python-Dev] next steps in SSL work
Message-ID: <07Aug27.173328pdt."57996"@synergy1.parc.xerox.com>

I think the next steps to take are as follows, in order:

1)  Generate a patch to the trunk to remove all use of socket.ssl in
    library modules (and elsewhere except for
    test/test_socket_ssl.py), and switch them to use the ssl module.
    This would affect httplib, imaplib, poplib, smtplib, urllib,
    and xmlrpclib.

    This patch should also deprecate the use of socket.ssl, and
    particularly the "server" and "issuer" methods on it, which can
    return bad data.

    I don't know how to deprecate something...  Pointers?

2)  Expand the test suite to exhaustively test edge cases, particularly
    things like invalid protocol ids, bad cert files, bad key files,
    etc.

3)  Take the threaded server example in test/test_ssl.py, clean it up,
    and add it to the Demos directory (maybe it should be a HOWTO?).

4)  Generate a patch for the Py3K branch.  This patch would remove the
    "ssl" function from the socket module, and would also remove the
    "server" and "issuer" methods on the SSL context.  The ssl.sslsocket
    class would be renamed to SSLSocket (PEP 8), and would inherit
    from socket.socket and io.RawIOBase.  The current improvements to
    the Modules/_ssl.c file would be folded in.  The patch would
    also fix all uses of socket.ssl in the other library modules.

5)  Generate a package for older Pythons (2.3-2.5).  This would
    install the ssl module, plus the improved version of _ssl.c.
    Needs more design.

Bill


From djm at mindrot.org  Tue Aug 28 14:55:07 2007
From: djm at mindrot.org (Damien Miller)
Date: Tue, 28 Aug 2007 22:55:07 +1000 (EST)
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <07Aug26.102820pdt."57996"@synergy1.parc.xerox.com>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<07Aug26.102820pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <Pine.BSO.4.64.0708282253500.4400@fuyu.mindrot.org>

On Sun, 26 Aug 2007, Bill Janssen wrote:

> This must be the call to os.system in test_ssl.py:create_cert_files().
> It's very UNIX-y.  Can any bi-platform folks suggest a good
> alternative to
> 
>     os.system(
>         "openssl req -batch -new -x509 -days 10 -nodes -config %s "
>         "-keyout \"%s\" -out \"%s\" > /dev/null < /dev/null 2>&1" %
>         (conffile, crtfile, crtfile))
> 
> that would be more Windows-friendly?

Bundle a cert and key as part of the test? Is there a requirement to
use a freshly-minted certificate each time?

-d

From martin at v.loewis.de  Tue Aug 28 17:46:09 2007
From: martin at v.loewis.de (=?ISO-8859-2?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 28 Aug 2007 17:46:09 +0200
Subject: [Python-Dev] tarfile and directory traversal vulnerability
In-Reply-To: <46D30B64.3060903@suse.cz>
References: <46CF1729.5020308@suse.cz> <46CF5DD6.9080701@v.loewis.de>
	<46D30B64.3060903@suse.cz>
Message-ID: <46D44341.9010304@v.loewis.de>

> GNU tar is not supposed to place files outside its working directory,
> unless explicitly specified otherwise. So this is considered a security
> vulnerability.

So that's a vulnerability in GNU tar, sure - it does something that it
is not supposed to do.

But why is there also a vulnerability in tarfile.py? It does very well
what it is supposed to do.

> AFAIK there is no specified behavior and other tars might act
> differently. 

I think you are mistaken here. POSIX specifies something (although
I'm uncertain what precisely) for pax(1); this ended the tar wars.

> Furthermore, extract() and extractall() documentation says "Extract
> (...) from the archive to the *current working directory* or directory
> [path]."
> So current behavior is actually inconsistent with the documentation.

Ok. However, what does it mean to create a file with an absolute path
in the current directory?

Also, it's fairly easy to see what creating "../foo" should do when
done in the current directory: create a sibling of the current
directory.

> No, the tar file itself is correct, according to POSIX. You can put
> anything into a tar. Point is, you should be able to untar any file
> 'safely'.

I see, you are asking for an option. If people want to have this option,
it should be added.

Then, of course, the question is what default it should take.

Regards,
Martin

From janssen at parc.com  Tue Aug 28 18:35:32 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 28 Aug 2007 09:35:32 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <Pine.BSO.4.64.0708282253500.4400@fuyu.mindrot.org> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<07Aug26.102820pdt."57996"@synergy1.parc.xerox.com>
	<Pine.BSO.4.64.0708282253500.4400@fuyu.mindrot.org>
Message-ID: <07Aug28.093540pdt."57996"@synergy1.parc.xerox.com>

> On Sun, 26 Aug 2007, Bill Janssen wrote:
> 
> > This must be the call to os.system in test_ssl.py:create_cert_files().
> > It's very UNIX-y.  Can any bi-platform folks suggest a good
> > alternative to
> > 
> >     os.system(
> >         "openssl req -batch -new -x509 -days 10 -nodes -config %s "
> >         "-keyout \"%s\" -out \"%s\" > /dev/null < /dev/null 2>&1" %
> >         (conffile, crtfile, crtfile))
> > 
> > that would be more Windows-friendly?
> 
> Bundle a cert and key as part of the test? Is there a requirement to
> use a freshly-minted certificate each time?
> 
> -d

That's what we've done, and it seems to have turned the Ubuntu and
Debian buildbots green again.  The Windows question is harder, though
-- I'm still not seeing a clean buildbot test on Windows.

By the way, this solution introduces a Y2K problem into the test code.
The included cert runs out in February of 2013, and will need to be
regenerated then.

Bill

From janssen at parc.com  Tue Aug 28 19:25:25 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 28 Aug 2007 10:25:25 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
Message-ID: <07Aug28.102525pdt."57996"@synergy1.parc.xerox.com>

Looking at the buildbots today, Windows is still unhappy.  It seems to
have gotten further, though.

The problematic test is this one:

  File "C:\buildbot_py25\trunk.mcintyre-windows\build\lib\test\test_ssl.py", line 77, in testSSLconnect
      s.connect(("pop.gmail.com", 995))
  File "C:\buildbot_py25\trunk.mcintyre-windows\build\lib\ssl.py", line 170, in connect
    if self._sslobj or (self.getsockname()[1] != 0):
  File "<string>", line 1, in getsockname
    error: (10022, 'Invalid argument')

Looks like a valid complaint, actually.  It should probably read

    if self._sslobj or (int(getnameinfo(self.getsockname(), 0)[1]) != 0):

where "getnameinfo" is from the socket module.  We're trying to see if
the socket has been bound or connected already.

I'm loath to generate a patch till I actually try this on a Windows
machine.  So I'm going to try that this afternoon, and generate a new
patch for you.

Bill

From skip at pobox.com  Tue Aug 28 21:24:47 2007
From: skip at pobox.com (skip at pobox.com)
Date: Tue, 28 Aug 2007 14:24:47 -0500
Subject: [Python-Dev] test_calendar broken on trunk
Message-ID: <18132.30335.222550.80693@montanaro.dyndns.org>

On the community trunk buildbots this checkin:

   1.

      Changed by: walter.doerwald
      Changed at: Tue 28 Aug 2007 16:38:27
      Branch: trunk
      Revision: 57620

      Changed files:
          * trunk/Doc/library/calendar.rst
          * trunk/Lib/calendar.py
      Comments:

      Fix title endtag in HTMLCalender.formatyearpage(). Fix documentation
      for
      HTMLCalender.formatyearpage() (there's no themonth parameter).

      This fixes issue1046.

broke test_calendar.  Details here:

    http://www.python.org/dev/buildbot/community/all/

Skip

From facundobatista at gmail.com  Tue Aug 28 23:05:41 2007
From: facundobatista at gmail.com (Facundo Batista)
Date: Tue, 28 Aug 2007 18:05:41 -0300
Subject: [Python-Dev] Merging a branch, and new docs
Message-ID: <e04bdf310708281405t444a3a2paa763cacb2b04001@mail.gmail.com>

Hello everybody.

Mark Dickinson helped a lot (*a lot*) with the decimal branch, and
we're near to pass the brand new test cases from Cowlishaw.

My original idea is to update all the documentation before merging the
branch into the trunk, but now that they changed so much, I don't know
what to do:

- Update all the Doc dir in the branch with the new Doc dir in the
trunk, fix the docs, and make the merge.

- Merge the branch, and fix the docs directly in the trunk (I promise
to fix them months before 2.6a0).

Thanks for the advice!

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From jafo at tummy.com  Tue Aug 28 23:37:45 2007
From: jafo at tummy.com (Sean Reifschneider)
Date: Tue, 28 Aug 2007 15:37:45 -0600
Subject: [Python-Dev] Triage of old tracker bugs: Any use?
Message-ID: <20070828213745.GA12428@tummy.com>

I've taken the week off and I'm trying to do something useful for Python in
some of my time.  I've basically been looking through the entries sorted by
priority and least recent activity.

Some items I've been able to do something with (like the "immediate"
priority %formatting bug #1467929, and the "high" priority bz2 module bug
#1597011).  Others I've been just kind of prodding people to take some
action on, just kind of getting them in front of people again.  Keeping
them "fresh" instead of just letting them stagnate...

I kind of figure that something that's in "high" priority, that has been
sitting there for 46 months, either needs to have some activity on it
or should be pushed to a lower priority.

I've also been tempted to try to triage some of the bugs without assigned
priorities, guessing a priority, that sort of thing.

Is doing this sort of triage or administration work useful?  Any
recommendations on what you'd like to have happen in this sort of task?

Thanks,
Sean
-- 
 Obtuse: Not pointed or acute. Exceeding 90 degrees but less than 180 degrees.
 OOOooh!  Rounded at the free end.  Dull...  Hey!  That's an insult!  -- WKRP
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability


From brett at python.org  Tue Aug 28 23:58:11 2007
From: brett at python.org (Brett Cannon)
Date: Tue, 28 Aug 2007 14:58:11 -0700
Subject: [Python-Dev] Triage of old tracker bugs: Any use?
In-Reply-To: <20070828213745.GA12428@tummy.com>
References: <20070828213745.GA12428@tummy.com>
Message-ID: <bbaeab100708281458i7c459db7te38ed8eba30e1aaf@mail.gmail.com>

On 8/28/07, Sean Reifschneider <jafo at tummy.com> wrote:
> I've taken the week off and I'm trying to do something useful for Python in
> some of my time.  I've basically been looking through the entries sorted by
> priority and least recent activity.
>
> Some items I've been able to do something with (like the "immediate"
> priority %formatting bug #1467929, and the "high" priority bz2 module bug
> #1597011).  Others I've been just kind of prodding people to take some
> action on, just kind of getting them in front of people again.  Keeping
> them "fresh" instead of just letting them stagnate...
>
> I kind of figure that something that's in "high" priority, that has been
> sitting there for 46 months, either needs to have some activity on it
> or should be pushed to a lower priority.
>
> I've also been tempted to try to triage some of the bugs without assigned
> priorities, guessing a priority, that sort of thing.
>
> Is doing this sort of triage or administration work useful?  Any
> recommendations on what you'd like to have happen in this sort of task?

I think closing off old bugs is helpful.  We have 1281 bugs open right
now and that is not great.  Getting that number down would be useful.
If an old bug needs a test, then write the test.  If it is no longer
valid, just close it.  And if it needs more info and prodding from
somone, set the status as "Pending" so that if someone who reported it
is not bothering to update the info it can be easily closed off if
that reply never happens.

I would say assigning bugs to the right person is a good thing in
hopes of prodding someone into action, but that doesn't always work.

One thing, Sean, while you are doing all of this (and thanks for
that!) is to be thinking about ways to possibly change the workflow
for issues (and this goes for anyone else using the new tracker).  At
some point we should have a discussion about how we want to change how
things are handled so that we are happy with it and not trying to
match how SF did things.

-Brett

From skip at pobox.com  Wed Aug 29 00:27:16 2007
From: skip at pobox.com (skip at pobox.com)
Date: Tue, 28 Aug 2007 17:27:16 -0500
Subject: [Python-Dev] Triage of old tracker bugs: Any use?
In-Reply-To: <20070828213745.GA12428@tummy.com>
References: <20070828213745.GA12428@tummy.com>
Message-ID: <18132.41284.883306.599063@montanaro.dyndns.org>


    Sean> I kind of figure that something that's in "high" priority, that
    Sean> has been sitting there for 46 months, either needs to have some
    Sean> activity on it or should be pushed to a lower priority.

Note that it might have been high priority for the submitter.  That doesn't
necessarily mean it should have been marked that way though.

    Sean> Is doing this sort of triage or administration work useful?  Any
    Sean> recommendations on what you'd like to have happen in this sort of
    Sean> task?

I think it should be useful.  If nothing else, you might add a comment
asking the submitter if this is still a problem.  If something's been there
for nearly four years it's possible that it was solved in a later release or
worked around by other means.

Skip

From janssen at parc.com  Wed Aug 29 00:27:06 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 28 Aug 2007 15:27:06 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <07Aug28.102525pdt."57996"@synergy1.parc.xerox.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
	<07Aug28.102525pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug28.152711pdt."57996"@synergy1.parc.xerox.com>

> The problematic test is this one:
> 
>   File "C:\buildbot_py25\trunk.mcintyre-windows\build\lib\test\test_ssl.py", line 77, in testSSLconnect
>       s.connect(("pop.gmail.com", 995))
>   File "C:\buildbot_py25\trunk.mcintyre-windows\build\lib\ssl.py", line 170, in connect
>     if self._sslobj or (self.getsockname()[1] != 0):
>   File "<string>", line 1, in getsockname
>     error: (10022, 'Invalid argument')

Actually, I think this is a bug in the Windows implementation of the
socket interface.  The question is, what should s.getsockname() return
(or throw) if called on an unbound socket instance?  Unix decided one
way, and Windows another (and not a particularly good way).  I'll post
an issue to the tracker.

Bill




From guido at python.org  Wed Aug 29 00:40:15 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 28 Aug 2007 15:40:15 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <6032561498811032467@unknownmsgid>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
	<6032561498811032467@unknownmsgid>
Message-ID: <ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com>

On 8/28/07, Bill Janssen <janssen at parc.com> wrote:
> > The problematic test is this one:
> >
> >   File "C:\buildbot_py25\trunk.mcintyre-windows\build\lib\test\test_ssl.py", line 77, in testSSLconnect
> >       s.connect(("pop.gmail.com", 995))
> >   File "C:\buildbot_py25\trunk.mcintyre-windows\build\lib\ssl.py", line 170, in connect
> >     if self._sslobj or (self.getsockname()[1] != 0):
> >   File "<string>", line 1, in getsockname
> >     error: (10022, 'Invalid argument')
>
> Actually, I think this is a bug in the Windows implementation of the
> socket interface.  The question is, what should s.getsockname() return
> (or throw) if called on an unbound socket instance?  Unix decided one
> way, and Windows another (and not a particularly good way).  I'll post
> an issue to the tracker.

Well, that's just what you get with two independent implementations of
a spec. We don't try to hide the differences between the sockets stack
in Unix and Windows -- you'll just have to work around it.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From janssen at parc.com  Wed Aug 29 01:19:04 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 28 Aug 2007 16:19:04 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
	<6032561498811032467@unknownmsgid>
	<ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com>
Message-ID: <07Aug28.161906pdt."57996"@synergy1.parc.xerox.com>

> Well, that's just what you get with two independent implementations of
> a spec. We don't try to hide the differences between the sockets stack
> in Unix and Windows -- you'll just have to work around it.

No problem.  But I think it's still a bug -- the "spec" (where is it?)
should say what we want.  I'll file an issue on it.

Bill

From walter at livinglogic.de  Wed Aug 29 01:38:56 2007
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Wed, 29 Aug 2007 01:38:56 +0200
Subject: [Python-Dev] test_calendar broken on trunk
In-Reply-To: <18132.30335.222550.80693@montanaro.dyndns.org>
References: <18132.30335.222550.80693@montanaro.dyndns.org>
Message-ID: <46D4B210.8070501@livinglogic.de>

skip at pobox.com wrote:

> On the community trunk buildbots this checkin:
> 
>    1.
> 
>       Changed by: walter.doerwald
>       Changed at: Tue 28 Aug 2007 16:38:27
>       Branch: trunk
>       Revision: 57620
> 
>       Changed files:
>           * trunk/Doc/library/calendar.rst
>           * trunk/Lib/calendar.py
>       Comments:
> 
>       Fix title endtag in HTMLCalender.formatyearpage(). Fix documentation
>       for
>       HTMLCalender.formatyearpage() (there's no themonth parameter).
> 
>       This fixes issue1046.
> 
> broke test_calendar.  Details here:
> 
>     http://www.python.org/dev/buildbot/community/all/

Should be fixed in r57628.

Servus,
    Walter

From janssen at parc.com  Wed Aug 29 02:46:07 2007
From: janssen at parc.com (Bill Janssen)
Date: Tue, 28 Aug 2007 17:46:07 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
	<6032561498811032467@unknownmsgid>
	<ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com>
Message-ID: <07Aug28.174614pdt."57996"@synergy1.parc.xerox.com>

So, the patch is attached to issue 1052.  I sent it out to python-dev,
but it got blocked (too big).

http://bugs.python.org/file8352/ssl-patch-5

This contains a number of things:

1) Improve the documentation of the SSL module, with a fuller
   explanation of certificate usage, another reference, proper 
   formatting of this and that.

2) Fix Windows bug in ssl.py, and general bug in sslsocket.close().  
   Remove some unused code from ssl.py.  Allow accept() to be called on 
   sslsocket sockets.

3) Use try-except-else in import of ssl in socket.py.  Deprecate use of
   socket.ssl().

4) Remove use of socket.ssl() in every library module, except for
   test_socket_ssl.py and test_ssl.py.

Bill

From guido at python.org  Wed Aug 29 03:45:00 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 28 Aug 2007 18:45:00 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <8516002944057766852@unknownmsgid>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
	<6032561498811032467@unknownmsgid>
	<ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com>
	<8516002944057766852@unknownmsgid>
Message-ID: <ca471dc20708281845u6559587djbde1a836d82c06d8@mail.gmail.com>

Hi Bill,

I think it's time you get to do your own checkins. There's a protocol
but I forget how to do it -- I think you mail your ssh key to Martin.

--Guido

On 8/28/07, Bill Janssen <janssen at parc.com> wrote:
> So, the patch is attached to issue 1052.  I sent it out to python-dev,
> but it got blocked (too big).
>
> http://bugs.python.org/file8352/ssl-patch-5
>
> This contains a number of things:
>
> 1) Improve the documentation of the SSL module, with a fuller
>    explanation of certificate usage, another reference, proper
>    formatting of this and that.
>
> 2) Fix Windows bug in ssl.py, and general bug in sslsocket.close().
>    Remove some unused code from ssl.py.  Allow accept() to be called on
>    sslsocket sockets.
>
> 3) Use try-except-else in import of ssl in socket.py.  Deprecate use of
>    socket.ssl().
>
> 4) Remove use of socket.ssl() in every library module, except for
>    test_socket_ssl.py and test_ssl.py.
>
> Bill
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From collinw at gmail.com  Wed Aug 29 05:15:38 2007
From: collinw at gmail.com (Collin Winter)
Date: Tue, 28 Aug 2007 20:15:38 -0700
Subject: [Python-Dev] Avoiding cascading test failures
In-Reply-To: <acd65fa20708221644y2ab376bemd4d550ff43fdd91@mail.gmail.com>
References: <acd65fa20708221644y2ab376bemd4d550ff43fdd91@mail.gmail.com>
Message-ID: <43aa6ff70708282015q56699bebk4bc52a38749d121e@mail.gmail.com>

On 8/22/07, Alexandre Vassalotti <alexandre at peadrop.com> wrote:
> When I was fixing tests failing in the py3k branch, I found the number
> duplicate failures annoying. Often, a single bug, in an important
> method or function, caused a large number of testcase to fail. So, I
> thought of a simple mechanism for avoiding such cascading failures.
>
> My solution is to add a notion of dependency to testcases. A typical
> usage would look like this:
>
>     @depends('test_getvalue')
>     def test_writelines(self):
>         ...
>         memio.writelines([buf] * 100)
>         self.assertEqual(memio.getvalue(), buf * 100)
>         ...

This definitely seems like a neat idea. Some thoughts:

* How do you deal with dependencies that cross test modules? Say test
A depends on test B, how do we know whether it's worthwhile to run A
if B hasn't been run yet? It looks like you run the test anyway (I
haven't studied the code closely), but that doesn't seem ideal.

* This might be implemented in the wrong place. For example, the [x
for x in dir(self) if x.startswith('test')] you do is most certainly
better-placed in a custom TestLoader implementation. That might also
make it possible to order tests based on their dependency graph, which
could be a step toward addressing the above point.

But despite that, I think it's a cool idea and worth pursuing. Could
you set up a branch (probably of py3k) so we can see how this plays
out in the large?

Collin Winter

From nnorwitz at gmail.com  Wed Aug 29 06:24:07 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 28 Aug 2007 21:24:07 -0700
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <ca471dc20708281845u6559587djbde1a836d82c06d8@mail.gmail.com>
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
	<6032561498811032467@unknownmsgid>
	<ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com>
	<8516002944057766852@unknownmsgid>
	<ca471dc20708281845u6559587djbde1a836d82c06d8@mail.gmail.com>
Message-ID: <ee2a432c0708282124k55e0af13y4a83a726b8ce5df9@mail.gmail.com>

Bill,

Please mail your ssh key as an attachment to pydotorg at python.org.

n
--

On 8/28/07, Guido van Rossum <guido at python.org> wrote:
> Hi Bill,
>
> I think it's time you get to do your own checkins. There's a protocol
> but I forget how to do it -- I think you mail your ssh key to Martin.
>
> --Guido
>
> On 8/28/07, Bill Janssen <janssen at parc.com> wrote:
> > So, the patch is attached to issue 1052.  I sent it out to python-dev,
> > but it got blocked (too big).
> >
> > http://bugs.python.org/file8352/ssl-patch-5
> >
> > This contains a number of things:
> >
> > 1) Improve the documentation of the SSL module, with a fuller
> >    explanation of certificate usage, another reference, proper
> >    formatting of this and that.
> >
> > 2) Fix Windows bug in ssl.py, and general bug in sslsocket.close().
> >    Remove some unused code from ssl.py.  Allow accept() to be called on
> >    sslsocket sockets.
> >
> > 3) Use try-except-else in import of ssl in socket.py.  Deprecate use of
> >    socket.ssl().
> >
> > 4) Remove use of socket.ssl() in every library module, except for
> >    test_socket_ssl.py and test_ssl.py.
> >
> > Bill
> >
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>

From martin at v.loewis.de  Wed Aug 29 07:34:40 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 29 Aug 2007 07:34:40 +0200
Subject: [Python-Dev] Triage of old tracker bugs: Any use?
In-Reply-To: <20070828213745.GA12428@tummy.com>
References: <20070828213745.GA12428@tummy.com>
Message-ID: <46D50570.6020407@v.loewis.de>

> Is doing this sort of triage or administration work useful?  Any
> recommendations on what you'd like to have happen in this sort of task?

Down-grading things that were once high-priority is certainly reasonable.

Of the very old bugs, it would be useful to find out whether they are
still reproducable, and, if there is no straight-forward way to
reproduce them, ask the submitter to provide a test case (if there
is a test case that now passes in a case where it apparently did
not pass back then, close it as works-for-me).

It would also be good to triage all reports that have not seen any
feedback. Unfortunately, it's not easy currently to query the tracker
for these, so perhaps the "chatting" state needs to be brought back
first.

Regards,
Martin



From danzat at gmail.com  Wed Aug 29 17:29:36 2007
From: danzat at gmail.com (dany)
Date: Wed, 29 Aug 2007 18:29:36 +0300
Subject: [Python-Dev] PEP 238 - The // operator should truncate instead of
	floor
Message-ID: <46D590E0.5030504@gmail.com>

Hi all

First, some background.
I've recently tried to port a certain library that manipulates dates 
from C. Some of the core functions contain heavy integer math (very long 
formulae), and after implementing them I tested them to see the give the 
desired results.
I got very odd results, which surprised me since I copied the formula 
letter to letter.

I decided to investigate further, and after trying to evaluate several 
expressions in the python console, I realized it was the integer 
division's fault.

For some reason, the integer division behaves unexpectedly for negative 
integers.

Looking deeper in the python PEPs, I saw that division on integers is 
defined as: idiv(a, b) = floor(fdiv(a, b)).
This non-quotient division leads to some odd results, e.g. Python seems 
to think -3/2+3/2 = -1. This is clearly, and correct me if I'm mistaken 
- wrong.

Now, seeing as Python 3000 is getting closer, and backwards 
compatibility isn't an issue, I believe it would be a good idea to 
change the // operator (which will be used for integer division) to 
behave as quotient, i.e.: a // b = trunc(a / b)

Dany

From l.mastrodomenico at gmail.com  Wed Aug 29 18:05:19 2007
From: l.mastrodomenico at gmail.com (Lino Mastrodomenico)
Date: Wed, 29 Aug 2007 18:05:19 +0200
Subject: [Python-Dev] PEP 238 - The // operator should truncate instead
	of floor
In-Reply-To: <46D590E0.5030504@gmail.com>
References: <46D590E0.5030504@gmail.com>
Message-ID: <cc93256f0708290905r22de32abk35c4ce15712dcd86@mail.gmail.com>

2007/8/29, dany <danzat at gmail.com>:
> For some reason, the integer division behaves unexpectedly for negative
> integers.
>
> Looking deeper in the python PEPs, I saw that division on integers is
> defined as: idiv(a, b) = floor(fdiv(a, b)).
> This non-quotient division leads to some odd results, e.g. Python seems
> to think -3/2+3/2 = -1. This is clearly, and correct me if I'm mistaken
> - wrong.

FAQ: Why does -22 / 10 return -3?

http://www.python.org/doc/faq/programming/#why-does-22-10-return-3

Past discussion:

<http://mail.python.org/pipermail/python-list/2005-February/thread.html#306577>

Also, FYI, TCL and Ruby behave the same way.

-- 
Lino Mastrodomenico
E-mail: l.mastrodomenico at gmail.com

From fredrik.johansson at gmail.com  Wed Aug 29 18:23:40 2007
From: fredrik.johansson at gmail.com (Fredrik Johansson)
Date: Wed, 29 Aug 2007 18:23:40 +0200
Subject: [Python-Dev] PEP 238 - The // operator should truncate instead
	of floor
In-Reply-To: <46D590E0.5030504@gmail.com>
References: <46D590E0.5030504@gmail.com>
Message-ID: <3d0cebfb0708290923w85ba386kf8673111b10919c0@mail.gmail.com>

On 8/29/07, dany <danzat at gmail.com> wrote:
> Looking deeper in the python PEPs, I saw that division on integers is
> defined as: idiv(a, b) = floor(fdiv(a, b)).
> This non-quotient division leads to some odd results, e.g. Python seems
> to think -3/2+3/2 = -1. This is clearly, and correct me if I'm mistaken
> - wrong.

Rounding integer division can't be defined such that fractional
information is preserved. For example, the sum 1/2 + 1/2 (with
rounding division), doesn't equal 1 whether the rounding is done up or
down. Python's choice to round to the floor of the exact quotient is
entirely in order, and in fact, I've found it very useful in some of
my code.

There is an easy fix for your problem: write a custom division function, e.g.

def div(a, b):
    if a*b > 0:
        return a // b
    else:
        return -(abs(a)//abs(b))

Either that, or use any of the modules for rational arithmetic
available for Python.

Fredrik

From martin at v.loewis.de  Wed Aug 29 18:30:16 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 29 Aug 2007 18:30:16 +0200
Subject: [Python-Dev] PEP 238 - The // operator should truncate instead
 of	floor
In-Reply-To: <46D590E0.5030504@gmail.com>
References: <46D590E0.5030504@gmail.com>
Message-ID: <46D59F18.9010202@v.loewis.de>

> This non-quotient division leads to some odd results, e.g. Python seems 
> to think -3/2+3/2 = -1. This is clearly, and correct me if I'm mistaken 
> - wrong.

Lino already pointed you to the relevant FAQ entry, but since you
explicitly asked for it: you are mistaken, the result is right.

It would have been good if you had indicated what result you had
expected instead. I assume 0; to get 0, you have to write -(3/2)+3/2,
or 0-3/2+3/2.

Regards,
Martin

From martin at v.loewis.de  Wed Aug 29 18:35:40 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 29 Aug 2007 18:35:40 +0200
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <52dc1c820708262056o3ceb03ffr93d8ba2dd633f79e@mail.gmail.com>
References: <166458247748783241@unknownmsgid>	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>	<5531275898429170325@unknownmsgid>	<ca471dc20708261235k1434d3e9r4a7c1837576da806@mail.gmail.com>	<-4187035365711058012@unknownmsgid>	<ee2a432c0708261520q1d1bcdbfkf776dd31283d5da4@mail.gmail.com>	<4484419364585880902@unknownmsgid>
	<52dc1c820708262056o3ceb03ffr93d8ba2dd633f79e@mail.gmail.com>
Message-ID: <46D5A05C.3020604@v.loewis.de>

> apt-get install openssl  will fix that on those systems.  on windows
> you're unlikely to ever have an openssl binary present and available to
> execute.

OTOH, having openssl(1) installed is not a prerequisite for Python
buildbots currently, and IMO shouldn't be.

Regards,
Martin

From martin at v.loewis.de  Wed Aug 29 18:44:09 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 29 Aug 2007 18:44:09 +0200
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <07Aug28.161906pdt."57996"@synergy1.parc.xerox.com>
References: <166458247748783241@unknownmsgid>	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>	<7051374911840721061@unknownmsgid>	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>	<6032561498811032467@unknownmsgid>	<ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com>
	<07Aug28.161906pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <46D5A259.5070007@v.loewis.de>

Bill Janssen schrieb:
>> Well, that's just what you get with two independent implementations of
>> a spec. We don't try to hide the differences between the sockets stack
>> in Unix and Windows -- you'll just have to work around it.
> 
> No problem.  But I think it's still a bug -- the "spec" (where is it?)
> should say what we want.  I'll file an issue on it.

I think "the spec" would be the socket API. For sockets, that is POSIX.
According to

http://www.opengroup.org/pubs/online/7908799/xns/getsockname.html

the result is unspecified if the socket has not been bound to a local
name.

However, Windows does not implement POSIX here; instead, it implements
WinSock, which specifies that WSAEINVAL is returned.

Regards,
Martin

From facundobatista at gmail.com  Wed Aug 29 19:32:46 2007
From: facundobatista at gmail.com (Facundo Batista)
Date: Wed, 29 Aug 2007 14:32:46 -0300
Subject: [Python-Dev] Order of operations (was: PEP 238 - The // operator
	should truncate instead of floor)
Message-ID: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>

2007/8/29, "Martin v. L?wis" <martin at v.loewis.de>:

> It would have been good if you had indicated what result you had
> expected instead. I assume 0; to get 0, you have to write -(3/2)+3/2,
> or 0-3/2+3/2.

Wow, that caught me:

>>> -3/2+3/2
-1
>>> 0-3/2+3/2
0
>>>

I'm not talking about division here, just the fact that in the first
case, it was (-3) / 2 and not -(3/2).

It's a surprise to me, taking into account that

>>> -3**2
-9
>>> 0-3**2
-9

Here, the behavior is always the same: -(3**2).

Do you know why? Thanks!

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From janssen at parc.com  Wed Aug 29 19:42:53 2007
From: janssen at parc.com (Bill Janssen)
Date: Wed, 29 Aug 2007 10:42:53 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <46D5A259.5070007@v.loewis.de> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
	<6032561498811032467@unknownmsgid>
	<ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com>
	<07Aug28.161906pdt."57996"@synergy1.parc.xerox.com>
	<46D5A259.5070007@v.loewis.de>
Message-ID: <07Aug29.104254pdt."57996"@synergy1.parc.xerox.com>

> I think "the spec" would be the socket API. For sockets, that is POSIX.
> According to
> 
> http://www.opengroup.org/pubs/online/7908799/xns/getsockname.html
> 
> the result is unspecified if the socket has not been bound to a local
> name.
> 
> However, Windows does not implement POSIX here; instead, it implements
> WinSock, which specifies that WSAEINVAL is returned.

So, two different specs meet in the Python API, which means that we
have to define the behavior ourselves.  I'd suggest returning None from
an unbound socket.

Bill

From martin at v.loewis.de  Wed Aug 29 20:17:57 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 29 Aug 2007 20:17:57 +0200
Subject: [Python-Dev] Order of operations (was: PEP 238 - The //
 operator should truncate instead of floor)
In-Reply-To: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>
Message-ID: <46D5B855.7020101@v.loewis.de>

> Do you know why? Thanks!

It's a matter of precedence: ** has higher precedence
than unary + and -, which has higher precedence than / and *,
which have higher precedence than binary + and -. See
power, factor, term in Grammar/Grammar.

I'm not sure why precedence was defined that
way, though.

Regards,
Martin

From janssen at parc.com  Wed Aug 29 21:15:10 2007
From: janssen at parc.com (Bill Janssen)
Date: Wed, 29 Aug 2007 12:15:10 PDT
Subject: [Python-Dev] whitespace-normalized commit hook...
Message-ID: <07Aug29.121510pdt."57996"@synergy1.parc.xerox.com>

> Transmitting file data .........subversion/libsvn_client/commit.c:832: (apr_err=165001)
> svn: Commit failed (details follow):
> /build/buildd/subversion-1.4.2dfsg1/subversion/libsvn_repos/hooks.c:135: (apr_err=165001)
> svn: 'pre-commit' hook failed with error output:
> file /python/trunk/Lib/ssl.py is not whitespace-normalized

So, the pre-commit hook is blocking me; how do I trouble-shoot this?

What's the client-side way to run this scan?  Can I add it to the
regression-test harness?

Bill

From thomas at python.org  Wed Aug 29 21:31:05 2007
From: thomas at python.org (Thomas Wouters)
Date: Wed, 29 Aug 2007 21:31:05 +0200
Subject: [Python-Dev] whitespace-normalized commit hook...
In-Reply-To: <-220758937327998711@unknownmsgid>
References: <-220758937327998711@unknownmsgid>
Message-ID: <9e804ac0708291231p2ca3d478ic9392f61ccb81e67@mail.gmail.com>

On 8/29/07, Bill Janssen <janssen at parc.com> wrote:
>
> > Transmitting file data .........subversion/libsvn_client/commit.c:832:
> (apr_err=165001)
> > svn: Commit failed (details follow):
> > /build/buildd/subversion-1.4.2dfsg1/subversion/libsvn_repos/hooks.c:135:
> (apr_err=165001)
> > svn: 'pre-commit' hook failed with error output:
> > file /python/trunk/Lib/ssl.py is not whitespace-normalized
>
> So, the pre-commit hook is blocking me; how do I trouble-shoot this?
>
> What's the client-side way to run this scan?  Can I add it to the
> regression-test harness?


It basically calls Tools/scripts/reindent.py with --dry-run (although I
notice the version we call from SVN is more forgiving towards tokenizer
errors. I'm not sure we need that here.) Incorporating it in the normal test
suit would be good, as I've been caught by the same problem several times
:-)

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070829/6cc63240/attachment-0001.htm 

From martin at v.loewis.de  Wed Aug 29 21:53:25 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 29 Aug 2007 21:53:25 +0200
Subject: [Python-Dev] whitespace-normalized commit hook...
In-Reply-To: <07Aug29.121510pdt."57996"@synergy1.parc.xerox.com>
References: <07Aug29.121510pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <46D5CEB5.2060303@v.loewis.de>

> So, the pre-commit hook is blocking me; how do I trouble-shoot this?

Run Tools/scripts/reindent.py Lib/ssl.py

Alternatively, run "reindent.py -drv .|grep ' changed'" to analyse your
entire sandbox.

Regards,
Martin

From kevinar18 at hotmail.com  Wed Aug 29 21:56:33 2007
From: kevinar18 at hotmail.com (Kevin Ar18)
Date: Wed, 29 Aug 2007 15:56:33 -0400
Subject: [Python-Dev] zipfile module can't handle large files
Message-ID: <BLU110-W13BB1AB812562662C21EB0AACC0@phx.gbl>


Create a zip file with a file inside consisting of several GB (say 2 or 5 GB).  The zipfile module will fail, when attempting to extract the large file.

The issue is near line 490 in zifile.py.  It appears that read (a file operation) is unable to read such large amounts of data.  I tried editing zipfile.py so that read would read things piece by piece but just got a memory error.

Does anyone know how to fix this limitation in the zipfile module?
_________________________________________________________________
See what you?re getting into?before you go there
http://newlivehotmail.com/?ocid=TXT_TAGHM_migration_HM_viral_preview_0507

From scott+python-dev at scottdial.com  Wed Aug 29 21:38:49 2007
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Wed, 29 Aug 2007 15:38:49 -0400
Subject: [Python-Dev] Order of operations (was: PEP 238 - The //
 operator should truncate instead of floor)
In-Reply-To: <46D5B855.7020101@v.loewis.de>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>
	<46D5B855.7020101@v.loewis.de>
Message-ID: <46D5CB49.5030902@scottdial.com>

Martin v. L?wis wrote:
>> Do you know why? Thanks!
> 
> I'm not sure why precedence was defined that
> way, though.
> 

Because it is consistent with C's precedence rules.

-Scott

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu

From martin at v.loewis.de  Wed Aug 29 22:12:38 2007
From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 29 Aug 2007 22:12:38 +0200
Subject: [Python-Dev] zipfile module can't handle large files
In-Reply-To: <BLU110-W13BB1AB812562662C21EB0AACC0@phx.gbl>
References: <BLU110-W13BB1AB812562662C21EB0AACC0@phx.gbl>
Message-ID: <46D5D336.4070305@v.loewis.de>

> Does anyone know how to fix this limitation in the zipfile module? 

See http://bugs.python.org/issue1189216

Regards,
Martin

From guido at python.org  Wed Aug 29 22:16:58 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 29 Aug 2007 13:16:58 -0700
Subject: [Python-Dev] zipfile module can't handle large files
In-Reply-To: <BLU110-W13BB1AB812562662C21EB0AACC0@phx.gbl>
References: <BLU110-W13BB1AB812562662C21EB0AACC0@phx.gbl>
Message-ID: <ca471dc20708291316k6389ef95n9970222ed1eb3916@mail.gmail.com>

Please file a bug at the new bug tracker (bugs.python.org). Please add
a small self-contained code sample that shows the issue, and specify
what you would like to see instead (keeping in mind that you just
can't allocate 2 GB on a 32-bit box).

--Guido

On 8/29/07, Kevin Ar18 <kevinar18 at hotmail.com> wrote:
>
> Create a zip file with a file inside consisting of several GB (say 2 or 5 GB).  The zipfile module will fail, when attempting to extract the large file.
>
> The issue is near line 490 in zifile.py.  It appears that read (a file operation) is unable to read such large amounts of data.  I tried editing zipfile.py so that read would read things piece by piece but just got a memory error.
>
> Does anyone know how to fix this limitation in the zipfile module?
> _________________________________________________________________
> See what you're getting into?before you go there
> http://newlivehotmail.com/?ocid=TXT_TAGHM_migration_HM_viral_preview_0507
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip at pobox.com  Wed Aug 29 22:20:58 2007
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 29 Aug 2007 15:20:58 -0500
Subject: [Python-Dev] Order of operations (was: PEP 238 - The //
 operator should truncate instead of floor)
In-Reply-To: <46D5CB49.5030902@scottdial.com>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>
	<46D5B855.7020101@v.loewis.de> <46D5CB49.5030902@scottdial.com>
Message-ID: <18133.54570.888609.584006@montanaro.dyndns.org>


    >>> Do you know why? Thanks!
    >> 
    >> I'm not sure why precedence was defined that
    >> way, though.

    Scott> Because it is consistent with C's precedence rules.

True enough, however Python doesn't support negative numbers as individual
tokens at the lexical analysis level.  -3.41 is '-' followed by '3.41'.  In
C it's a single token resolved at compile time.  In the not-too-distant past
negative numeric constants were tried, but that caused some other grief.  I
can't remember what.

Skip

From ty.newton at copperchipgames.com  Wed Aug 29 22:06:46 2007
From: ty.newton at copperchipgames.com (Ty Newton)
Date: Thu, 30 Aug 2007 06:06:46 +1000
Subject: [Python-Dev] newb question
Message-ID: <46D5D1D6.40402@copperchipgames.com>

Hi,
I'm new to the Python source and I have a (hopefully) quick question 
that someone more familiar with it should be able to answer so I can 
continue on my way to understanding how Python is put together.

I have started looking at the parser and have gotten a little confused 
about how the grammar is instantiated (perhaps the wrong term).

Here's how I understand things at the moment.  graminit.c contains the 
definition of the grammar in terms of static structures.  Everything 
seems simple enough until I get to the DFA (in graminit.c):

static dfa dfas[84] = {
{256, "single_input", 0, 3, states_0, 
"\004\050\014\000\000\000\000\025\074\005\023\310\011\020\004\000\300\020\222\006\201"},
...

I assume that the last bit of my snippet (\004\050\014\ ...) is a bitset 
structure.  When I look at bitset it is defined as a char array.

Can someone explain how this works please?  I've never come across 
escape sequences like this; I've only ever seen \0 (nul) before; not \2, 
\3 etc.  or are they not escape sequences, but literal forward slashes.


Thanks,
Ty

From martin at v.loewis.de  Thu Aug 30 00:25:24 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 30 Aug 2007 00:25:24 +0200
Subject: [Python-Dev] Order of operations
In-Reply-To: <46D5CB49.5030902@scottdial.com>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>	<46D5B855.7020101@v.loewis.de>
	<46D5CB49.5030902@scottdial.com>
Message-ID: <46D5F254.9030406@v.loewis.de>

Scott Dial schrieb:
> Martin v. L?wis wrote:
>>> Do you know why? Thanks!
>> I'm not sure why precedence was defined that
>> way, though.
>>
> 
> Because it is consistent with C's precedence rules.

Maybe I'm missing something - how exactly is the exponentiation
operator spelled in C?

Regards,
Martin

From martin at v.loewis.de  Thu Aug 30 00:33:48 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 30 Aug 2007 00:33:48 +0200
Subject: [Python-Dev] Order of operations
In-Reply-To: <18133.54570.888609.584006@montanaro.dyndns.org>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>	<46D5B855.7020101@v.loewis.de>
	<46D5CB49.5030902@scottdial.com>
	<18133.54570.888609.584006@montanaro.dyndns.org>
Message-ID: <46D5F44C.3050501@v.loewis.de>

> True enough, however Python doesn't support negative numbers as individual
> tokens at the lexical analysis level.  -3.41 is '-' followed by '3.41'.  In
> C it's a single token resolved at compile time.

That is not true. ISO C99 defines (6.4.4.1)

               integer-constant:
                       decimal-constant integer-suffix-opt
                       octal-constant integer-suffix-opt
                       hexadecimal-constant integer-suffix-opt

               decimal-constant:
                       nonzero-digit
                       decimal-constant digit

and then (6.5.3)

               unary-expr:
                       postfix-expr
                       ++ unary-expr
                       -- unary-expr
                       unary-operator cast-expr
                       sizeof unary-expr
                       sizeof ( type-name )

               unary-operator: one of
                       &  *  +  -  ~  !

So -3.41 is definitely two tokens in C.

Regards,
Martin

From martin at v.loewis.de  Thu Aug 30 00:38:22 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 30 Aug 2007 00:38:22 +0200
Subject: [Python-Dev] newb question
In-Reply-To: <46D5D1D6.40402@copperchipgames.com>
References: <46D5D1D6.40402@copperchipgames.com>
Message-ID: <46D5F55E.1040708@v.loewis.de>

> Can someone explain how this works please?

Don't try reading these files. They are generated; read Parser/pgen
instead (which generates these files), or see how they are used.

> I've never come across 
> escape sequences like this; I've only ever seen \0 (nul) before; not \2, 
> \3 etc.  or are they not escape sequences, but literal forward slashes.

They are octal escapes; this is a standard C construct. You need three
octal digits to build one character. So it's \222, not \2, and \310, not
\3. The integer values are 0222 and 0310, respectively (i.e. 146 and
200).

HTH,
Martin

From alexandre at peadrop.com  Thu Aug 30 00:41:58 2007
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Wed, 29 Aug 2007 18:41:58 -0400
Subject: [Python-Dev] Order of operations
In-Reply-To: <46D5F254.9030406@v.loewis.de>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>
	<46D5B855.7020101@v.loewis.de> <46D5CB49.5030902@scottdial.com>
	<46D5F254.9030406@v.loewis.de>
Message-ID: <acd65fa20708291541m6f789352l425a33ebd9a596bb@mail.gmail.com>

On 8/29/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Scott Dial schrieb:
> > Martin v. L?wis wrote:
> >>> Do you know why? Thanks!
> >> I'm not sure why precedence was defined that
> >> way, though.
> >>
> >
> > Because it is consistent with C's precedence rules.
>
> Maybe I'm missing something - how exactly is the exponentiation
> operator spelled in C?
>

C doesn't have an exponentiation operator. You use the pow() function, instead:

  #include <math.h>
  double pow(double x, double y);

-- Alexandre

From janssen at parc.com  Thu Aug 30 00:48:24 2007
From: janssen at parc.com (Bill Janssen)
Date: Wed, 29 Aug 2007 15:48:24 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <07Aug28.174614pdt."57996"@synergy1.parc.xerox.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
	<6032561498811032467@unknownmsgid>
	<ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com>
	<07Aug28.174614pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug29.154829pdt."57996"@synergy1.parc.xerox.com>

All right, this is committed.  Let's see if the Windows buildbots like
it.

Bill

> So, the patch is attached to issue 1052.  I sent it out to python-dev,
> but it got blocked (too big).
> 
> http://bugs.python.org/file8352/ssl-patch-5
> 
> This contains a number of things:
> 
> 1) Improve the documentation of the SSL module, with a fuller
>    explanation of certificate usage, another reference, proper 
>    formatting of this and that.
> 
> 2) Fix Windows bug in ssl.py, and general bug in sslsocket.close().  
>    Remove some unused code from ssl.py.  Allow accept() to be called on 
>    sslsocket sockets.
> 
> 3) Use try-except-else in import of ssl in socket.py.  Deprecate use of
>    socket.ssl().
> 
> 4) Remove use of socket.ssl() in every library module, except for
>    test_socket_ssl.py and test_ssl.py.
> 
> Bill
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/janssen%40parc.com



From janssen at parc.com  Thu Aug 30 04:01:16 2007
From: janssen at parc.com (Bill Janssen)
Date: Wed, 29 Aug 2007 19:01:16 PDT
Subject: [Python-Dev] Other SSL issues in the tracker have been marked
In-Reply-To: <07Aug29.154829pdt."57996"@synergy1.parc.xerox.com> 
References: <166458247748783241@unknownmsgid>
	<ee2a432c0708252027t282c447ev45ceb6140b686e9d@mail.gmail.com>
	<7051374911840721061@unknownmsgid>
	<ee2a432c0708261139i6a4f39b0w50d00b0e87b7746d@mail.gmail.com>
	<6032561498811032467@unknownmsgid>
	<ca471dc20708281540q3f1ec7a2gda4c0ee64b188c24@mail.gmail.com>
	<07Aug28.174614pdt."57996"@synergy1.parc.xerox.com>
	<07Aug29.154829pdt."57996"@synergy1.parc.xerox.com>
Message-ID: <07Aug29.190119pdt."57996"@synergy1.parc.xerox.com>

Yes, the Windows buildbots seem happy now with ssl.  I'm going to
expand the test suite.

Bill

From scott+python-dev at scottdial.com  Thu Aug 30 07:49:10 2007
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Thu, 30 Aug 2007 01:49:10 -0400
Subject: [Python-Dev] Order of operations
In-Reply-To: <46D6538E.5020002@v.loewis.de>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>	
	<46D5B855.7020101@v.loewis.de> <46D5CB49.5030902@scottdial.com>	
	<46D5F254.9030406@v.loewis.de>
	<acd65fa20708291541m6f789352l425a33ebd9a596bb@mail.gmail.com>
	<46D6538E.5020002@v.loewis.de>
Message-ID: <46D65A56.2070203@scottdial.com>

Martin v. L?wis wrote:
> Alexandre Vassalotti schrieb:
>> C doesn't have an exponentiation operator. You use the pow() function, instead:
> 
> Right. Then Scott's claim ("Because it is consistent with C's precedence
> rules.") can't be true - from C precedence, it does not follow that **
> must have higher precedence than unary -.
> 

I believe the confusion with my claim is that I thought the oddity in 
question was that -3/2 != 0-3/2, but I believe you took the other half 
about exponentiation as the oddity in question that -3**2 == -(3**2). My 
original comment was to say that unary - was higher precedence than 
division because of C's ordering. With respect to the original question, 
then my answer was poor as I ignored the exponentiation part. Since we 
tacked this on, my answer was to be an implicit zen answer of "because 
it makes the most sense."

Sorry for the added noise.

-Scott

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu

From dirkjan at ochtman.nl  Thu Aug 30 02:48:51 2007
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Thu, 30 Aug 2007 02:48:51 +0200
Subject: [Python-Dev] Order of operations
In-Reply-To: <acd65fa20708291541m6f789352l425a33ebd9a596bb@mail.gmail.com>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>	<46D5B855.7020101@v.loewis.de>
	<46D5CB49.5030902@scottdial.com>	<46D5F254.9030406@v.loewis.de>
	<acd65fa20708291541m6f789352l425a33ebd9a596bb@mail.gmail.com>
Message-ID: <fb545q$tpf$1@sea.gmane.org>

Alexandre Vassalotti wrote:
> C doesn't have an exponentiation operator. You use the pow() function, instead:

Wouldn't it make more sense, then, to have unary +/- have higher 
precedence than the ** operator, so that -3**2 == 9?

Regards,

Dirkjan


From jason.orendorff at gmail.com  Thu Aug 30 13:02:13 2007
From: jason.orendorff at gmail.com (Jason Orendorff)
Date: Thu, 30 Aug 2007 07:02:13 -0400
Subject: [Python-Dev] Order of operations
In-Reply-To: <fb545q$tpf$1@sea.gmane.org>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>
	<46D5B855.7020101@v.loewis.de> <46D5CB49.5030902@scottdial.com>
	<46D5F254.9030406@v.loewis.de>
	<acd65fa20708291541m6f789352l425a33ebd9a596bb@mail.gmail.com>
	<fb545q$tpf$1@sea.gmane.org>
Message-ID: <bb8868b90708300402x3670da30yb6152a0a93bd0f61@mail.gmail.com>

On 8/29/07, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:
> Alexandre Vassalotti wrote:
> > C doesn't have an exponentiation operator. You use the pow() function, instead:
>
> Wouldn't it make more sense, then, to have unary +/- have higher
> precedence than the ** operator, so that -3**2 == 9?

No, that would have been really bad.  Anyone who's had high school
algebra expects -x**2 to be -(x**2) and not (-x)**2.

I think the weirdness comes from parsing -a/b as (-a)/b rather than
-(a/b).  It should be the latter, if compatibility with math notation
is more important than compatibility with C.   Oh well.  Maybe in
Python 4.  :)

-j

From alex.neundorf at kitware.com  Thu Aug 30 17:54:55 2007
From: alex.neundorf at kitware.com (Alexander Neundorf)
Date: Thu, 30 Aug 2007 11:54:55 -0400
Subject: [Python-Dev] HAVE_DECL_TZNAME in pyconfig.h.in, but no test for it ?
Message-ID: <200708301154.55722.alex.neundorf@kitware.com>

Hi,

in current python svn there is a 
#undef HAVE_DECL_TZNAME
but I can't find a configure test for it.
Did I miss something ?

Bye
Alex

From martin at v.loewis.de  Thu Aug 30 22:11:31 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 30 Aug 2007 22:11:31 +0200
Subject: [Python-Dev] HAVE_DECL_TZNAME in pyconfig.h.in,
 but no test for it ?
In-Reply-To: <200708301154.55722.alex.neundorf@kitware.com>
References: <200708301154.55722.alex.neundorf@kitware.com>
Message-ID: <46D72473.5010707@v.loewis.de>

> in current python svn there is a 
> #undef HAVE_DECL_TZNAME
> but I can't find a configure test for it.
> Did I miss something ?

Perhaps that it is generated through autoheader?

pyconfig.h.in was last generated by autoconf 2.61,
and configure by autoconf 2.59. I have now made
sure that the trunk will always use autoconf 2.61.

Regards,
Martin

From alex.neundorf at kitware.com  Thu Aug 30 22:28:56 2007
From: alex.neundorf at kitware.com (Alexander Neundorf)
Date: Thu, 30 Aug 2007 16:28:56 -0400
Subject: [Python-Dev] Building Python with CMake
In-Reply-To: <f78mad$876$1@sea.gmane.org>
References: <200707131359.17030.alex.neundorf@kitware.com>
	<e04bdf310707131153x780293b3p9561b8e5d9952069@mail.gmail.com>
	<f78mad$876$1@sea.gmane.org>
Message-ID: <200708301628.57127.alex.neundorf@kitware.com>

On Friday 13 July 2007 16:11, Giovanni Bajo wrote:
> On 13/07/2007 20.53, Facundo Batista wrote:
> >> as I wrote in my previous email, I'm currently porting Python to some
> >> more unusual platforms, namely to a super computer
> >> (http://www.research.ibm.com/bluegene/) and a tiny embedded operating
> >> system (http://ecos.sourceware.org), which have more or less
> >> surprisingly quite similar properties.
> >
> > Sorry, missed the previous mail. Have two questions for you:
> >
> > - Why?
>
> Because it would be a single unified build system instead of having two
> build systems like we have one (UNIX and Windows).
>
> Also, it would be much easier to maintain because Visual Studio projects
> are generated from a simple description, while right now if you want to
> change something you need to go through the hassle of defining it within
> the Visual Studio GUI.
>
> Consider for instance if you want to change the Windows build so that a
> builtin module is compiled as an external .pyd instead. Right now, you need
> to go through the hassle of manually defining a new project, setting all
> the include/libraries dependencies correctly, ecc. ecc. With CMake or a
> similar tool, it would be a matter of a couple of textual line changes.
>
> [ I'll also remember that "ease of maintanance for developers" is the #1
> reason for having a 2.1Mb python25.dll under Windows, which I would really
> love to reduce. ]

The cmake files for building python are now in a cvs repository:
http://www.cmake.org/cgi-bin/viewcvs.cgi/Utilities/CMakeBuildForPython/?root=ParaView3

This is inside the ParaView3 repository:
http://www.paraview.org/New/download.html

I used them today to build Python from svn trunk.

I'll add some documentation how to use them, how to get them and what works 
and what doesn't work tomorrow.

Bye
Alex

From martin at v.loewis.de  Thu Aug 30 22:32:34 2007
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 30 Aug 2007 22:32:34 +0200
Subject: [Python-Dev] Notification of assignment in new tracker
Message-ID: <46D72962.1020800@v.loewis.de>

Erik just fixed the tracker so that the Assigned-to person
gets email for any message added to an issue.

However, *no* email is sent if one merely changes the
Assigned-to field. Sending a message for every status change
would cause significant amounts of spam, so mere status
changes are performed "silently".

So if you assign an issue to somebody but yourself, you
should leave a message also so that this person actually
learns about the assignment. Doing so is probably a good
idea, anyway, as you should explain why you did this
assignment.

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Fri Aug 31 05:17:36 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 31 Aug 2007 15:17:36 +1200
Subject: [Python-Dev] Order of operations
In-Reply-To: <bb8868b90708300402x3670da30yb6152a0a93bd0f61@mail.gmail.com>
References: <e04bdf310708291032g68aa308fuce722e3fa1ffa493@mail.gmail.com>
	<46D5B855.7020101@v.loewis.de> <46D5CB49.5030902@scottdial.com>
	<46D5F254.9030406@v.loewis.de>
	<acd65fa20708291541m6f789352l425a33ebd9a596bb@mail.gmail.com>
	<fb545q$tpf$1@sea.gmane.org>
	<bb8868b90708300402x3670da30yb6152a0a93bd0f61@mail.gmail.com>
Message-ID: <46D78850.2000502@canterbury.ac.nz>

Jason Orendorff wrote:
> I think the weirdness comes from parsing -a/b as (-a)/b rather than
> -(a/b).

This will sort of be fixed in 3.0, at least for /, because
it will always mean float division, for which -(a/b) == (-a)/b.

You'll have to use // to get weirdness, then. :-)

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | Carpe post meridiem!          	  |
Christchurch, New Zealand	   | (I'm not a morning person.)          |
greg.ewing at canterbury.ac.nz	   +--------------------------------------+

From guido at python.org  Fri Aug 31 18:24:47 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 31 Aug 2007 09:24:47 -0700
Subject: [Python-Dev] Python 3.0a1 released!
Message-ID: <ca471dc20708310924r45d66728oe6a2a17a4a686b35@mail.gmail.com>

The release is available from http://python.org/download/releases/3.0/

I'll send a longer announcement to python-list and
python-announce-list. Please blog about this if you have a blog!

Thanks to all who helped out! It's been a great ride.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From janssen at parc.com  Fri Aug 31 19:47:18 2007
From: janssen at parc.com (Bill Janssen)
Date: Fri, 31 Aug 2007 10:47:18 PDT
Subject: [Python-Dev] the FakeSocket class in httplib.py -- remove it?
Message-ID: <07Aug31.104723pdt."57996"@synergy1.parc.xerox.com>

I'm wondering if it would be OK to remove the FakeSocket class in
httplib?  Actually, it's currently a function that returns one of its
arguments.  It's not documented as part of the module, and not used
elsewhere in the rest of the library.

Bill

From brett at python.org  Fri Aug 31 20:06:53 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 31 Aug 2007 11:06:53 -0700
Subject: [Python-Dev] the FakeSocket class in httplib.py -- remove it?
In-Reply-To: <-1982970389544522167@unknownmsgid>
References: <-1982970389544522167@unknownmsgid>
Message-ID: <bbaeab100708311106g181782e5l2fa1b9f93815e920@mail.gmail.com>

On 8/31/07, Bill Janssen <janssen at parc.com> wrote:
> I'm wondering if it would be OK to remove the FakeSocket class in
> httplib?  Actually, it's currently a function that returns one of its
> arguments.  It's not documented as part of the module, and not used
> elsewhere in the rest of the library.

Since I assume that includes unit tests, I have no issue seeing it go
away, especially if it is not being kept up-to-date.  But I am willing
to bet someone out there is using it so it might be best to add a
DeprecationWarning for 2.6 and then remove in 3.0/2.7.

-Brett

From janssen at parc.com  Fri Aug 31 20:50:51 2007
From: janssen at parc.com (Bill Janssen)
Date: Fri, 31 Aug 2007 11:50:51 PDT
Subject: [Python-Dev] the FakeSocket class in httplib.py -- remove it?
In-Reply-To: <bbaeab100708311106g181782e5l2fa1b9f93815e920@mail.gmail.com> 
References: <-1982970389544522167@unknownmsgid>
	<bbaeab100708311106g181782e5l2fa1b9f93815e920@mail.gmail.com>
Message-ID: <07Aug31.115053pdt."57996"@synergy1.parc.xerox.com>

Yep, that includes unit tests.  OK, I'll add the deprecation warning.

Bill

From db3l.net at gmail.com  Tue Aug 28 06:23:23 2007
From: db3l.net at gmail.com (David Bolen)
Date: Tue, 28 Aug 2007 00:23:23 -0400
Subject: [Python-Dev] Windows buildbot needed?
Message-ID: <9f94e2360708272123sed278c0s217b00da73afdc27@mail.gmail.com>

I was wondering if an additional Windows buildbot would be helpful, in
particular for py3k?  If so, I should be able to run one, and wanted to
request a buildbot account, as mentioned on the buildbot page on python.org.
I've got some idle hardware - nothing fancy (x86 P4, 2.8GHz) but it's up and
infrequently used, with a solid broadband connection.

I had some time to test, so set up a Windows XP SP2 virtual machine with all
the various pieces needed for a buildbot.  The host machine happens to run
XP too, but using a virtual machine was a bit cleaner (including easy
relocation over time if needed).  I've manually done full builds of Python
2.5.1 (from release25-maint), 2.6a0 (from trunk), and 3.0x (from py3k) using
the buildbot build and test batch files under tools, so I think the
environment is good.  2.5.1 tested clean, 2.6a0 had two errors in test_ssl
and test_winreg but they match some existing buildbots.  The 3.0 tests are
ugly, including a NULL pointer debug assertion triggered in test_os that
kills the test run, but I'm assuming that's just the current state of 3.0.
I've lurked on 3000-devel, but am not intimately up to date on expected
behavior under Windows.

Anyway, if it would be useful, I figured I could use this virtual machine
for a buildbot for at least trunk and 3k, perhaps with 2.5 too.  Not sure if
that requires multiple accounts, or if you configure each buildbot instance
with an SVN path (I didn't see anywhere obvious for the latter).

-- David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070828/7b4cb98d/attachment.htm