From tim.peters at gmail.com  Mon Nov  1 02:26:33 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Mon Nov  1 02:26:40 2004
Subject: [Python-Dev] Tests broken on Windows
In-Reply-To: <20041031195218.GA7395@surfboy>
References: <1f7befae0410311127b31675c@mail.gmail.com>
	<20041031195218.GA7395@surfboy>
Message-ID: <1f7befae04103117261e4d7f97@mail.gmail.com>

[Johannes Gijsbers]
> Hum, I didn't realize os.chmod() worked this differently on Windows.

Well, it can't work the same.  At the chmod() level, there are no
"user", "group" or "world"/"other" concepts on Windows.  There also
aren't distinct "execute"/"search", "write" and "read" bits.  There
are "hidden", "system", and "archive" bits on Windows, which have no
equivalents on Unix, and there's a "read only" bit on Windows, setting
which is kind of like turning off 0222 on Unix.  The only thing
chmod() is useful for on Windows is fiddling the state of the "read
only" bit.

> Seems like I bumped into bug #755617 (os module: Need a better description
> of "mode"). Per that bug, I googled for '_chmod msdn' and came up with the
> attached patch.  However, I don't have a Windows machine handy to test. Could
> someone with a Windows machine try the patch?

No cigar.  The patched test appears to assume that if you strip write
permission from a directory, that also makes it impossible to delete
files within the directory.  But directories on Windows aren't files
-- they're directories <wink>.

The attached patch allows the test to pass on WinXP, although I'm not
sure about Win95/98/ME.  If there are no objections, I'll check it in.
 I don't remember enough about Unix's equally obscure permission
gimmicks to be sure that this patch will leave the test working on
non-Windows boxes.
-------------- next part --------------
Index: Lib/test/test_shutil.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/test/test_shutil.py,v
retrieving revision 1.8
diff -u -r1.8 test_shutil.py
--- Lib/test/test_shutil.py	31 Oct 2004 12:05:31 -0000	1.8
+++ Lib/test/test_shutil.py	1 Nov 2004 01:10:32 -0000
@@ -1,4 +1,3 @@
-
 # Copyright (C) 2003 Python Software Foundation
 
 import unittest
@@ -20,21 +19,28 @@
         def test_on_error(self):
             self.errorState = 0
             os.mkdir(TESTFN)
-            f = open(os.path.join(TESTFN, 'a'), 'w')
+            self.childpath = os.path.join(TESTFN, 'a')
+            f = open(self.childpath, 'w')
             f.close()
-            # Make TESTFN unwritable.
-            os.chmod(TESTFN, stat.S_IRUSR)
+            old_dir_mode = os.stat(TESTFN).st_mode
+            old_child_mode = os.stat(self.childpath).st_mode
+            # Make unwritable.
+            os.chmod(self.childpath, stat.S_IREAD)
+            os.chmod(TESTFN, stat.S_IREAD)
 
             shutil.rmtree(TESTFN, onerror=self.check_args_to_onerror)
 
-            # Make TESTFN writable again.
-            os.chmod(TESTFN, stat.S_IRWXU)
+            # Make writable again.
+            os.chmod(TESTFN, old_dir_mode)
+            os.chmod(self.childpath, old_child_mode)
+
+            # Clean up.
             shutil.rmtree(TESTFN)
 
     def check_args_to_onerror(self, func, arg, exc):
         if self.errorState == 0:
             self.assertEqual(func, os.remove)
-            self.assertEqual(arg, os.path.join(TESTFN, 'a'))
+            self.assertEqual(arg, self.childpath)
             self.assertEqual(exc[0], OSError)
             self.errorState = 1
         else:
From jlg at dds.nl  Mon Nov  1 03:38:47 2004
From: jlg at dds.nl (Johannes Gijsbers)
Date: Mon Nov  1 03:36:47 2004
Subject: [Python-Dev] Tests broken on Windows
In-Reply-To: <1f7befae04103117261e4d7f97@mail.gmail.com>
References: <1f7befae0410311127b31675c@mail.gmail.com>
	<20041031195218.GA7395@surfboy>
	<1f7befae04103117261e4d7f97@mail.gmail.com>
Message-ID: <20041101023847.GA17801@surfboy>

On Sun, Oct 31, 2004 at 08:26:33PM -0500, Tim Peters wrote:
> The attached patch allows the test to pass on WinXP, although I'm not
> sure about Win95/98/ME.  If there are no objections, I'll check it in.
>  I don't remember enough about Unix's equally obscure permission
> gimmicks to be sure that this patch will leave the test working on
> non-Windows boxes.

It works on my Debian machine, so no objections from me.

Cheers,

Johannes
From python at rcn.com  Mon Nov  1 04:02:52 2004
From: python at rcn.com (Raymond Hettinger)
Date: Mon Nov  1 04:05:01 2004
Subject: [Python-Dev] Tests broken on Windows
In-Reply-To: <20041101023847.GA17801@surfboy>
Message-ID: <006601c4bfbf$478258c0$e841fea9@oemcomputer>

> On Sun, Oct 31, 2004 at 08:26:33PM -0500, Tim Peters wrote:
> > The attached patch allows the test to pass on WinXP, although I'm
not
> > sure about Win95/98/ME.  If there are no objections, I'll check it
in.
> >  I don't remember enough about Unix's equally obscure permission
> > gimmicks to be sure that this patch will leave the test working on
> > non-Windows boxes.
> 
> It works on my Debian machine, so no objections from me.

Run fine on WinME.

Am still getting a failure for test_traceback.py's test_bug737473.


Raymond

From tim.peters at gmail.com  Mon Nov  1 06:52:03 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Mon Nov  1 06:52:04 2004
Subject: [Python-Dev] Tests broken on Windows
In-Reply-To: <006601c4bfbf$478258c0$e841fea9@oemcomputer>
References: <20041101023847.GA17801@surfboy>
	<006601c4bfbf$478258c0$e841fea9@oemcomputer>
Message-ID: <1f7befae04103121526ff7a94c@mail.gmail.com>

[Raymond Hettinger]
> Run fine on WinME.
>
> Am still getting a failure for test_traceback.py's test_bug737473.

Fixed that already for WinXP.  No current failures in the -uall test
suite.  Is there a bug report for this ME problem?  Exactly how does
it fail?  Does it work if you force the test to take the time.sleep()
path instead of the os.utime() path?  Is posixmodule.c's posix_utime()
taking the utime() or _wutime() path on ME?  If the latter, does it
work if you force it to take the utime() path instead?  MS docs claim
utime() and _wutime() work "under Windows 98/Me and Windows
NT/2000/XP", but they could be lying:

<http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclib/html/_crt__utime.2c_._wutime.asp>

That link has a wonderful C example:  if you study the sample output,
it shows that the sample code they ran had no visible effect <wink>.
From python at rcn.com  Mon Nov  1 07:50:29 2004
From: python at rcn.com (Raymond Hettinger)
Date: Mon Nov  1 07:52:40 2004
Subject: [Python-Dev] Tests broken on Windows
In-Reply-To: <1f7befae04103121526ff7a94c@mail.gmail.com>
Message-ID: <007c01c4bfdf$138f7f00$e841fea9@oemcomputer>

> Fixed that already for WinXP.  No current failures in the -uall test
> suite.  Is there a bug report for this ME problem?  Exactly how does
> it fail?  

Traceback (most recent call last):
  File "test_traceback.py", line 79, in test_bug737473
    test_bug737473.test()
  File "c:\windows\temp\tmpsj7w5a\test_bug737473.py", line 2, in test
ValueError



> Does it work if you force the test to take the time.sleep()
> path instead of the os.utime() path?

Yes.



>  Is posixmodule.c's posix_utime()
> taking the utime() or _wutime() path on ME?  If the latter, does it
> work if you force it to take the utime() path instead?  MS docs claim
> utime() and _wutime() work "under Windows 98/Me and Windows
> NT/2000/XP", but they could be lying:
> 
> <http://msdn.microsoft.com/library/default.asp?url=/library/en-
> us/vclib/html/_crt__utime.2c_._wutime.asp>
> 
> That link has a wonderful C example:  if you study the sample output,
> it shows that the sample code they ran had no visible effect <wink>.


Am challenged for time in the next couple of days.  Will delve deeper at
the first opportunity.


For the time being, I think it would be best to use sleep() instead of
utime().  While it costs three seconds, at least we know it to be a
platform independent test.  It is rather late in the game to be
debugging OS specific problems introduced by a new check-in.



Raymond

From hyeshik at gmail.com  Mon Nov  1 09:19:24 2004
From: hyeshik at gmail.com (Hye-Shik Chang)
Date: Mon Nov  1 09:19:31 2004
Subject: [Python-Dev] Tests broken on Windows
In-Reply-To: <007c01c4bfdf$138f7f00$e841fea9@oemcomputer>
References: <1f7befae04103121526ff7a94c@mail.gmail.com>
	<007c01c4bfdf$138f7f00$e841fea9@oemcomputer>
Message-ID: <4f0b69dc04110100194b7335f4@mail.gmail.com>

On Mon, 1 Nov 2004 01:50:29 -0500, Raymond Hettinger <python@rcn.com> wrote:
> For the time being, I think it would be best to use sleep() instead of
> utime().  While it costs three seconds, at least we know it to be a
> platform independent test.  It is rather late in the game to be
> debugging OS specific problems introduced by a new check-in.

Okay.  I am convinced about using sleep() always.  And two seconds
will be enough to wait for new timestamp value. :)  Will check it
in right now.

Hye-Shik
From walter at livinglogic.de  Mon Nov  1 15:17:22 2004
From: walter at livinglogic.de (=?ISO-8859-15?Q?Walter_D=F6rwald?=)
Date: Mon Nov  1 15:17:25 2004
Subject: [Python-Dev] Code coverage tool updated
Message-ID: <41864572.4000806@livinglogic.de>

The Python code coverage tool available at
http://coverage.livinglogic.de/ has been updated. (It runs the
Python test suite once a month and makes the resulting code
coverage info available as a web application).

The backend is now Oracle instead of Postgres and is on a much
faster machine. The frontend has been replaced too, so the result
should be much faster and more flexible.

Bye,
    Walter D?rwald

From tim.peters at gmail.com  Mon Nov  1 15:40:55 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Mon Nov  1 15:40:58 2004
Subject: [Python-Dev] Tests broken on Windows
In-Reply-To: <007c01c4bfdf$138f7f00$e841fea9@oemcomputer>
References: <1f7befae04103121526ff7a94c@mail.gmail.com>
	<007c01c4bfdf$138f7f00$e841fea9@oemcomputer>
Message-ID: <1f7befae0411010640413d9148@mail.gmail.com>

>  Exactly how does it fail?

[Raymond Hettinger, on bug737473]
> Traceback (most recent call last):
>  File "test_traceback.py", line 79, in test_bug737473
>    test_bug737473.test()
>  File "c:\windows\temp\tmpsj7w5a\test_bug737473.py", line 2, in test
> ValueError

That helps.

>> Does it work if you force the test to take the time.sleep()
>> path instead of the os.utime() path?

> Yes.

Progress <wink>.

...

> Am challenged for time in the next couple of days.  Will delve deeper at
> the first opportunity.
>
> For the time being, I think it would be best to use sleep() instead of
> utime().  While it costs three seconds, at least we know it to be a
> platform independent test.  It is rather late in the game to be
> debugging OS specific problems introduced by a new check-in.

The checkin isn't that new anymore.  Just try changing

                past = time.time() - 3

to

                past = time.time() - 6

I added a note to <http://www.python.org/1057992> explaining why that
may matter on FAT or FAT32 systems.
From jlg at dds.nl  Mon Nov  1 15:40:27 2004
From: jlg at dds.nl (Johannes Gijsbers)
Date: Mon Nov  1 16:09:29 2004
Subject: [Python-Dev] Bug Day 4: this Sunday
Message-ID: <20041101144027.GA16108@surfboy>

The fourth Python Bug Day is coming up this Sunday, November 7 2004.

When?
=====

This Sunday, November 7, 2004. I'll be online at least from 11AM to 6PM CET
(10AM to 5PM GMT/ 6AM to 1PM EDT). That's a rather short time for the people in
the USian timezones, so I would like to see someone with CVS access from those
zones extend the bug day. Anyone interested?

Where?
======

* The #python-dev channel on irc.freenode.net.

* In real life, at the Amaze office:

  Korte Leidsedwarsstraat 12
  1017 RC Amsterdam
  tel: 020-4688336 

More information
================

For instructions and more information, see
    http://www.python.org/moin/PythonBugDay

Cheers,

Johannes
From mwh at python.net  Mon Nov  1 18:22:03 2004
From: mwh at python.net (Michael Hudson)
Date: Mon Nov  1 18:22:06 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <41864572.4000806@livinglogic.de> (
	=?iso-8859-1?q?Walter_D=F6rwald's_message_of?= "Mon,
	01 Nov 2004 15:17:22 +0100")
References: <41864572.4000806@livinglogic.de>
Message-ID: <2mu0s95vt0.fsf@starship.python.net>

Walter D?rwald <walter@livinglogic.de> writes:

> The Python code coverage tool available at
> http://coverage.livinglogic.de/ has been updated. (It runs the
> Python test suite once a month and makes the resulting code
> coverage info available as a web application).
>
> The backend is now Oracle instead of Postgres and is on a much
> faster machine. The frontend has been replaced too, so the result
> should be much faster and more flexible.

Using the back button in a browser seems to break it fairly badly --
if you click on a file that has some results, then click 'back' you
get a java.lang.NullPointerException!  If you click on a file
_without_ results and click 'back' you merely get a message in German
that I don't understand...

Cheers,
mwh


-- 
  Programming languages should be designed not by piling feature on
  top of feature, but by removing the weaknesses and restrictions
  that make the additional features appear necessary.
               -- Revised(5) Report on the Algorithmic Language Scheme
From skip at pobox.com  Mon Nov  1 23:01:56 2004
From: skip at pobox.com (Skip Montanaro)
Date: Mon Nov  1 23:02:20 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <41864572.4000806@livinglogic.de>
References: <41864572.4000806@livinglogic.de>
Message-ID: <16774.45652.835103.327656@montanaro.dyndns.org>


    Walter> The Python code coverage tool available at
    Walter> http://coverage.livinglogic.de/ has been updated. (It runs the
    Walter> Python test suite once a month and makes the resulting code
    Walter> coverage info available as a web application).

This looks very interesting, but the results seem a bit mystifying.  For
example, it says that Lib/Queue.py has 169 lines, none of which are
coverable.  It also shows that no lines were covered, which seems odd
considering the fact the distribution does have a test_queue.py test module.

Skip

From satyarth.negi at gmail.com  Tue Nov  2 10:13:12 2004
From: satyarth.negi at gmail.com (Satyarth Negi)
Date: Tue Nov  2 10:13:16 2004
Subject: [Python-Dev] Help Needed ..Urgent Plzz
Message-ID: <546a6a000411020113195cb03f@mail.gmail.com>

Hello , I have a problem with a peice of code :

.........
self.child = pexpect.spawn('ssh -l ' + user +' ' + host)
self.child.setmaxread (1000)
self.child.expect('[pP]assword:', 30)
self.child.sendline(password)
........

When i run the script it gives me following error :
" Exception exceptions.OSError: (10, 'No child processes') in <bound
method spawn.__del__ of <pexpect.spawn instance at 0x403d938c>>
ignored"

Then I added , self.child.close(False) before 3rd line , but even that
is not working .
 

Please help , im totaly stuck .
Thanx in anticipation .
Regards
Satyarth Negi
India
From walter at livinglogic.de  Tue Nov  2 12:58:21 2004
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Tue Nov  2 12:58:25 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <16774.45652.835103.327656@montanaro.dyndns.org>
References: <41864572.4000806@livinglogic.de>
	<16774.45652.835103.327656@montanaro.dyndns.org>
Message-ID: <4187765D.3000705@livinglogic.de>

Skip Montanaro wrote:

>     Walter> The Python code coverage tool available at
>     Walter> http://coverage.livinglogic.de/ has been updated. (It runs the
>     Walter> Python test suite once a month and makes the resulting code
>     Walter> coverage info available as a web application).
> 
> This looks very interesting, but the results seem a bit mystifying.  For
> example, it says that Lib/Queue.py has 169 lines, none of which are
> coverable.  It also shows that no lines were covered, which seems odd
> considering the fact the distribution does have a test_queue.py test module.

The same happened to string.py. The run from yesterday *does* include
string.py as a covered file, but only two lines were covered, which is
even stranger.

Maybe the problem is the fact, that I'm using an old version of
trace.py? I'll updated trace.py and rerun the job.

Bye,
    Walter D?rwald


From walter at livinglogic.de  Tue Nov  2 12:58:35 2004
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Tue Nov  2 12:58:38 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <2mu0s95vt0.fsf@starship.python.net>
References: <41864572.4000806@livinglogic.de>
	<2mu0s95vt0.fsf@starship.python.net>
Message-ID: <4187766B.7060407@livinglogic.de>

Michael Hudson wrote:
> Walter D?rwald <walter@livinglogic.de> writes:
> 
> 
>>The Python code coverage tool available at
>>http://coverage.livinglogic.de/ has been updated. (It runs the
>>Python test suite once a month and makes the resulting code
>>coverage info available as a web application).
>>
>>The backend is now Oracle instead of Postgres and is on a much
>>faster machine. The frontend has been replaced too, so the result
>>should be much faster and more flexible.
> 
> Using the back button in a browser seems to break it fairly badly --
> if you click on a file that has some results, then click 'back' you
> get a java.lang.NullPointerException!

I can reproduce that. I've forwarded this to the developer (who is
currently on vacation and will be back in December).

> If you click on a file
> _without_ results and click 'back' you merely get a message in German
> that I don't understand...

This message basically says "Don't click the back button!".

Bye,
    Walter D?rwald

From FBatista at uniFON.com.ar  Tue Nov  2 13:35:49 2004
From: FBatista at uniFON.com.ar (Batista, Facundo)
Date: Tue Nov  2 13:37:26 2004
Subject: [Python-Dev] Help Needed ..Urgent Plzz
Message-ID: <A128D751272CD411BC9200508BC2194D053C7B40@escpl.tcp.com.ar>

[Satyarth Negi]

#- Hello , I have a problem with a peice of code :
#- 
#- .........
#- ........

Satyarth, this list is for development of Python itself, not for development
of programs in Python.

This post should go to the standard python newsgroup, comp.lang.python,
which can be accessed as a news or as a mailing list (for further reference
and ways to subscribe, check http://www.python.org/community/lists.html).

.	Facundo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20041102/3c69cf07/attachment.html
From anthony at interlink.com.au  Tue Nov  2 13:46:38 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Tue Nov  2 13:46:16 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
Message-ID: <418781AE.70303@interlink.com.au>

The trunk should be considered frozen for the 2.4b2 release from 0000
UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd 
November from 1900 US Eastern time, and 1600 US Pacific time. Or,
to put it another way, in about 11 hours from when I'm sending this
message.

As usual, if you're not in the set (Martin, Fred, Anthony) please hold
off on any checkins until the b2 release is done.

Thanks,
Anthony


-- 
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.
From anthony at interlink.com.au  Tue Nov  2 13:48:49 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Tue Nov  2 13:49:22 2004
Subject: [Python-Dev] Bug Day 4: this Sunday
In-Reply-To: <20041101144027.GA16108@surfboy>
References: <20041101144027.GA16108@surfboy>
Message-ID: <41878231.9060001@interlink.com.au>

Johannes Gijsbers wrote:
> This Sunday, November 7, 2004. I'll be online at least from 11AM to 6PM CET
> (10AM to 5PM GMT/ 6AM to 1PM EDT). That's a rather short time for the people in
> the USian timezones, so I would like to see someone with CVS access from those
> zones extend the bug day. Anyone interested?

Could I request that people are _really_ careful about checking in bugs
that might lead to breakage, as this will be between 2.4b2, and what I
_hope_ will be 2.4rc1.

Thanks,
Anthony

-- 
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.
From walter at livinglogic.de  Tue Nov  2 18:55:40 2004
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Tue Nov  2 18:55:44 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <4187765D.3000705@livinglogic.de>
References: <41864572.4000806@livinglogic.de>	<16774.45652.835103.327656@montanaro.dyndns.org>
	<4187765D.3000705@livinglogic.de>
Message-ID: <4187CA1C.9050508@livinglogic.de>

Walter D?rwald wrote:

> Skip Montanaro wrote:
> 
>>     Walter> The Python code coverage tool available at
>>     Walter> http://coverage.livinglogic.de/ has been updated. (It runs 
>> the
>>     Walter> Python test suite once a month and makes the resulting code
>>     Walter> coverage info available as a web application).
>>
>> This looks very interesting, but the results seem a bit mystifying.  For
>> example, it says that Lib/Queue.py has 169 lines, none of which are
>> coverable.  It also shows that no lines were covered, which seems odd
>> considering the fact the distribution does have a test_queue.py test 
>> module.
> 
> 
> The same happened to string.py. The run from yesterday *does* include
> string.py as a covered file, but only two lines were covered, which is
> even stranger.
> 
> Maybe the problem is the fact, that I'm using an old version of
> trace.py? I'll updated trace.py and rerun the job.

This doesn't help. The following assert in trace.py raises
an AssertionError:
    assert filename.endswith('.py')

I've added print statements to find_executable_linenos()
and got the following:

[...]
/home/coverage/LivingLogic/Python/PythonCodeCoverage/jobs/python/dist/src/Lib/sre_compile.py
/home/coverage/LivingLogic/Python/PythonCodeCoverage/jobs/python/dist/src/Lib/test/__init__.py
<doctest _threading_local[7]>
Traceback (most recent call last):
   File "../../../trace.py", line 790, in ?
     main()
   File "../../../trace.py", line 787, in main
     results.write_results(missing, summary=summary, coverdir=coverdir)
   File "../../../trace.py", line 301, in write_results
     lnotab = find_executable_linenos(filename)
   File "../../../trace.py", line 420, in find_executable_linenos
     assert filename.endswith('.py')
AssertionError

So what is <doctest _threading_local[7]>?

Bye,
    Walter D?rwald



From tim.peters at gmail.com  Tue Nov  2 19:53:59 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Tue Nov  2 19:54:12 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <4187CA1C.9050508@livinglogic.de>
References: <41864572.4000806@livinglogic.de>
	<16774.45652.835103.327656@montanaro.dyndns.org>
	<4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de>
Message-ID: <1f7befae04110210537c35e514@mail.gmail.com>

[Walter D?rwald]
> This doesn't help. The following assert in trace.py raises
> an AssertionError:
>    assert filename.endswith('.py')
> 
> I've added print statements to find_executable_linenos()
> and got the following:
> 
> [...]
> /home/coverage/LivingLogic/Python/PythonCodeCoverage/jobs/python/dist/src/Lib/sre_compile.py
> /home/coverage/LivingLogic/Python/PythonCodeCoverage/jobs/python/dist/src/Lib/test/__init__.py
> <doctest _threading_local[7]>
> Traceback (most recent call last):
>   File "../../../trace.py", line 790, in ?
>     main()
>   File "../../../trace.py", line 787, in main
>     results.write_results(missing, summary=summary, coverdir=coverdir)
>   File "../../../trace.py", line 301, in write_results
>     lnotab = find_executable_linenos(filename)
>   File "../../../trace.py", line 420, in find_executable_linenos
>     assert filename.endswith('.py')
> AssertionError
> 
> So what is <doctest _threading_local[7]>?

I haven't followed this thread, but can answer that question literally
<wink>:  it's code synthesized for the seventh doctest example in
Lib/_threading_local.py's module docstring.  The file name is
constructed by this line in doctest.py:

            # Use a special filename for compile(), so we can retrieve
            # the source code during interactive debugging (see
            # __patched_linecache_getlines).
            filename = '<doctest %s[%d]>' % (test.name, examplenum)
From bac at OCF.Berkeley.EDU  Tue Nov  2 21:24:38 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Tue Nov  2 21:25:01 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <418781AE.70303@interlink.com.au>
References: <418781AE.70303@interlink.com.au>
Message-ID: <4187ED06.70104@ocf.berkeley.edu>

Anthony Baxter wrote:
> The trunk should be considered frozen for the 2.4b2 release from 0000
> UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd 
> November from 1900 US Eastern time, and 1600 US Pacific time. Or,
> to put it another way, in about 11 hours from when I'm sending this
> message.
> 

Is a deprecation of a function okay in the Mac C API?  Specifically, 
PyMac_GetAppletScriptFile() is not being used by any known code and is a 
problem for http://www.python.org/sf/1035255 (removes dependency of compiling 
against some frameworks for the core on OS X).  Since I didn't feel totally 
safe applying the patch for b2 since it changed compilation and I didn't get in 
early enough to get testing from CVS users I was planning on deprecating 
PyMac_GetAppletScriptFile() for 2.4 and removing it in 2.5 so I could apply the 
patch then.

But it is a feature change so I wanted to get clearance first.  As I said, to 
the best of my and Bob Ippolito's knowledge no one is using the function and it 
is a hinderance in removing unneeded linking against the core on OS X.

Sorry I didn't do this sooner but I didn't think about deprecating the thing 
until late last night.

-Brett
From bob at redivi.com  Tue Nov  2 21:46:19 2004
From: bob at redivi.com (Bob Ippolito)
Date: Tue Nov  2 21:46:33 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <4187ED06.70104@ocf.berkeley.edu>
References: <418781AE.70303@interlink.com.au> <4187ED06.70104@ocf.berkeley.edu>
Message-ID: <3FEE83B6-2D10-11D9-955F-000A95BA5446@redivi.com>


On Nov 2, 2004, at 15:24, Brett C. wrote:

> Anthony Baxter wrote:
>> The trunk should be considered frozen for the 2.4b2 release from 0000
>> UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd 
>> November from 1900 US Eastern time, and 1600 US Pacific time. Or,
>> to put it another way, in about 11 hours from when I'm sending this
>> message.
>
> Is a deprecation of a function okay in the Mac C API?  Specifically, 
> PyMac_GetAppletScriptFile() is not being used by any known code and is 
> a problem for http://www.python.org/sf/1035255 (removes dependency of 
> compiling against some frameworks for the core on OS X).  Since I 
> didn't feel totally safe applying the patch for b2 since it changed 
> compilation and I didn't get in early enough to get testing from CVS 
> users I was planning on deprecating PyMac_GetAppletScriptFile() for 
> 2.4 and removing it in 2.5 so I could apply the patch then.
>
> But it is a feature change so I wanted to get clearance first.  As I 
> said, to the best of my and Bob Ippolito's knowledge no one is using 
> the function and it is a hinderance in removing unneeded linking 
> against the core on OS X.
>
> Sorry I didn't do this sooner but I didn't think about deprecating the 
> thing until late last night.

If it's absolutely necessary to keep this undocumented function that 
*nobody* uses in the API, I'll write a version that delegates the 
functionality to an extension module that has business linking to said 
frameworks.  I really want to see this change for 2.4.

-bob

From anthony at interlink.com.au  Wed Nov  3 03:34:39 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed Nov  3 03:34:14 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <4187ED06.70104@ocf.berkeley.edu>
References: <418781AE.70303@interlink.com.au> <4187ED06.70104@ocf.berkeley.edu>
Message-ID: <418843BF.7020107@interlink.com.au>

Brett C. wrote:
> Anthony Baxter wrote:
> 
>> The trunk should be considered frozen for the 2.4b2 release from 0000
>> UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd 
>> November from 1900 US Eastern time, and 1600 US Pacific time. Or,
>> to put it another way, in about 11 hours from when I'm sending this
>> message.
>>
> 
> Is a deprecation of a function okay in the Mac C API?  Specifically, 
> PyMac_GetAppletScriptFile() is not being used by any known code and is a 
> problem for http://www.python.org/sf/1035255 (removes dependency of 
> compiling against some frameworks for the core on OS X).  Since I didn't 
> feel totally safe applying the patch for b2 since it changed compilation 
> and I didn't get in early enough to get testing from CVS users I was 
> planning on deprecating PyMac_GetAppletScriptFile() for 2.4 and removing 
> it in 2.5 so I could apply the patch then.

It sounds like it's fine to put in once the trunk is unfrozen.



-- 
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.
From bac at OCF.Berkeley.EDU  Wed Nov  3 04:43:24 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Wed Nov  3 04:43:56 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <418843BF.7020107@interlink.com.au>
References: <418781AE.70303@interlink.com.au> <4187ED06.70104@ocf.berkeley.edu>
	<418843BF.7020107@interlink.com.au>
Message-ID: <418853DC.10501@ocf.berkeley.edu>

Anthony Baxter wrote:
> Brett C. wrote:
> 
>> Anthony Baxter wrote:
>>
>>> The trunk should be considered frozen for the 2.4b2 release from 0000
>>> UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd 
>>> November from 1900 US Eastern time, and 1600 US Pacific time. Or,
>>> to put it another way, in about 11 hours from when I'm sending this
>>> message.
>>>
>>
>> Is a deprecation of a function okay in the Mac C API?  Specifically, 
>> PyMac_GetAppletScriptFile() is not being used by any known code and is 
>> a problem for http://www.python.org/sf/1035255 (removes dependency of 
>> compiling against some frameworks for the core on OS X).  Since I 
>> didn't feel totally safe applying the patch for b2 since it changed 
>> compilation and I didn't get in early enough to get testing from CVS 
>> users I was planning on deprecating PyMac_GetAppletScriptFile() for 
>> 2.4 and removing it in 2.5 so I could apply the patch then.
> 
> 
> It sounds like it's fine to put in once the trunk is unfrozen.
> 

OK, great.  I will add the deprecation warning once the trunk is unfrozen and 
plan on applying the full patch for removing the compilation once 2.5 is open 
for work.

Sorry I couldn't get my act together fast enough to get this in for 2.4, Bob.

-Brett
From tim.peters at gmail.com  Wed Nov  3 05:40:05 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Wed Nov  3 05:40:10 2004
Subject: [Python-Dev] weakref gc semantics
Message-ID: <1f7befae04110220402077a6fa@mail.gmail.com>

This example bothers me a little, under current Python CVS:

"""
import gc, weakref

def C_gone(ignored):
    print "An object of type C went away."

class C:
    def __init__(self):
        self.wr = weakref.ref(self, C_gone)

print "creating non-cyclic C instance"
c = C()

print "creating cyclic C instance and destroying old one"
c = C()  # triggers the print
c.loop = c

print "getting rid of the cyclic one"
c = None  # doesn't trigger the print
gc.collect()  # ditto
 """

Output:

    creating non-cyclic C instance
    creating cyclic C instance and destroying old one
    An object of type C went away.
    getting rid of the cyclic one

An instance c of C gets a strong reference (self.wr) to a weak
reference W to itself.  At the end, c is in a cycle, but W isn't.  W
ends up reachable only *from* stuff in a cycle (namely c).

gc suppresses the callback then because W is unreachable, not really
because W is in a cycle.  If self.wr had a __del__ method instead, it
would have been executed.  So it bothers me a little that the callback
isn't:  a reachable callback can have visible effects, and there
really isn't ambiguity about which of c and W "dies first" (c does) in
the example.

The question is whether gc should really be looking at whether the
weakref *callback* is reachable, regardless of whether the weakref
itself is reachable (if the weakref is reachable, its callback is too,
and 2.4b2 invokes it -- the only case in question is the one in the
example, where the weakref and the weakref's referent are unreachable
but the weakref's callback is reachable).

Pythons 2.2.3 and 2.3.4 produce the same output, so no way do I want
to change Python 2.4 at this late stage.  It's a question for 2.5.  It
gets more complicated than the above, of course.  For example, we can
end up with weakrefs *in* cycles, with reachable callbacks, and then
there's no good way to decide which callback to invoke first.  That's
the same ordering problem we have when objects with __del__ methods
are in trash cycles.  It's not intractable, though, because gc invokes
weakref callbacks before tp_clear has been invoked on anything, and
reachable weakref callbacks can't resurrect any trash (anything they
can access is, by definition of "reachable", not trash).

A hidden agenda is that I like weakref callbacks a lot more than
__del__ methods, and would like to see them strong enough so that
Python 3 could leave __del__ out.  Planting a weakref to self *in*
self (as in the example at the top) is one way to approach
critical-resource cleanup in a __del__-free world (a weakref subclass
could remember the critical resource when an instance was constructed,
and the callback could access the critical resource to clean it up --
but, as the example shows, that doesn't work now if the containing
instance ends up in cyclic trash; but there are other ways that do
work).
From just at letterror.com  Wed Nov  3 08:38:54 2004
From: just at letterror.com (Just van Rossum)
Date: Wed Nov  3 08:39:39 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <418853DC.10501@ocf.berkeley.edu>
Message-ID: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>

Brett C. wrote:

> OK, great.  I will add the deprecation warning once the trunk is
> unfrozen and plan on applying the full patch for removing the
> compilation once 2.5 is open for work.
> 
> Sorry I couldn't get my act together fast enough to get this in for
> 2.4, Bob.

I don't get it. This is about an undocumented Mac-specific API function
that noone uses (not even Python itself) which was introduced in 2.3. It
just needs to be ripped out. I don't mind if it's done after b2, but I
agree with Bob that it needs to be taken out before 2.4 final. Jack,
would you agree?

Just
From anthony at interlink.com.au  Wed Nov  3 08:49:34 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed Nov  3 08:50:14 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>
References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>
Message-ID: <41888D8E.5050705@interlink.com.au>

Just van Rossum wrote:
> I don't get it. This is about an undocumented Mac-specific API function
> that noone uses (not even Python itself) which was introduced in 2.3. It
> just needs to be ripped out. I don't mind if it's done after b2, but I
> agree with Bob that it needs to be taken out before 2.4 final. Jack,
> would you agree?

FWIW, I don't mind having this done either way...

From anthony at interlink.com.au  Wed Nov  3 12:16:49 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed Nov  3 13:19:52 2004
Subject: [Python-Dev] interlocking dependencies on the path to a release
Message-ID: <4188BE21.6040205@interlink.com.au>

So I've been documenting the current state of the release process,
and it's a bit messier than I'd like - particularly with people
scattered across timezones. Here's the list of things that need
to happen, and what depends on each.

1. Include/patchlevel.h
2. Doc build (depends on 1.)
3. Misc/NEWS, Lib/idlelib/NEWS.txt, Lib/idlelib/idlever.py
4. PCbuild/BUILDno.txt, pythoncore.dsp
5. Tag the tree (depends on 1, 3, 5)
6. export/build the tarballs (depends on 5)
7. Build the windows release (depends on 2, 5)
8. Update the webpages, sign the files (depends on 6, 7)
9. Send the email (depends on 8)


Stage 2 is a Fred thing, stages 4 and 7 are Martin, most of the rest
are things I do, although Fred will often do 1 because he needs it
to do 2, and the timezone thing makes it easier for him to just do
it.

(*) The names 'Fred', 'Martin', and 'I' are just the names of the
current set of release people - feel free to substitute roles in
there instead.

I'm not really going anywhere with this yet, but I'm curious
if anyone can see any paths that might lead to a simpler result?
It's particularly bad at the moment, with me being UTC+11, Martin
UTC+1, and Fred UTC-4 (I might be off by an hour or two for Fred
and Martin).
From bac at OCF.Berkeley.EDU  Wed Nov  3 23:57:26 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Wed Nov  3 23:57:57 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <41888D8E.5050705@interlink.com.au>
References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>
	<41888D8E.5050705@interlink.com.au>
Message-ID: <41896256.1040104@ocf.berkeley.edu>

Anthony Baxter wrote:
> Just van Rossum wrote:
> 
>> I don't get it. This is about an undocumented Mac-specific API function
>> that noone uses (not even Python itself) which was introduced in 2.3. It
>> just needs to be ripped out. I don't mind if it's done after b2, but I
>> agree with Bob that it needs to be taken out before 2.4 final. Jack,
>> would you agree?
> 

I think Bob also wanted to get the whole patch in that removed compiling the 
core against CoreFoundation and CoreServices.  I am not comfortable doing that 
without more lead into a beta than this so I am holding off until 2.5 for that.

> 
> FWIW, I don't mind having this done either way...
> 

OK, then I will just rip it out and skip the deprecation warning.

-Brett
From bac at OCF.Berkeley.EDU  Thu Nov  4 00:13:34 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Thu Nov  4 00:13:54 2004
Subject: [Python-Dev] interlocking dependencies on the path to a release
In-Reply-To: <4188BE21.6040205@interlink.com.au>
References: <4188BE21.6040205@interlink.com.au>
Message-ID: <4189661E.7040002@ocf.berkeley.edu>

Anthony Baxter wrote:
> So I've been documenting the current state of the release process,
> and it's a bit messier than I'd like - particularly with people
> scattered across timezones. Here's the list of things that need
> to happen, and what depends on each.
> 
> 1. Include/patchlevel.h
> 2. Doc build (depends on 1.)
> 3. Misc/NEWS, Lib/idlelib/NEWS.txt, Lib/idlelib/idlever.py
> 4. PCbuild/BUILDno.txt, pythoncore.dsp
> 5. Tag the tree (depends on 1, 3, 5)
> 6. export/build the tarballs (depends on 5)
> 7. Build the windows release (depends on 2, 5)
> 8. Update the webpages, sign the files (depends on 6, 7)
> 9. Send the email (depends on 8)
> 

The only thing I can see possibly making things easier is to script stuff. 
Steps 1, 3, and 4 could all be automated.  And if generating the HTML for the 
reST text is a slight pain because of errors in the syntax we could try to 
check that more often or even designate that someone keeps random track that 
the syntax is good.  Otherwise we could probably try to have some build script 
on creosote (web server for python.org) that tries to generate it nightly to 
catch errors early.

Best solution I can think of is somehow come up with a way to remove the need 
for Martin to do 4 somehow.  If that happens then Fred can do 1-2, Anthony can 
do 3-6, Martin does 7, and then Anthony finished up with 8-10.  Without Martin 
doing 7-10 I don't see a good way to rework this.

-Brett
From bob at redivi.com  Thu Nov  4 00:14:08 2004
From: bob at redivi.com (Bob Ippolito)
Date: Thu Nov  4 00:14:18 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <41896256.1040104@ocf.berkeley.edu>
References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>
	<41888D8E.5050705@interlink.com.au>
	<41896256.1040104@ocf.berkeley.edu>
Message-ID: <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com>


On Nov 3, 2004, at 17:57, Brett C. wrote:

> Anthony Baxter wrote:
>> Just van Rossum wrote:
>>> I don't get it. This is about an undocumented Mac-specific API 
>>> function
>>> that noone uses (not even Python itself) which was introduced in 
>>> 2.3. It
>>> just needs to be ripped out. I don't mind if it's done after b2, but 
>>> I
>>> agree with Bob that it needs to be taken out before 2.4 final. Jack,
>>> would you agree?
>
> I think Bob also wanted to get the whole patch in that removed 
> compiling the core against CoreFoundation and CoreServices.  I am not 
> comfortable doing that without more lead into a beta than this so I am 
> holding off until 2.5 for that.

Argh!  It's nothing crazy, it removes incorrect linker flags and 
delegates two Mac-specific API functions to where they belong.

-bob

From martin at v.loewis.de  Thu Nov  4 00:17:59 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu Nov  4 00:17:53 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path to a release
In-Reply-To: <4188BE21.6040205@interlink.com.au>
References: <4188BE21.6040205@interlink.com.au>
Message-ID: <41896727.7030206@v.loewis.de>

Anthony Baxter wrote:
> 1. Include/patchlevel.h
> 2. Doc build (depends on 1.)
> 3. Misc/NEWS, Lib/idlelib/NEWS.txt, Lib/idlelib/idlever.py
> 4. PCbuild/BUILDno.txt, pythoncore.dsp
> 5. Tag the tree (depends on 1, 3, 5)
> 6. export/build the tarballs (depends on 5)
> 7. Build the windows release (depends on 2, 5)
> 8. Update the webpages, sign the files (depends on 6, 7)
> 9. Send the email (depends on 8)
> 
> 
> Stage 2 is a Fred thing, stages 4 and 7 are Martin, most of the rest
> are things I do, although Fred will often do 1 because he needs it
> to do 2, and the timezone thing makes it easier for him to just do
> it.

It would be possible to automate the bumps to version numbers, and
you don't have to be on Windows to do 4, anymore (as the project
file is now a fairly stable XML file).

Also, it would be possible to bump version numbers immediately after
a snapshot (alpha, beta) release, rather than just before. I tend
to forget that I have to do 4, and remember at earliest when you
announce the freeze.

> (*) The names 'Fred', 'Martin', and 'I' are just the names of the
> current set of release people - feel free to substitute roles in
> there instead.

I think it should be possible to come up with a release process that
combines Fred's and my role, wrt. documentation creation (I create
the chm file); theoretically, whoever builds the documentation should
be able to do so on Windows as well.

Also, I think it would be good to eliminate Fred's role altogether:
the documentation build process should be much more automatic, and
less relying on third-party tools. I think aiming at rewriting ht2html
in Python for 2.5 would be a worthwhile goal (I have dropped the
idea that conversion to xml would be worthwhile).

Then, the release manager could trivially build the documentation.
For that to work, it is necessary that the documentation builds out
of the box; for that to happen, it is necessary that a broad audience
can build the documentation, so that problems are detected long
before the release (i.e. short after a checkin - or better yet,
before). For the PostScript/PDF release, one certainly needs TeX,
but that is free software.

The same could be said about the Windows release, except that we
stand little chance of ever dropping the requirement for special,
non-free tools (Windows, MSVC). However, anybody possessing these
tools and owning a sufficiently powerful machine should be able
to release Python.

Regards,
Martin
From martin at v.loewis.de  Thu Nov  4 00:20:22 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu Nov  4 00:20:16 2004
Subject: [Python-Dev] interlocking dependencies on the path to a release
In-Reply-To: <4189661E.7040002@ocf.berkeley.edu>
References: <4188BE21.6040205@interlink.com.au>
	<4189661E.7040002@ocf.berkeley.edu>
Message-ID: <418967B6.4040106@v.loewis.de>

Brett C. wrote:
> Best solution I can think of is somehow come up with a way to remove the 
> need for Martin to do 4 somehow.  

This shouldn't be that bad as it currently is. *Anybody* could actually
do this - there is no need to do it within the release process (except
perhaps during a1). I don't see a problem with the CVS reporting numbers
that wait for a release a few weeks.

Regards,
Martin
From martin at v.loewis.de  Thu Nov  4 00:26:59 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu Nov  4 00:26:55 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com>
References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>	<41888D8E.5050705@interlink.com.au>	<41896256.1040104@ocf.berkeley.edu>
	<103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com>
Message-ID: <41896943.9000906@v.loewis.de>

Bob Ippolito wrote:
> Argh!  It's nothing crazy, it removes incorrect linker flags and 
> delegates two Mac-specific API functions to where they belong.

I think Brett's thinking is that if this was a serious problem,
it should have been fixed months ago. The fact that nobody bothered
pushing that change before 2.4b1 indicates that the problem is not
serious. As an incompatible change that fixes a non-serious problem,
it shouldn't be applied before 2.5, in accordance with PEP 5
(which says that an incompatible change must see a deprecation first).

That said, the release manager has said what he thinks about the
issue, so anybody with commit privileges please do what they think
is needed (I personally don't care much enough beyond writing this
message).

Regards,
Martin
From anthony at python.org  Thu Nov  4 00:50:50 2004
From: anthony at python.org (Anthony Baxter)
Date: Thu Nov  4 00:46:30 2004
Subject: [Python-Dev] RELEASED Python 2.4, beta 2
Message-ID: <41896EDA.5020805@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On behalf of the Python development team and the Python community, I'm
happy to announce the second beta of Python 2.4.

Python 2.4b2 is a beta release. We'd greatly appreciate it if you could
download it, kick the tires and let us know of any problems you find,
but it is probably not suitable for production usage.

~    http://www.python.org/2.4

Notable changes in beta2 include another bug fix in the intersection
of weakrefs and the garbage collector, time.strptime() correctly
handling %U and %W arguments, and a fix for a problem of old source code
being cached and shown in tracebacks. There's a number of other bugfixes
as well, see either the highlights, the What's New in Python 2.4, or the
detailed NEWS file -- all available from the Python 2.4 webpage.

This is the second beta release of Python 2.4. Python 2.4 is now in the
beta part of the release cycle - there should be few or no new features,
merely bug fixes. From here, we plan to have a release candidate in
a couple of weeks, and a final release at the start of December - but
obviously this plan is subject to change, based on any problems that
might crop up.

Please log any problems you have with this release in the SourceForge
bug tracker (noting that you're using 2.4b2):

~    http://sourceforge.net/bugs/?group_id=5470

Enjoy the new release,
Anthony

Anthony Baxter
anthony@python.org
Python Release Manager
(on behalf of the entire python-dev team)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBiW7YDt3F8mpFyBYRAq7DAKCskVfPO4LEQlC3zijupOnd9snCWACgpMHX
bH7xepODsyo1115TMQ9CQkw=
=0UOq
-----END PGP SIGNATURE-----
From anthony at interlink.com.au  Thu Nov  4 00:52:31 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu Nov  4 00:48:14 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com>
References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>
	<41888D8E.5050705@interlink.com.au>
	<41896256.1040104@ocf.berkeley.edu>
	<103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com>
Message-ID: <41896F3F.8090803@interlink.com.au>

Bob Ippolito wrote:
> Argh!  It's nothing crazy, it removes incorrect linker flags and 
> delegates two Mac-specific API functions to where they belong.

I asked this question on #macpython, but didn't get a response - I
understand the costs for removing this are slight, but I'm not sure
what the benefits of not linking the frameworks in are? Anyone?
From bob at redivi.com  Thu Nov  4 01:03:56 2004
From: bob at redivi.com (Bob Ippolito)
Date: Thu Nov  4 01:04:03 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <41896943.9000906@v.loewis.de>
References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>	<41888D8E.5050705@interlink.com.au>	<41896256.1040104@ocf.berkeley.edu>
	<103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com>
	<41896943.9000906@v.loewis.de>
Message-ID: <051E8585-2DF5-11D9-8F1A-000A95BA5446@redivi.com>


On Nov 3, 2004, at 18:26, Martin v. L?wis wrote:

> Bob Ippolito wrote:
>> Argh!  It's nothing crazy, it removes incorrect linker flags and 
>> delegates two Mac-specific API functions to where they belong.
>
> I think Brett's thinking is that if this was a serious problem,
> it should have been fixed months ago. The fact that nobody bothered
> pushing that change before 2.4b1 indicates that the problem is not
> serious. As an incompatible change that fixes a non-serious problem,
> it shouldn't be applied before 2.5, in accordance with PEP 5
> (which says that an incompatible change must see a deprecation first).

The thing is that the incompatible change is going in, but the 
compatible changes (the reason for the incompatible change) are not.  
This is very peculiar.

-bob

From bob at redivi.com  Thu Nov  4 01:14:31 2004
From: bob at redivi.com (Bob Ippolito)
Date: Thu Nov  4 01:14:40 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <41896F3F.8090803@interlink.com.au>
References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>
	<41888D8E.5050705@interlink.com.au>
	<41896256.1040104@ocf.berkeley.edu>
	<103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com>
	<41896F3F.8090803@interlink.com.au>
Message-ID: <80269A7A-2DF6-11D9-8F1A-000A95BA5446@redivi.com>

On Nov 3, 2004, at 18:52, Anthony Baxter wrote:

> Bob Ippolito wrote:
>> Argh!  It's nothing crazy, it removes incorrect linker flags and 
>> delegates two Mac-specific API functions to where they belong.
>
> I asked this question on #macpython, but didn't get a response - I
> understand the costs for removing this are slight, but I'm not sure
> what the benefits of not linking the frameworks in are? Anyone?

I don't see your question in my backlog of #macpython, my client must 
have been disconnected at that time?

It should shrink the memory footprint of Python and cause it to start 
up a bit faster (unsure if this difference is going to be very 
measurable).  It might also produce a binary that's compatible with 
both Darwin and Mac OS X (untested, I have no Darwin installation).

-bob

From Scott.Daniels at Acm.Org  Thu Nov  4 01:33:02 2004
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Thu Nov  4 01:31:49 2004
Subject: [Python-Dev] Re: RELEASED Python 2.4, beta 2
In-Reply-To: <41896EDA.5020805@python.org>
References: <41896EDA.5020805@python.org>
Message-ID: <cmbt9d$ecu$1@sea.gmane.org>

Anthony Baxter wrote:
> On behalf of the Python development team and the Python community, I'm
> happy to announce the second beta of Python 2.4.

Congratulations, another gold star.

-- 
-- Scott David Daniels
Scott.Daniels@Acm.Org

From jcarlson at uci.edu  Thu Nov  4 01:52:27 2004
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu Nov  4 02:00:45 2004
Subject: [Python-Dev] previously discussed binhex additions
Message-ID: <20041103160824.F7F2.JCARLSON@uci.edu>


Since it looks as though we're coming down to the wire on Python 2.4, I
thought I would mention the previously discussed binhex additions.

If desired, I would implement the long2bytes and bytes2long functions
with documentation and test cases as described a month ago by Tim
Peters:
http://mail.python.org/pipermail/python-dev/2004-October/049206.html

I could have it done this Saturday evening for testing/checking during
Sunday's bug day.


 - Josiah

From anthony at interlink.com.au  Thu Nov  4 04:13:33 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu Nov  4 04:13:58 2004
Subject: [Python-Dev] 
	Re: PEP 336, "Make None Callable", by Andrew McClelland
In-Reply-To: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net>
References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net>
Message-ID: <41899E5D.2080303@interlink.com.au>


> Abstract
> 
>     None should be a callable object that when called with any
>     arguments has no side effect and returns None.

My response to this is simply "yuck". This is a hack, abd I
don't think it's worthwhile. If you want to test for None, test
for None. Similarly, if you want a dictionary lookup to default
to a no-op method, there's a perfectly simple way to spell it now:

    def defaultNoopMethod(*args, **kwargs): pass
    somedict.get(key, defaultNoopMethod)

You can even do this as a one-liner using a lambda if you want.

Anthony



-- 
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.
From tim.peters at gmail.com  Thu Nov  4 04:25:41 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Thu Nov  4 04:25:43 2004
Subject: [Python-Dev] Re: PEP 336, "Make None Callable",
	by Andrew McClelland
In-Reply-To: <41899E5D.2080303@interlink.com.au>
References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net>
	<41899E5D.2080303@interlink.com.au>
Message-ID: <1f7befae0411031925522b6bb@mail.gmail.com>

[Anthony Baxter]
>> Abstract
>>
>>     None should be a callable object that when called with any
>>     arguments has no side effect and returns None.

> My response to this is simply "yuck". This is a hack, abd I
> don't think it's worthwhile.

Maybe it's not a coincidence that it's listed in the PEP index next to
"Reject Foolish Indentation" <0.9 wink>.

IMO, it's worse than not worthwhile:  it would be actively bad to
implement this.

    TypeError: 'NoneType' object is not callable

is a frequent runtime occurrence now, and performs a valuable service
when it happens by correctly pointing out a programming error. 
Python's None isn't meant to be neutral -- it's meant to mean "nothing
is there", and supplying nothing where something is required is a
logic error.
From anthony at interlink.com.au  Thu Nov  4 04:29:37 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu Nov  4 04:42:12 2004
Subject: [Python-Dev] TRUNK UNFROZEN
Message-ID: <4189A221.4020306@interlink.com.au>

The trunk is open again for checkins. As before, we're in
feature-freeze mode. Next release, which will hopefully be
2.4rc1, in 2 weeks time.

-- 
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.
From bac at OCF.Berkeley.EDU  Thu Nov  4 05:24:53 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Thu Nov  4 05:25:11 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <051E8585-2DF5-11D9-8F1A-000A95BA5446@redivi.com>
References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>	<41888D8E.5050705@interlink.com.au>	<41896256.1040104@ocf.berkeley.edu>
	<103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com>
	<41896943.9000906@v.loewis.de>
	<051E8585-2DF5-11D9-8F1A-000A95BA5446@redivi.com>
Message-ID: <4189AF15.9060407@ocf.berkeley.edu>

Bob Ippolito wrote:
> 
> On Nov 3, 2004, at 18:26, Martin v. L?wis wrote:
> 
>> Bob Ippolito wrote:
>>
>>> Argh!  It's nothing crazy, it removes incorrect linker flags and 
>>> delegates two Mac-specific API functions to where they belong.
>>
>>
>> I think Brett's thinking is that if this was a serious problem,
>> it should have been fixed months ago. The fact that nobody bothered
>> pushing that change before 2.4b1 indicates that the problem is not
>> serious. As an incompatible change that fixes a non-serious problem,
>> it shouldn't be applied before 2.5, in accordance with PEP 5
>> (which says that an incompatible change must see a deprecation first).
> 

Exactly.  It's too late in the release cycle for me to put it in unless Anthony 
says it is okay.

> 
> The thing is that the incompatible change is going in, but the 
> compatible changes (the reason for the incompatible change) are not.  
> This is very peculiar.
> 

Not really.  Only one function is being removed.  The function rearrangement is 
not under discussion here.  Plus there is the issue of testing.  I obviously 
checked it out and it seems to work, but this is build stuff which can be touchy.

If Anthony clears applying all the full patch I will, but otherwise I am 
playing it safe and waiting until 2.5 .

-Brett
From python at rcn.com  Thu Nov  4 05:54:14 2004
From: python at rcn.com (Raymond Hettinger)
Date: Thu Nov  4 05:57:08 2004
Subject: [Python-Dev] previously discussed binhex additions
In-Reply-To: <20041103160824.F7F2.JCARLSON@uci.edu>
Message-ID: <002801c4c22a$6e4a6200$e841fea9@oemcomputer>

> Since it looks as though we're coming down to the wire on Python 2.4,
I
> thought I would mention the previously discussed binhex additions.

Too late for Py2.4.  
Bugfixes only.
Not too early to submit a Py2.5 patch.


Raymond

From anthony at interlink.com.au  Thu Nov  4 05:58:40 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu Nov  4 05:59:14 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd
	November (tomorrrow)
In-Reply-To: <4189AF15.9060407@ocf.berkeley.edu>
References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]>	<41888D8E.5050705@interlink.com.au>	<41896256.1040104@ocf.berkeley.edu>	<103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com>	<41896943.9000906@v.loewis.de>	<051E8585-2DF5-11D9-8F1A-000
Message-ID: <4189B700.4010004@interlink.com.au>

Brett C. wrote:
> Not really.  Only one function is being removed.  The function 
> rearrangement is not under discussion here.  Plus there is the issue of 
> testing.  I obviously checked it out and it seems to work, but this is 
> build stuff which can be touchy.
> 
> If Anthony clears applying all the full patch I will, but otherwise I am 
> playing it safe and waiting until 2.5 .


Bear in mind that we've had _no_ macpython releases in the 2.4 cycle
so far, that I'm aware of. This is really a platform-specific build
issue.

From python at rcn.com  Thu Nov  4 06:06:10 2004
From: python at rcn.com (Raymond Hettinger)
Date: Thu Nov  4 06:09:12 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC
	3rdNovember (tomorrrow)
In-Reply-To: <051E8585-2DF5-11D9-8F1A-000A95BA5446@redivi.com>
Message-ID: <000001c4c22c$1daf6500$e841fea9@oemcomputer>

[Bob Ippolito]
> The thing is that the incompatible change is going in, but the
> compatible changes (the reason for the incompatible change) are not.
> This is very peculiar.

[Anthony]
> Bear in mind that we've had _no_ macpython releases in the 2.4 cycle
> so far, that I'm aware of. This is really a platform-specific build
> issue.


Poor Bob, this reasonable request has been kicked around too much.

It is a simple thing that strips unnecessary stuff out of the build.
Will someone who has a Mac please apply.



Raymond

From tim.peters at gmail.com  Thu Nov  4 06:25:32 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Thu Nov  4 06:25:35 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <1f7befae04110220402077a6fa@mail.gmail.com>
References: <1f7befae04110220402077a6fa@mail.gmail.com>
Message-ID: <1f7befae041103212576c59c42@mail.gmail.com>

[Tim]
> This example bothers me a little, under current Python CVS:
>
> """
> import gc, weakref
> 
> def C_gone(ignored):
>    print "An object of type C went away."
>
> class C:
>    def __init__(self):
>        self.wr = weakref.ref(self, C_gone)
>
> print "creating non-cyclic C instance"
> c = C()
> 
> print "creating cyclic C instance and destroying old one"
> c = C()  # triggers the print
> c.loop = c
> 
> print "getting rid of the cyclic one"
> c = None  # doesn't trigger the print
> gc.collect()  # ditto
> """

Recapping, it bothers me because the callback doesn't trigger now when
c is part of cyclic trash, but if c.wr were an instance with a __del__
method instead, the __del__ method *would* trigger.

There are (at least) two easy-to-code and pretty-easy-to-explain ways
to get that callback invoked safely.

One I sketched last time, and is in the attached patch-callback.txt. 
(Patches here just have gcmodule.c code changes, no corresponding
changes to comments.)  The difference is that it invokes a callback on
a weakref to trash iff the callback is reachable, instead of invoking
a callback on a weakref to trash iff the weakref it's attached to is
reachable.  All the tests we've accumulated still pass with this
change, and the example above does trigger a second print.

Another approach is in the attached patch-finalizer.txt.  This is more
(I think) like what Neil was aiming at last week, and amounts to
changing has_finalizer() to return true when it sees a weakref with a
callback.  Then the weakref, and everything reachable from it, gets
moved to the reachable set.  After that, no callback can access an
unreachable object directly, so callbacks are safe to call on that
count.  We still (and in the other patch too) clear all weakrefs to
trash before breaking cycles, so callbacks can't resurrect trash via
active weakrefs either.  But in the patch-finalizer case, the
callbacks attached to weakrefs moved to the reachable set no longer
get invoked "artificially" (forced by gc) -- they trigger iff their
referents eventually go away.

Some current tests fail with the patch-finalizer approach, but for
shallow reasons:  fewer callbacks get invoked, and some of the tests
currently use "global" (reachable) weakref callbacks as a way to
verify that gc really did clean up cyclic trash.  Those stop getting
invoked.  Some of the tests use bound method objects as callbacks, and
when a weakref contains a callback that's a bound method object, and
the weakref gets moved to the reachable set, its callback gets moved
there too, and so the instance wrt which the callback is a bound
method becomes reachable too, and so the class of that instance also
becomes reachable.  As a result, gc doesn't get rid of any of that
stuff, so the "external" callbacks never trigger.  Instead, the
weakrefs with the bound-method callbacks end up in gc.garbage.

I don't like that, because if it's possible to get rid of __del__
methods, I'd really like to see gc.garbage go away too.  Nothing we've
done with weakrefs so far can add anything to gc.garbage, so I like
them better on that count.  Pragmatically, sticking a weakref in
gc.garbage isn't of much use to the end-user now either, because a
weakref's callback is neither clearable nor even discoverable now at
the Python level.  But that could be changed too.

What I'm missing is a pile of practical <heh> use cases to distinguish these.

Let's look at weak value dicts, for reasons that will become clear. 
An entry in a WVD d now is of the form

   key: W

where W is an instance of a subclass of weakref.ref, and W has the
associated key as an attribute.  Every value in d shares the same
callback, set up by WVD.__init__():

   def __init__(self, *args, **kw):
       ...
       def remove(wr, selfref=ref(self)):
           self = selfref()
           if self is not None:
               del self.data[wr.key]
       self._remove = remove

So the callback has a weakref to d, and when it's invoked extracts the
specific key from the weakref instance and uses that to delete the
appropriate

   key: W

entry from d via the shared callback's own weakref to d.

Now it's not clear *why* the callback has a weakref to d.  It *could*
have been written like this instead:

       def remove(wr):
           del self.data[wr.key]

In fact, at one time it was written in that kind of way.  Rev 1.13 of
weakref.py changed it to use a weakref to self, and the motivations in
the patch that started this are feeble:

   http://www.python.org/sf/417795

"It's bad to rely on the garbage collector" and "this could lead to
uncollectable cycles if a subclass of Weak*Dictionary defines
__del__".  The proper responses were "no, it isn't" and "insane code
deserves whatever it gets" <0.5 wink>.

Anyway, if WVD *had* been left simple(r), it would be relevant to the
choices above:  if WVD d became part of cyclic trash, patch-finalizer
would move all its weakrefs to the reachable set, and then d itself
would be reachable too (via the shared callback's strong reference to
self).  As a result, none of the callbacks would trigger, and d would
"leak" (it wouldn't get cleaned up, and wouldn't show up in gc.garbage
either -- it would end up in an older generation, clogging gc forever
after because it would remain forever after reachable from its
(resurrected) weakrefs in gc.garbage).  Under patch-callback, d would
still go away (the simplest way to understand why is just that d *is*
in cyclic trash, and patch-callback never moves anything to the
reachable set because of weakrefs or weakref callbacks).

The *current* implementation of WVD doesn't have this problem under
patch-finalizer.  If d is part of cyclic trash, all its weakrefs get
moved to the reachable set, but d isn't directly reachable from their
shared callback, so d is not moved to the reachable set.  d's
tp_clear() gets invoked by gc in its cycle-breaking phase, and that
decrefs all of d's weakrefs that were moved to the unreachable set,
and they go away then (without triggering their callbacks) via the
normal refcount route.

The point is that this is all pretty subtle, and it bothers me that
"the obvious" way to implement WVD (the pre-patch-417795 way) would
lead to a worse outcome under patch-finalizer, so I also fear that
Python's current weakref users may be doing similar things now that
would also suffer under patch-finalizer.

OTOH, there's something conceptually nice about patch-finalizer -- it
is the easiest approach to explain (well, except for approaches that
allow segfaults to occur ...).  On the third hand, gc.garbage appears
poorly understood by users now, and patch-finalizer is actually easy
to explain only to those who already understand gc.garbage.
-------------- next part --------------
Index: Modules/gcmodule.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Modules/gcmodule.c,v
retrieving revision 2.80
diff -u -r2.80 gcmodule.c
--- Modules/gcmodule.c	1 Nov 2004 16:39:57 -0000	2.80
+++ Modules/gcmodule.c	4 Nov 2004 03:37:50 -0000
@@ -566,9 +566,8 @@
 	 * to imagine how calling it later could create a problem for us.  wr
 	 * is moved to wrcb_to_call in this case.
 	 */
-	 		if (IS_TENTATIVELY_UNREACHABLE(wr))
+	 		if (IS_TENTATIVELY_UNREACHABLE(wr->wr_callback))
 	 			continue;
-			assert(IS_REACHABLE(wr));
 
 			/* Create a new reference so that wr can't go away
 			 * before we can process it again.
@@ -577,9 +576,8 @@
 
 			/* Move wr to wrcb_to_call, for the next pass. */
 			wrasgc = AS_GC(wr);
-			assert(wrasgc != next); /* wrasgc is reachable, but
-			                           next isn't, so they can't
-			                           be the same */
+			if (wrasgc == next)
+				next = wrasgc->gc.gc_next;
 			gc_list_move(wrasgc, &wrcb_to_call);
 		}
 	}
@@ -593,7 +591,6 @@
 
 		gc = wrcb_to_call.gc.gc_next;
 		op = FROM_GC(gc);
-		assert(IS_REACHABLE(op));
 		assert(PyWeakref_Check(op));
 		wr = (PyWeakReference *)op;
 		callback = wr->wr_callback;
@@ -621,6 +618,7 @@
 		if (wrcb_to_call.gc.gc_next == gc) {
 			/* object is still alive -- move it */
 			gc_list_move(gc, old);
+			gc->gc.gc_refs = GC_REACHABLE;
 		}
 		else
 			++num_freed;












-------------- next part --------------
Index: Modules/gcmodule.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Modules/gcmodule.c,v
retrieving revision 2.80
diff -u -r2.80 gcmodule.c
--- Modules/gcmodule.c	1 Nov 2004 16:39:57 -0000	2.80
+++ Modules/gcmodule.c	4 Nov 2004 03:27:10 -0000
@@ -413,6 +413,9 @@
 		assert(delstr != NULL);
 		return _PyInstance_Lookup(op, delstr) != NULL;
 	}
+	else if (PyWeakref_Check(op) &&
+			((PyWeakReference *)op)->wr_callback != NULL)
+		return 1;
 	else if (PyType_HasFeature(op->ob_type, Py_TPFLAGS_HEAPTYPE))
 		return op->ob_type->tp_del != NULL;
 	else
@@ -566,10 +569,6 @@
 	 * to imagine how calling it later could create a problem for us.  wr
 	 * is moved to wrcb_to_call in this case.
 	 */
-	 		if (IS_TENTATIVELY_UNREACHABLE(wr))
-	 			continue;
-			assert(IS_REACHABLE(wr));
-
 			/* Create a new reference so that wr can't go away
 			 * before we can process it again.
 			 */












From jcarlson at uci.edu  Thu Nov  4 09:43:07 2004
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu Nov  4 09:51:01 2004
Subject: [Python-Dev] previously discussed binhex additions
In-Reply-To: <002801c4c22a$6e4a6200$e841fea9@oemcomputer>
References: <20041103160824.F7F2.JCARLSON@uci.edu>
	<002801c4c22a$6e4a6200$e841fea9@oemcomputer>
Message-ID: <20041104004215.F809.JCARLSON@uci.edu>


"Raymond Hettinger" <python@rcn.com> wrote:
> 
> > Since it looks as though we're coming down to the wire on Python 2.4,
> I
> > thought I would mention the previously discussed binhex additions.
> 
> Too late for Py2.4.  
> Bugfixes only.
> Not too early to submit a Py2.5 patch.


I'll wait until 2.4 is released then.  If there's no rush, then I guess
I'm not in one.

 - Josiah

From FBatista at uniFON.com.ar  Thu Nov  4 13:48:34 2004
From: FBatista at uniFON.com.ar (Batista, Facundo)
Date: Thu Nov  4 13:50:13 2004
Subject: [Python-Dev] Policy about old Python versions
Message-ID: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar>

Sometimes, reviewing bugs, I found some ones referring problems about old
Python versions.

Is there a policy about supporting old Python versions? At some time an old
version becomes "deprecated"?

For example, an open bug about a problem of Python 2.1, that is solved in
Python 2.2 and subsequents, when should be closed?

Thank you!

.	Facundo
From walter at livinglogic.de  Thu Nov  4 13:54:51 2004
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Thu Nov  4 13:54:56 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <1f7befae04110210537c35e514@mail.gmail.com>
References: <41864572.4000806@livinglogic.de>	
	<16774.45652.835103.327656@montanaro.dyndns.org>	
	<4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de>
	<1f7befae04110210537c35e514@mail.gmail.com>
Message-ID: <418A269B.6010207@livinglogic.de>

Tim Peters wrote:
> [Walter D?rwald]
> 
>>This doesn't help. The following assert in trace.py raises
>>an AssertionError:
>>   assert filename.endswith('.py')
>>
>>[...]
>>
>>So what is <doctest _threading_local[7]>?
> 
> I haven't followed this thread, but can answer that question literally
> <wink>:  it's code synthesized for the seventh doctest example in
> Lib/_threading_local.py's module docstring.  The file name is
> constructed by this line in doctest.py:
> 
>             # Use a special filename for compile(), so we can retrieve
>             # the source code during interactive debugging (see
>             # __patched_linecache_getlines).
>             filename = '<doctest %s[%d]>' % (test.name, examplenum)

I guess it's not worth it to try to fix doctest/trace to
provide sourcecode for doctest code, but IMHO trace should
be able to survive a doctest.

Removing the assert statement, so that trace.py runs to
completion shows a different problem: There are only
32 files covered according to the trace output. The
complete test log can be found here:
    http://styx.livinglogic.de/~walter/brokentrace.txt

The trace call looked like this:
./python ../../../trace.py --count --summary --missing Lib/test/regrtest.py
with ../../../trace.py being the trace.py from current
CVS with the assert statement removed.

So am I doing something wrong or is trace.py broken?

Bye,
    Walter D?rwald

From aahz at pythoncraft.com  Thu Nov  4 15:16:32 2004
From: aahz at pythoncraft.com (Aahz)
Date: Thu Nov  4 15:16:35 2004
Subject: [Python-Dev] Policy about old Python versions
In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar>
References: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar>
Message-ID: <20041104141632.GA27459@panix.com>

On Thu, Nov 04, 2004, Batista, Facundo wrote:
>
> Sometimes, reviewing bugs, I found some ones referring problems about
> old Python versions.
>
> Is there a policy about supporting old Python versions? At some time
> an old version becomes "deprecated"?
>
> For example, an open bug about a problem of Python 2.1, that is solved
> in Python 2.2 and subsequents, when should be closed?

As soon as the new version is released, if there is no bugfix planned
for that issue.  Two versions later (i.e. 2.3 in this case) even if it
looks like it ought to be fixed in 2.1 -- we haven't been doing bugfixes
more than one version old.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

WiFi is the SCSI of the 21st Century -- there are fundamental technical
reasons for sacrificing a goat.  (with no apologies to John Woods)
From misa at redhat.com  Thu Nov  4 16:38:19 2004
From: misa at redhat.com (Mihai Ibanescu)
Date: Thu Nov  4 16:38:23 2004
Subject: [Python-Dev] Overflow in socketmodule.c?
Message-ID: <20041104153819.GS13571@abulafia.devel.redhat.com>

Hello,

Can someone confirm this is indeed an overflow by one in socketmodule.c?


static PyObject *
socket_inet_ntop(PyObject *self, PyObject *args)
{
        int af;
        char* packed;
        int len;
        const char* retval;
#ifdef ENABLE_IPV6
        char ip[MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN) + 1];
#else
        char ip[INET_ADDRSTRLEN + 1];
#endif

        /* Guarantee NUL-termination for PyString_FromString() below */
        memset((void *) &ip[0], '\0', sizeof(ip) + 1);


If it is I'll go ahead and file it.

Thanks,
Misa
From gjc at inescporto.pt  Thu Nov  4 16:56:07 2004
From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro)
Date: Thu Nov  4 16:56:30 2004
Subject: [Python-Dev] Overflow in socketmodule.c?
In-Reply-To: <20041104153819.GS13571@abulafia.devel.redhat.com>
References: <20041104153819.GS13571@abulafia.devel.redhat.com>
Message-ID: <1099583767.31972.15.camel@spectrum.inescn.pt>

Qui, 2004-11-04 ?s 10:38 -0500, Mihai Ibanescu escreveu:
> Hello,
> 
> Can someone confirm this is indeed an overflow by one in socketmodule.c?
> 
> 
> static PyObject *
> socket_inet_ntop(PyObject *self, PyObject *args)
> {
>         int af;
>         char* packed;
>         int len;
>         const char* retval;
> #ifdef ENABLE_IPV6
>         char ip[MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN) + 1];
> #else
>         char ip[INET_ADDRSTRLEN + 1];
> #endif
> 
>         /* Guarantee NUL-termination for PyString_FromString() below */
>         memset((void *) &ip[0], '\0', sizeof(ip) + 1);
> 
> 
> If it is I'll go ahead and file it.

  Indeed, looks like buffer overflow to me..

> 
> Thanks,
> Misa
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/gjc%40inescporto.pt
-- 
Gustavo J. A. M. Carneiro
<gjc@inescporto.pt> <gustavo@users.sourceforge.net>
The universe is always one step beyond logic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3086 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20041104/2ddbd9ea/smime.bin
From tim at zope.com  Thu Nov  4 18:19:54 2004
From: tim at zope.com (Tim Peters)
Date: Thu Nov  4 18:20:01 2004
Subject: [Python-Dev] patch-finalizer vs Zope3
Message-ID: <20041104171957.A0C633B8038@smtp.zope.com>

FYI, and without taking time for non-trivial analysis.  I ran the non-ZEO
Zope3 tests with a Python patched to treat weakrefs with callbacks as
reachable (this was patch-finalizer.txt, attached to a msg I sent to
python-dev last night).

The first test to leave stuff in gc.garbage then:

    The following test left garbage:
    test_created (bugtracker.browser.tests.test_bug.BugBaseViewTest)
    65

where 65 is len(gc.garbage) after the test finished.  From there it climbed
steadily.  By the time the functional tests ended, the Python process was
about 575MB, and gc.garbage contained 324,056 weakrefs.

I haven't investigated why it happens, but that it *does* happen in an app
that believes it's well-behaved suggests that "practicality beats purity"
would win in the end (i.e., no matter how elegant the approach, if it's too
hard to live with, it won't fly).

Of course nothing is left in gc.garbage here under 2.3.4 or 2.4b2, or under
2.4b2 + patch-callback.  Under the latter, the Python process tops out at
about 150MB.

These were all with debug builds of Python, so process sizes are
substantially larger than with release builds (the object header is 2
pointers bigger in a debug build, and every object gotten via pymalloc grows
16 debug pad bytes, and everything gotten via PyMem_XYZ() gets pymalloc
debug padding too).

From jim at zope.com  Thu Nov  4 18:24:53 2004
From: jim at zope.com (Jim Fulton)
Date: Thu Nov  4 18:24:57 2004
Subject: [Python-Dev] Re: patch-finalizer vs Zope3
In-Reply-To: <20041104171957.A0C633B8038@smtp.zope.com>
References: <20041104171957.A0C633B8038@smtp.zope.com>
Message-ID: <418A65E5.6070403@zope.com>

Tim Peters wrote:
> FYI, and without taking time for non-trivial analysis.  I ran the non-ZEO
> Zope3 tests with a Python patched to treat weakrefs with callbacks as
> reachable (this was patch-finalizer.txt, attached to a msg I sent to
> python-dev last night).
> 
> The first test to leave stuff in gc.garbage then:
> 
>     The following test left garbage:
>     test_created (bugtracker.browser.tests.test_bug.BugBaseViewTest)
>     65
> 
> where 65 is len(gc.garbage) after the test finished.  From there it climbed
> steadily.  By the time the functional tests ended, the Python process was
> about 575MB, and gc.garbage contained 324,056 weakrefs.
> 
> I haven't investigated why it happens, but that it *does* happen in an app
> that believes it's well-behaved

It's all a question of your definition of good behavior. ;)

 > suggests that "practicality beats purity"
> would win in the end (i.e., no matter how elegant the approach, if it's too
> hard to live with, it won't fly).

Treating weakrefs with callbacks as reachable is a *big* change,
though in subtle ways.  I agree that this is too big for 2.4, but I *do*
think it should be considered for 2.5.

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org
From nas at arctrix.com  Thu Nov  4 18:38:35 2004
From: nas at arctrix.com (Neil Schemenauer)
Date: Thu Nov  4 18:38:39 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <1f7befae041103212576c59c42@mail.gmail.com>
References: <1f7befae04110220402077a6fa@mail.gmail.com>
	<1f7befae041103212576c59c42@mail.gmail.com>
Message-ID: <20041104173835.GA15004@mems-exchange.org>

On Thu, Nov 04, 2004 at 12:25:32AM -0500, Tim Peters wrote:
> One I sketched last time, and is in the attached patch-callback.txt. 
> (Patches here just have gcmodule.c code changes, no corresponding
> changes to comments.)  The difference is that it invokes a callback on
> a weakref to trash iff the callback is reachable, instead of invoking
> a callback on a weakref to trash iff the weakref it's attached to is
> reachable.

I like this idea best.  I don't see why it should be hard to
explain.  Instead of saying:

    If you want the weakref callback to be invoked, ensure the
    WEAKREF outlives the referent.

we say:

    If you want the weakref callback to be invoked, ensure the
    CALLBACK outlives the referent.

I think there may be one small problem though.  Callbacks get passed
the weakref object.  Doesn't that break the rule that trash should
not be exposed to Python code while the collection is taking place?

  Neil
From jim at zope.com  Thu Nov  4 18:47:00 2004
From: jim at zope.com (Jim Fulton)
Date: Thu Nov  4 18:47:08 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <20041104173835.GA15004@mems-exchange.org>
References: <1f7befae04110220402077a6fa@mail.gmail.com>	<1f7befae041103212576c59c42@mail.gmail.com>
	<20041104173835.GA15004@mems-exchange.org>
Message-ID: <418A6B14.8080206@zope.com>

Neil Schemenauer wrote:
> On Thu, Nov 04, 2004 at 12:25:32AM -0500, Tim Peters wrote:
> 
>>One I sketched last time, and is in the attached patch-callback.txt. 
>>(Patches here just have gcmodule.c code changes, no corresponding
>>changes to comments.)  The difference is that it invokes a callback on
>>a weakref to trash iff the callback is reachable, instead of invoking
>>a callback on a weakref to trash iff the weakref it's attached to is
>>reachable.
> 
> 
> I like this idea best.  I don't see why it should be hard to
> explain.  Instead of saying:
> 
>     If you want the weakref callback to be invoked, ensure the
>     WEAKREF outlives the referent.
> 
> we say:
> 
>     If you want the weakref callback to be invoked, ensure the
>     CALLBACK outlives the referent.
> 
> I think there may be one small problem though.  Callbacks get passed
> the weakref object.  Doesn't that break the rule that trash should
> not be exposed to Python code while the collection is taking place?

Exactly, that's why I prefer the other approach.

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org
From misa at redhat.com  Thu Nov  4 18:47:26 2004
From: misa at redhat.com (Mihai Ibanescu)
Date: Thu Nov  4 18:47:29 2004
Subject: [Python-Dev] Overflow in socketmodule.c?
In-Reply-To: <1099583767.31972.15.camel@spectrum.inescn.pt>
References: <20041104153819.GS13571@abulafia.devel.redhat.com>
	<1099583767.31972.15.camel@spectrum.inescn.pt>
Message-ID: <20041104174726.GT13571@abulafia.devel.redhat.com>

On Thu, Nov 04, 2004 at 03:56:07PM +0000, Gustavo J. A. M. Carneiro wrote:
> Qui, 2004-11-04 ?s 10:38 -0500, Mihai Ibanescu escreveu:
> > Hello,
> > 
> > Can someone confirm this is indeed an overflow by one in socketmodule.c?
> > 
> > 
> > static PyObject *
> > socket_inet_ntop(PyObject *self, PyObject *args)
> > {
> >         int af;
> >         char* packed;
> >         int len;
> >         const char* retval;
> > #ifdef ENABLE_IPV6
> >         char ip[MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN) + 1];
> > #else
> >         char ip[INET_ADDRSTRLEN + 1];
> > #endif
> > 
> >         /* Guarantee NUL-termination for PyString_FromString() below */
> >         memset((void *) &ip[0], '\0', sizeof(ip) + 1);
> > 
> > 
> > If it is I'll go ahead and file it.
> 
>   Indeed, looks like buffer overflow to me..

Filed as SF bug 105470

Misa
From martin at v.loewis.de  Thu Nov  4 18:51:56 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu Nov  4 18:51:48 2004
Subject: [Python-Dev] Policy about old Python versions
In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar>
References: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar>
Message-ID: <418A6C3C.3050008@v.loewis.de>

Batista, Facundo wrote:
> Sometimes, reviewing bugs, I found some ones referring problems about old
> Python versions.
> 
> Is there a policy about supporting old Python versions? At some time an old
> version becomes "deprecated"?
> 
> For example, an open bug about a problem of Python 2.1, that is solved in
> Python 2.2 and subsequents, when should be closed?

It is (IMO) fine to close anything now that is fixed in 2.3 and newer.
The proper "Resolution" is "Fixed", with a Comment "Fixed in 2.3".

There likely will be one more 2.3 release, so you might not want to
close something as "Fixed in 2.4" that is still incorrect in 2.3;
instead, consider backporting the patch (we are all volunteers, so
this is clearly not mandatory). If you find something is fixed in
2.4, but have no time to find out how precisely it was fixed, still
leave a comment saying so.

After the next 2.3 release, people are still free to backport to older
branches - after all, distributors may chose to roll out patches
collected even after the last release of some 2.x. However, in general,
expect that 2.3 and older needs no further attention RSN, and
"backporting" will mean "to 2.4".

Regards,
Martin
From martin at v.loewis.de  Thu Nov  4 19:05:28 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu Nov  4 19:05:20 2004
Subject: [Python-Dev] Overflow in socketmodule.c?
In-Reply-To: <20041104153819.GS13571@abulafia.devel.redhat.com>
References: <20041104153819.GS13571@abulafia.devel.redhat.com>
Message-ID: <418A6F68.8080304@v.loewis.de>

Mihai Ibanescu wrote:
> Can someone confirm this is indeed an overflow by one in socketmodule.c?

Yes. With some luck (e.g. on x86 with gcc), it might not cause buffer
overruns, as a) the compiler overallocates on the stack because of
padding, anyway, and b) the overwriting might write into the next
variable (i.e. retval), which is uninitialized at this point, anyway.

Regards,
Martin

From tim at zope.com  Thu Nov  4 19:38:38 2004
From: tim at zope.com (Tim Peters)
Date: Thu Nov  4 19:38:43 2004
Subject: [Python-Dev] RE: patch-finalizer vs Zope3
In-Reply-To: <418A65E5.6070403@zope.com>
Message-ID: <20041104183839.AABEF3B8038@smtp.zope.com>

[Jim Fulton]
> Treating weakrefs with callbacks as reachable is a *big* change, though
> in subtle ways.

It sure appears to have "more consequences" than I guessed it would.

> I agree that this is too big for 2.4, but I *do* think it should be
> considered for 2.5.

Me too.  It ensures that a callback is never ignored unless a referent
definitely outlives a weakref with that callback, even at the cost of
resurrecting trash weakrefs (to make their callbacks safe to call).
"Definitely outlives" is usually easy to talk about in CPython because of
refcount lifetime semantics, but hard to make sense of operationally without
refcount semantics.  When they're both in cyclic trash, a weakref and its
referent both become unreachable truly at the same instant.  The approach so
far has then been to say (in effect):

    Fine, then it's legitimate to impose any linear order on death, and
    if we say the weakref dies first, then there's no issue remaining --
    just as in non-cyclic trash, when a weakref dies before its
    referent, the wr's callback will never be invoked.

That's defensible but not compelling.  I'm still not sure any other approach
is compelling either.  patch-finalizer effectly reverses the current
approach, saying:

    Fine, then it's legitimate to impose any linear order on death, and
    if we say the referent dies first, then there's no issue remaining --
    just as in non-cyclic trash, when the referent dies before its
    weakref, the wr's callback will be invoked(*).

    (*) Except we have to resurrect the weakref to make this possible, and
        that may also end up resurrecting the referent, and then neither
        actually goes away, and so the callback doesn't actually get
        invoked either (although it may get invoked later, if the user
        fiddles with the stuff in gc.garbage enough to make that
        possible).

All I can say is that the callback better be damned important to justify all
the extra pain <wink>.

still-seeking-use-cases-ly y'rs  - tim

From foom at fuhm.net  Thu Nov  4 19:44:22 2004
From: foom at fuhm.net (James Y Knight)
Date: Thu Nov  4 19:44:26 2004
Subject: [Python-Dev] weakref gc semantics
In-Reply-To: <1f7befae04110220402077a6fa@mail.gmail.com>
References: <1f7befae04110220402077a6fa@mail.gmail.com>
Message-ID: <8AFE1902-2E91-11D9-AF2F-000A95A50FB2@fuhm.net>

On Nov 2, 2004, at 11:40 PM, Tim Peters wrote:
> a reachable callback can have visible effects, and there
> really isn't ambiguity about which of c and W "dies first" (c does) in
> the example.

An unreachable callback function can also have visible effects.

> The question is whether gc should really be looking at whether the
> weakref *callback* is reachable, regardless of whether the weakref
> itself is reachable (if the weakref is reachable, its callback is too,
> and 2.4b2 invokes it -- the only case in question is the one in the
> example, where the weakref and the weakref's referent are unreachable
> but the weakref's callback is reachable).

 From what I'm hearing here, the following two pieces of code would then 
have different behavior under patch-callback:

"""
def C_gone(ignored):
     print "An object of type C went away."

class C:
     def __init__(self):
         self.wr = weakref.ref(self, C_gone)
"""

and

"""
class C:
     def __init__(self):
         def C_gone(ignored):
             print "An object of type C went away."
         self.wr = weakref.ref(self, C_gone)
"""

If that's correct, it sounds like a bad idea to me.

James

From tim.peters at gmail.com  Thu Nov  4 20:32:51 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Thu Nov  4 20:32:57 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <20041104173835.GA15004@mems-exchange.org>
References: <1f7befae04110220402077a6fa@mail.gmail.com>
	<1f7befae041103212576c59c42@mail.gmail.com>
	<20041104173835.GA15004@mems-exchange.org>
Message-ID: <1f7befae04110411327af9bf3d@mail.gmail.com>

[Neil Schemenauer]
> I like this idea [patch-callback] best.  I don't see why it should be hard to
> explain.  Instead of saying:
>
>    If you want the weakref callback to be invoked, ensure the
>    WEAKREF outlives the referent.
>
> we say:
>
>    If you want the weakref callback to be invoked, ensure the
>    CALLBACK outlives the referent.

The former is accurate independent of whether we're talking about
cyclic trash now, but that latter would not be.  For example,

def callback(ignore):
    print 'hi'

class C:
    pass

c = C()
w = weakref.ref(c, callback)
w = None
c = None

The callback outlives c, but the callback doesn't get invoked, because
w didn't outlive c.  The callback does get invoked if the order of the
last two lines is swapped.  It's easy to explain this in terms of
which of {c, w} goes away first, but not in terms of callback's
lifetime.

> I think there may be one small problem though.  Callbacks get passed
> the weakref object.  Doesn't that break the rule that trash should
> not be exposed to Python code while the collection is taking place?

That's why I'm not touching anything here again for 2.4 <wink>.  For
builtin weakrefs to trash, it doesn't matter -- they get cleared
before gc invokes any callbacks, so the worst a callback on a
builtin-weakref-to-trash-object could do with its argument is
resurrect a weakref than returns None if you call it.  BFD.

But I think we could concoct tests where instances of a weakref
subclass did arbitrary damage in callbacks, via attributes bound to
trash objects.  The current "invoke the callback iff the weakref is
reachable" avoids getting in trouble there, and I had weakref
subclasses in the back of my mind when "settling" for that.  We should
write some nasty tests of weakref subclasses with callbacks
interacting with gc -- there aren't any now.
From tim.peters at gmail.com  Thu Nov  4 21:11:32 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Thu Nov  4 21:11:38 2004
Subject: [Python-Dev] weakref gc semantics
In-Reply-To: <8AFE1902-2E91-11D9-AF2F-000A95A50FB2@fuhm.net>
References: <1f7befae04110220402077a6fa@mail.gmail.com>
	<8AFE1902-2E91-11D9-AF2F-000A95A50FB2@fuhm.net>
Message-ID: <1f7befae041104121160f2b4ec@mail.gmail.com>

[James Y Knight]
> An unreachable callback function can also have visible effects.

Quite true -- it might even reformat your hard drives <wink>.

> ...
> From what I'm hearing here, the following two pieces of code would then
> have different behavior under patch-callback:
>
> """
> def C_gone(ignored):
>     print "An object of type C went away."
>
> class C:
>     def __init__(self):
>         self.wr = weakref.ref(self, C_gone)
> """
>
> and
>
> """
> class C:
>     def __init__(self):
>         def C_gone(ignored):
>             print "An object of type C went away."
>         self.wr = weakref.ref(self, C_gone)
> """
>
> If that's correct, it sounds like a bad idea to me.

Assuming you're assuming the same driver is getting used after these
snippets, yes, under patch-callback an instance of C in cyclic trash
in the first case would trigger the callback, but not in the second
case.

I don't know why you believe that's bad:  perhaps you could articulate
a precise set of rules for what would be "good"?  Maybe that's even
the set of rules Python 2.4b2 actually uses.  They seem more or less
arbitrary to me (and so do patch-callback's -- and so do
patch-finalizer's).  For example, you can fiddle your examples above
to get the same kind of "hmm, in one case it does, in the other case
it doesn't" outcome in 2.4b2 by manipulating whether the weakref is
externally reachable, instead of manipulating the reachability of the
callback.  Is that change in *what* you fiddle essentially different,
so that it's "not bad" to get different outcomes when weakref
reachability is fiddled?

I don't think it's obvious, but under patch-finalizer they both get
the same outcome (the callback is invoked, and the weakref gets
deallocated before gc.garbage gets built).  OTOH, if you make C_gone a
method of C, and use self.C_gone as the callback, then the outcome is
different:  then the callback doesn't get called, and c is actually
resurrected under patch-finalizer, even in the driver's first

    print "creating non-cyclic C instance"
    c = C()

case (although the print stmt is no longer truthful then).
From barry at python.org  Thu Nov  4 22:06:08 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Nov  4 22:06:18 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path to a
	release
In-Reply-To: <41896727.7030206@v.loewis.de>
References: <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de>
Message-ID: <1099602368.17767.77.camel@geddy.wooz.org>

On Wed, 2004-11-03 at 18:17, "Martin v. L?wis" wrote:

> I think aiming at rewriting ht2html
> in Python for 2.5 would be a worthwhile goal (I have dropped the
> idea that conversion to xml would be worthwhile).

Did you mean the Doc/perl stuff instead of ht2html?  The latter is
already Python (but you knew that).

> Then, the release manager could trivially build the documentation.
> For that to work, it is necessary that the documentation builds out
> of the box; for that to happen, it is necessary that a broad audience
> can build the documentation, so that problems are detected long
> before the release (i.e. short after a checkin - or better yet,
> before). For the PostScript/PDF release, one certainly needs TeX,
> but that is free software.

Actually, building the html documentation isn't hard, and I generally do
it whenever I'm making a change to .tex files.  I may not always write
Fred-happy markup but I try to make sure it's at least syntactically
correct.  It's also a useful thing to have around for when there's no
net access.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: This is a digitally signed message part
Url : http://mail.python.org/pipermail/python-dev/attachments/20041104/cdd83d0e/attachment.pgp
From greg at cosc.canterbury.ac.nz  Fri Nov  5 01:45:15 2004
From: greg at cosc.canterbury.ac.nz (Greg Ewing)
Date: Fri Nov  5 01:45:22 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <20041104173835.GA15004@mems-exchange.org>
References: <1f7befae04110220402077a6fa@mail.gmail.com>
	<1f7befae041103212576c59c42@mail.gmail.com>
	<20041104173835.GA15004@mems-exchange.org>
Message-ID: <418ACD1B.9060000@cosc.canterbury.ac.nz>

Neil Schemenauer wrote:
> I think there may be one small problem though.  Callbacks get passed
> the weakref object.  Doesn't that break the rule that trash should
> not be exposed to Python code while the collection is taking place?

Maybe the callback shouldn't be passed the weakref object?
I don't see what good that does anyway, since the weakref
is just an empty shell by then, isn't it?

Greg


From tim.peters at gmail.com  Fri Nov  5 02:42:15 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov  5 02:42:21 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <418ACD1B.9060000@cosc.canterbury.ac.nz>
References: <1f7befae04110220402077a6fa@mail.gmail.com>
	<1f7befae041103212576c59c42@mail.gmail.com>
	<20041104173835.GA15004@mems-exchange.org>
	<418ACD1B.9060000@cosc.canterbury.ac.nz>
Message-ID: <1f7befae04110417421a072d75@mail.gmail.com>

[Greg Ewing]
> Maybe the callback shouldn't be passed the weakref object?
> I don't see what good that does anyway, since the weakref
> is just an empty shell by then, isn't it?

If it's a builtin weakref object, yes.  Weakref subclasses can add
useful data, though.  The way Python's weak dictionaries exploit this
was explained earlier in the thread.
From anthony at interlink.com.au  Fri Nov  5 03:53:54 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri Nov  5 03:54:20 2004
Subject: [Python-Dev] Policy about old Python versions
In-Reply-To: <20041104141632.GA27459@panix.com>
References: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar>
	<20041104141632.GA27459@panix.com>
Message-ID: <418AEB42.8040306@interlink.com.au>

Aahz wrote:
> As soon as the new version is released, if there is no bugfix planned
> for that issue.  Two versions later (i.e. 2.3 in this case) even if it
> looks like it ought to be fixed in 2.1 -- we haven't been doing bugfixes
> more than one version old.

That's not to say that if someone wants to cut a 2.2.4 or 2.1.4 I'm
going to stop them. I've no interest in maintaining more than trunk
plus the most recent maint branch.

As for the original query, about what to do with bugs filed against
2.1.x - I'd suggest closing them. There's a vanishingly small chance
that there will ever be another 2.1 or 2.2 release.

Anthony
From bac at OCF.Berkeley.EDU  Fri Nov  5 06:37:53 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Fri Nov  5 06:38:14 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC
	3rdNovember (tomorrrow)
In-Reply-To: <000001c4c22c$1daf6500$e841fea9@oemcomputer>
References: <000001c4c22c$1daf6500$e841fea9@oemcomputer>
Message-ID: <418B11B1.8000106@ocf.berkeley.edu>

Raymond Hettinger wrote:
> [Bob Ippolito]
> 
>>The thing is that the incompatible change is going in, but the
>>compatible changes (the reason for the incompatible change) are not.
>>This is very peculiar.
> 
> 
> [Anthony]
> 
>>Bear in mind that we've had _no_ macpython releases in the 2.4 cycle
>>so far, that I'm aware of. This is really a platform-specific build
>>issue.
> 
> 
> 
> Poor Bob, this reasonable request has been kicked around too much.
> 
> It is a simple thing that strips unnecessary stuff out of the build.
> Will someone who has a Mac please apply.
> 

If someone else wants to apply it I have uploaded my revision of Bob's patch to 
SF (just make sure to run autoconf; patch is at 
http://www.python.org/sf/1035255 ).  And I don't see why a Mac user needs to 
apply it since I have personally tested it to the best of my abilities.

But as I have said I will not apply unless Anthony gives me a clear "yes" to do 
so.  Call me over-cautious and annoying.

-Brett
From mwh at python.net  Fri Nov  5 12:48:16 2004
From: mwh at python.net (Michael Hudson)
Date: Fri Nov  5 12:48:18 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <1f7befae04110411327af9bf3d@mail.gmail.com> (Tim Peters's
	message of "Thu, 4 Nov 2004 14:32:51 -0500")
References: <1f7befae04110220402077a6fa@mail.gmail.com>
	<1f7befae041103212576c59c42@mail.gmail.com>
	<20041104173835.GA15004@mems-exchange.org>
	<1f7befae04110411327af9bf3d@mail.gmail.com>
Message-ID: <2m7jp05xfj.fsf@starship.python.net>

Tim Peters <tim.peters@gmail.com> writes:

[stuff i admittedly haven't thought terribly hard about]

> The former is accurate independent of whether we're talking about
> cyclic trash now, but that latter would not be.  For example,
>
> def callback(ignore):
>     print 'hi'
>
> class C:
>     pass
>
> c = C()
> w = weakref.ref(c, callback)
> w = None
> c = None
>
> The callback outlives c, but the callback doesn't get invoked, because
> w didn't outlive c.  The callback does get invoked if the order of the
> last two lines is swapped.  It's easy to explain this in terms of
> which of {c, w} goes away first, but not in terms of callback's
> lifetime.

As disclaimed above, I haven't thought about this a great deal, but I
would expect it to be the lifetime of the weakref which matters -- in
the not-all-that-unlikley case of the callback not being a lambda or
local function, it is *always* going to be reachable, and indeed is
never going to die until the process exits...

I take it the changes you're proposing aren't going to change the
semantics of the quoted code?

Cheers,
mwh

-- 
  I'm about to search Google for contract assassins to go to Iomega
  and HP's programming groups and kill everyone there with some kind
  of electrically charged rusty barbed thing.
                -- http://bofhcam.org/journal/journal.html, 2002-01-08
From mwh at python.net  Fri Nov  5 12:50:51 2004
From: mwh at python.net (Michael Hudson)
Date: Fri Nov  5 12:50:55 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path to a
	release
In-Reply-To: <41896727.7030206@v.loewis.de> (
	=?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Thu,
	04 Nov 2004 00:17:59 +0100")
References: <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de>
Message-ID: <2m3bzo5xb8.fsf@starship.python.net>

"Martin v. L?wis" <martin@v.loewis.de> writes:

> Also, I think it would be good to eliminate Fred's role altogether:
> the documentation build process should be much more automatic, and
> less relying on third-party tools. I think aiming at rewriting ht2html
> in Python for 2.5 would be a worthwhile goal (I have dropped the
> idea that conversion to xml would be worthwhile).

Err, do you mean latex2html?  If so, I'm not at all sure that's
realistic.

Cheers,
mwh

-- 
  After a heavy night I travelled on, my face toward home - the comma
  being by no means guaranteed.           -- paraphrased from cam.misc
From jim at zope.com  Fri Nov  5 15:25:20 2004
From: jim at zope.com (Jim Fulton)
Date: Fri Nov  5 15:25:31 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <418ACD1B.9060000@cosc.canterbury.ac.nz>
References: <1f7befae04110220402077a6fa@mail.gmail.com>	<1f7befae041103212576c59c42@mail.gmail.com>	<20041104173835.GA15004@mems-exchange.org>
	<418ACD1B.9060000@cosc.canterbury.ac.nz>
Message-ID: <418B8D50.4090908@zope.com>

Greg Ewing wrote:
> Neil Schemenauer wrote:
> 
>> I think there may be one small problem though.  Callbacks get passed
>> the weakref object.  Doesn't that break the rule that trash should
>> not be exposed to Python code while the collection is taking place?
> 
> 
> Maybe the callback shouldn't be passed the weakref object?
> I don't see what good that does anyway, since the weakref
> is just an empty shell by then, isn't it?

No.  It can still be used as a key. It still has it's original
hash value and will compare equal to other weakrefs to the same
original object. Weak-key dictionaries depend on this. In addition,
as Tim points out, it could be a weakref-subclass instance and
could have additional useful data.

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org
From tim.peters at gmail.com  Fri Nov  5 16:05:23 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov  5 16:05:26 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <418B8D50.4090908@zope.com>
References: <1f7befae04110220402077a6fa@mail.gmail.com>
	<1f7befae041103212576c59c42@mail.gmail.com>
	<20041104173835.GA15004@mems-exchange.org>
	<418ACD1B.9060000@cosc.canterbury.ac.nz> <418B8D50.4090908@zope.com>
Message-ID: <1f7befae041105070549529618@mail.gmail.com>

[Greg Ewing]
>> Maybe the callback shouldn't be passed the weakref object?
>> I don't see what good that does anyway, since the weakref
>> is just an empty shell by then, isn't it?

[Jim Fulton]
> No.  It can still be used as a key. It still has it's original
> hash value and will compare equal to other weakrefs to the same
> original object. Weak-key dictionaries depend on this.

Right.

> In addition, as Tim points out, it could be a weakref-subclass instance
> and could have additional useful data.

Adn weak-value dictionaries depend on that; from weakref.py:

class KeyedRef(ref):
    """Specialized reference that includes a key corresponding to the value.

    This is used in the WeakValueDictionary to avoid having to create
    a function object for each key stored in the mapping.  A shared
    callback object can use the 'key' attribute of a KeyedRef instead
    of getting a reference to the key from an enclosing scope.

    """
    ...
From pje at telecommunity.com  Fri Nov  5 16:24:24 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri Nov  5 16:23:34 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path to
	a release
In-Reply-To: <2m3bzo5xb8.fsf@starship.python.net>
References: <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
Message-ID: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>

At 11:50 AM 11/5/04 +0000, Michael Hudson wrote:
>"Martin v. L?wis" <martin@v.loewis.de> writes:
>
> > Also, I think it would be good to eliminate Fred's role altogether:
> > the documentation build process should be much more automatic, and
> > less relying on third-party tools. I think aiming at rewriting ht2html
> > in Python for 2.5 would be a worthwhile goal (I have dropped the
> > idea that conversion to xml would be worthwhile).
>
>Err, do you mean latex2html?  If so, I'm not at all sure that's
>realistic.

I've dabbled in the guts of latex2html before; it's certainly not pretty.

IMO a better long term option might be to use the Python docutils and 
migrate to reStructuredText, since there are a bevy of backends available 
for latex, PDF, HTML, etc.  They would probably need more work before 
they'll be suitable to handle the entire doc generation process, though.

From tim.peters at gmail.com  Fri Nov  5 16:45:42 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov  5 16:45:47 2004
Subject: [Python-Dev] Re: weakref gc semantics
In-Reply-To: <2m7jp05xfj.fsf@starship.python.net>
References: <1f7befae04110220402077a6fa@mail.gmail.com>
	<1f7befae041103212576c59c42@mail.gmail.com>
	<20041104173835.GA15004@mems-exchange.org>
	<1f7befae04110411327af9bf3d@mail.gmail.com>
	<2m7jp05xfj.fsf@starship.python.net>
Message-ID: <1f7befae04110507454a504cf6@mail.gmail.com>

[Michael Hudson]
> As disclaimed above, I haven't thought about this a great deal, but I
> would expect it to be the lifetime of the weakref which matters -- in
> the not-all-that-unlikley case of the callback not being a lambda or
> local function, it is *always* going to be reachable, and indeed is
> never going to die until the process exits...

A difficulty remaining is that when a weakref and its referent both
become unreachable simultaneously, relative lifetime becomes fuzzy. 
Even when, in the "simple" examples, I write

    weakref = None
    referent = None # callback doesn't trigger

or

    referent = None # callback does trigger
    weakref = None

the idea that relative lifetime controls the outcome in a
*predictable* way depends on that CPython uses refcounting as its
primary gc strategy.  Indeed, Python the language (as opposed to
CPython the implementation) really can't guarantee that the callback
does or doesn't trigger in either simple example above.

These fuzzy issues have become important because the lack of clarity
about exact intended semantics has left us with segfault-producing
bugs in the interaction between cyclic gc and weakrefs, and we've been
thru more than one round of those now.

A complication is that, perhaps contrary to initial expectation,
outside of contrived tests I'm not sure callbacks that *aren't*
"lambdas or local functions" actually exist.  In the core, the two
flavors of weak dict use very different strategies, but both end up
using a unique function per dict instance as the callback, created in
the weak dict's __init__ and bound to an attribute of the dict.  So
these are guaranteed to create "fuzzy situations" when a weak dict
becomes part of cyclic trash (the dict, and the weakrefs it holds, and
their callbacks, generally all become unreachable at the same
instant).

> I take it the changes you're proposing aren't going to change the
> semantics of the quoted code?

I'm not proposing anything specific, and if it comes to it I bet I
could live with 2.4b2 forever.  Its rules seem ultimately arbitrary,
though, so aren't really attractive.  Other rules are possible, and
should be considered.  For the first time, I think we finally have a
deep understanding of what we can and cannot *get away* with now wrt
preventing segfault bugs (and other related nastiness).  But that
understanding doesn't reduce the universe of acceptable (bug-free and
defensible) policies to what 2.4b2 implements.

In any case, no change to gcmodule.c's weakref policy would change
anything about how CPython behaves in cases where refcounting does the
whole job.  So the answer to your literal <wink> question is "no". 
The behaviors of

> def callback(ignore):
>     print 'hi'
>
> class C:
>     pass
>
> c = C()
> w = weakref.ref(c, callback)
> w = None
> c = None

kinds of examples are wholly determined by refcount lifetime semantics
in CPython, and that won't change.
From trentm at ActiveState.com  Fri Nov  5 19:29:40 2004
From: trentm at ActiveState.com (Trent Mick)
Date: Fri Nov  5 19:31:04 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path to	a
	release
In-Reply-To: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
References: <41896727.7030206@v.loewis.de>
	<4188BE21.6040205@interlink.com.au>	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
Message-ID: <418BC694.9010306@activestate.com>

Phillip J. Eby wrote:
> I've dabbled in the guts of latex2html before; it's certainly not pretty.
> 
> IMO a better long term option might be to use the Python docutils and 
> migrate to reStructuredText, since there are a bevy of backends 
> available for latex, PDF, HTML, etc.  They would probably need more work 
> before they'll be suitable to handle the entire doc generation process, 
> though.

What do people think about docbook? I'm impressed with the kind of doc 
output that, for example, the PHP guys and Mark Pilgrim's Dive Into 
Python and the Subversion Book, can generate from docbook sources. I'm 
not suggesting any whole hog re-write of the Python docs anytime soon, 
just wanted to feel it out.

Trent

-- 
Trent Mick
trentm@activestate.com
From trentm at ActiveState.com  Fri Nov  5 19:26:02 2004
From: trentm at ActiveState.com (Trent Mick)
Date: Fri Nov  5 19:42:48 2004
Subject: [Python-Dev] Policy about old Python versions
In-Reply-To: <418AEB42.8040306@interlink.com.au>
References: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar>	<20041104141632.GA27459@panix.com>
	<418AEB42.8040306@interlink.com.au>
Message-ID: <418BC5BA.7090108@activestate.com>

Anthony Baxter wrote:
> As for the original query, about what to do with bugs filed against
> 2.1.x - I'd suggest closing them. There's a vanishingly small chance
> that there will ever be another 2.1 or 2.2 release.

Maybe a "Known Issues in Python 2.1" PEP which refers to these bug 
numbers and then close the bugs? I don't want to make more work for 
people, though.

Trent

-- 
Trent Mick
trentm@activestate.com
From bac at OCF.Berkeley.EDU  Fri Nov  5 20:42:43 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Fri Nov  5 20:43:06 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path to	a
	release
In-Reply-To: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
References: <41896727.7030206@v.loewis.de>
	<4188BE21.6040205@interlink.com.au>	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
Message-ID: <418BD7B3.7050306@ocf.berkeley.edu>

Phillip J. Eby wrote:
> At 11:50 AM 11/5/04 +0000, Michael Hudson wrote:
> 
>> "Martin v. L?wis" <martin@v.loewis.de> writes:
>>
>> > Also, I think it would be good to eliminate Fred's role altogether:
>> > the documentation build process should be much more automatic, and
>> > less relying on third-party tools. I think aiming at rewriting ht2html
>> > in Python for 2.5 would be a worthwhile goal (I have dropped the
>> > idea that conversion to xml would be worthwhile).
>>
>> Err, do you mean latex2html?  If so, I'm not at all sure that's
>> realistic.
> 
> 
> I've dabbled in the guts of latex2html before; it's certainly not pretty.
> 
> IMO a better long term option might be to use the Python docutils and 
> migrate to reStructuredText, since there are a bevy of backends 
> available for latex, PDF, HTML, etc.  They would probably need more work 
> before they'll be suitable to handle the entire doc generation process, 
> though.
> 

This has been proposed before but has been shot down.  If I remember correctly 
the reasoning was that LaTeX gave us more control and thus served the purpose 
better for the official docs.

-Brett
From pje at telecommunity.com  Fri Nov  5 21:04:21 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri Nov  5 21:03:32 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path
	to	a release
In-Reply-To: <418BD7B3.7050306@ocf.berkeley.edu>
References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>

At 11:42 AM 11/5/04 -0800, Brett C. wrote:
>Phillip J. Eby wrote:
>>At 11:50 AM 11/5/04 +0000, Michael Hudson wrote:
>>>Err, do you mean latex2html?  If so, I'm not at all sure that's
>>>realistic.
>>
>>I've dabbled in the guts of latex2html before; it's certainly not pretty.
>>IMO a better long term option might be to use the Python docutils and 
>>migrate to reStructuredText, since there are a bevy of backends available 
>>for latex, PDF, HTML, etc.  They would probably need more work before 
>>they'll be suitable to handle the entire doc generation process, though.
>
>This has been proposed before but has been shot down.  If I remember 
>correctly the reasoning was that LaTeX gave us more control and thus 
>served the purpose better for the official docs.

More control of what?  I thought that reST was specifically designed to 
accommodate all of the Python-specific markup we're using in the latex docs.

As a matter of language expressiveness, as far as I can tell, reST 
accomodates marking up both short strings and long blocks, with 
application-specific markup, so I don't really understand why there 
shouldn't be a largely 1:1 mapping between the markup systems.

To be fair, however, here are the downsides I'm aware of:

  * There are more tools that support editing LaTeX than reST

  * Python code would have to be written to replace the current 
Python-specific LaTeX macros with reST "interpreted text modes" and "block 
directives"

  * Conversion could be very time consuming, for the parts that can't be 
automated

  * There is a risk that the needed back-ends may not be sufficiently 
mature or feature-rich.  For example, I don't think there's an HTML 
back-end that can generate pages for different nodes, with navigation 
links.  I don't know if there's a GNU 'info' backend, either.

Anyway, I thought the whole point behind the docutils initiative was to 
create a doc format that could be used anywhere Python needs docs, ala 
Perl's POD.  If there are problems with reST, maybe we should let the 
docutils folks know what additional requirements exist, so that it can 
(eventually) be used for this purpose.

From tim.peters at gmail.com  Fri Nov  5 21:07:46 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov  5 22:54:39 2004
Subject: [Python-Dev] patch-finalizer vs Zope3
In-Reply-To: <20041104171957.A0C633B8038@smtp.zope.com>
References: <20041104171957.A0C633B8038@smtp.zope.com>
Message-ID: <1f7befae041105120727ae399e@mail.gmail.com>

[Tim Peters]
> FYI, and without taking time for non-trivial analysis.  I ran the non-ZEO
> Zope3 tests with a Python patched to treat weakrefs with callbacks as
> reachable (this was patch-finalizer.txt, attached to a msg I sent to
> python-dev last night).
> 
> The first test to leave stuff in gc.garbage then:
> 
>    The following test left garbage:
>    test_created (bugtracker.browser.tests.test_bug.BugBaseViewTest)
>    65
> 
> where 65 is len(gc.garbage) after the test finished.

OK, that actually wasn't the first test to produce garbage, it was
just the first time test.py *noticed* gc.garbage wasn't empty.  The
first test to produce garbage was BuddyCityState, and it produces
garbage in isolation:

C:\Code\Zope3>\code\python\PCbuild\python_d.exe test.py -vv . BuddyCityState
Configuration file found.
Running UNIT tests at level 1
Running UNIT tests from C:\Code\Zope3
BuddyCityState (buddydemo.buddy) ... ok
The following test left garbage:
BuddyCityState (buddydemo.buddy)
28 entries in gc.garbage:
  0: <weakref at 03C86F20; to 'InterfaceClass' at 012B5930 (IWriteContainer)>
      0x12B5930 is <InterfaceClass
zope.app.container.interfaces.IWriteContainer>
  1: <weakref at 03C8E080; to 'InterfaceClass' at 00B5EAB8 (Interface)>
      0xB5EAB8 is <InterfaceClass zope.interface.Interface>
  2: <weakref at 03C8E1A0; to 'Surrogate' at 03C8D4D0>
      0x3C8D4D0 is <Surrogate(<InterfaceClass
zope.app.container.interfaces.IWriteContainer>)>
  3: <weakref at 03C8E278; to 'InterfaceClass' at 012B5578 (IObjectEvent)>
      0x12B5578 is <InterfaceClass zope.app.event.interfaces.IObjectEvent>
  4: <weakref at 03C8E398; to 'Surrogate' at 03C8D6C8>
      0x3C8D6C8 is <Surrogate(<InterfaceClass
zope.app.event.interfaces.IObjectEvent>)>
  5: <weakref at 03C8E3E0; to 'InterfaceClass' at 01197230 (IHTTPRequest)>
      0x1197230 is <InterfaceClass zope.publisher.interfaces.http.IHTTPRequest>
  6: <weakref at 03C8E500; to 'InterfaceClass' at 0118ACE8 (IRequest)>
      0x118ACE8 is <InterfaceClass zope.publisher.interfaces.IRequest>
  7: <weakref at 03C8E620; to 'InterfaceClass' at 0118A818 (IPublisherRequest)>
      0x118A818 is <InterfaceClass zope.publisher.interfaces.IPublisherRequest>
  8: <weakref at 03C8E740; to 'InterfaceClass' at 0118A5E8
(IPublicationRequest)>
      0x118A5E8 is <InterfaceClass
zope.publisher.interfaces.IPublicationRequest>
  9: <weakref at 03C8E860; to 'InterfaceClass' at 00DB9690
(IPresentationRequest)>
      0xDB9690 is <InterfaceClass
zope.component.interfaces.IPresentationRequest>
 10: <weakref at 03C8E980; to 'Surrogate' at 03C8DAB8>
      0x3C8DAB8 is <Surrogate(<InterfaceClass
zope.component.interfaces.IPresentationRequest>)>
 11: <weakref at 03C8E9C8; to 'InterfaceClass' at 00DA0188 (IParticipation)>
      0xDA0188 is <InterfaceClass zope.security.interfaces.IParticipation>
 12: <weakref at 03C8EAE8; to 'Surrogate' at 03C8DB28>
      0x3C8DB28 is <Surrogate(<InterfaceClass
zope.security.interfaces.IParticipation>)>
 13: <weakref at 03C8EB30; to 'Surrogate' at 03C8DA10>
      0x3C8DA10 is <Surrogate(<InterfaceClass
zope.publisher.interfaces.IPublicationRequest>)>
 14: <weakref at 03C8EB78; to 'Surrogate' at 03C8DA10>
      0x3C8DA10 is <Surrogate(<InterfaceClass
zope.publisher.interfaces.IPublicationRequest>)>
 15: <weakref at 03C8EBC0; to 'Surrogate' at 03C8D968>
      0x3C8D968 is <Surrogate(<InterfaceClass
zope.publisher.interfaces.IPublisherRequest>)>
 16: <weakref at 03C8EC08; to 'InterfaceClass' at 0118AC08
(IApplicationRequest)>
      0x118AC08 is <InterfaceClass
zope.publisher.interfaces.IApplicationRequest>
 17: <weakref at 03C8ED28; to 'InterfaceClass' at 01177F18 (IEnumerableMapping)>
      0x1177F18 is <InterfaceClass
zope.interface.common.mapping.IEnumerableMapping>
 18: <weakref at 03C8EE48; to 'InterfaceClass' at 01177D58 (IReadMapping)>
      0x1177D58 is <InterfaceClass zope.interface.common.mapping.IReadMapping>
 19: <weakref at 03C8EF68; to 'InterfaceClass' at 01177C40 (IItemMapping)>
      0x1177C40 is <InterfaceClass zope.interface.common.mapping.IItemMapping>
 20: <weakref at 03C8F0C8; to 'Surrogate' at 03C8DD20>
      0x3C8DD20 is <Surrogate(<InterfaceClass
zope.interface.common.mapping.IItemMapping>)>
 21: <weakref at 03C8F110; to 'Surrogate' at 03C8DC78>
      0x3C8DC78 is <Surrogate(<InterfaceClass
zope.interface.common.mapping.IReadMapping>)>
 22: <weakref at 03C8F158; to 'Surrogate' at 03C8DBD0>
      0x3C8DBD0 is <Surrogate(<InterfaceClass
zope.interface.common.mapping.IEnumerableMapping>)>
 23: <weakref at 03C8F1A0; to 'Surrogate' at 03C8D9D8>
      0x3C8D9D8 is <Surrogate(<InterfaceClass
zope.publisher.interfaces.IApplicationRequest>)>
 24: <weakref at 03C8F1E8; to 'Surrogate' at 03C8D8C0>
      0x3C8D8C0 is <Surrogate(<InterfaceClass
zope.publisher.interfaces.IRequest>)>
 25: <weakref at 03C8F230; to 'Surrogate' at 03C8D8C0>
      0x3C8D8C0 is <Surrogate(<InterfaceClass
zope.publisher.interfaces.IRequest>)>
 26: <weakref at 03C8F278; to 'Surrogate' at 03C8D8C0>
      0x3C8D8C0 is <Surrogate(<InterfaceClass
zope.publisher.interfaces.IRequest>)>
 27: <weakref at 03C8F2C0; to 'Surrogate' at 03C8D818>
      0x3C8D818 is <Surrogate(<InterfaceClass
zope.publisher.interfaces.http.IHTTPRequest>)>
[703156 refs]

C:\Code\Zope3>

It appears generally true that, like in that test, gc.garbage gets
filled with (hundreds of thousands of) weakrefs specifically due to
adaptation machinery.  By inspection, there are a lot of weakrefs
flying around there <wink>.
From tim.peters at gmail.com  Fri Nov  5 23:06:41 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov  5 23:06:46 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <418A269B.6010207@livinglogic.de>
References: <41864572.4000806@livinglogic.de>
	<16774.45652.835103.327656@montanaro.dyndns.org>
	<4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de>
	<1f7befae04110210537c35e514@mail.gmail.com>
	<418A269B.6010207@livinglogic.de>
Message-ID: <1f7befae0411051406711618cf@mail.gmail.com>

[Walter D?rwald]
> I guess it's not worth it to try to fix doctest/trace to
> provide sourcecode for doctest code, but IMHO trace should
> be able to survive a doctest.

Sure!  The assert also triggers for "@test" on my box, BTW -- and also
did in 2.3.4.

> Removing the assert statement, so that trace.py runs to
> completion shows a different problem: There are only
> 32 files covered according to the trace output. The
> complete test log can be found here:
>    http://styx.livinglogic.de/~walter/brokentrace.txt
>
> The trace call looked like this:
> ./python ../../../trace.py --count --summary --missing Lib/test/regrtest.py
> with ../../../trace.py being the trace.py from current
> CVS with the assert statement removed.
>
> So am I doing something wrong or is trace.py broken?

Sorry, I don't know.  trace.py is like voting to me -- I'm highly in
favor of it, but never have time for it <0.5 wink>.  I only dropped in
here to explain the source of the synthesized doctest "file name".

FWIW, doing what you did with current CVS Python on Windows, I get
results similar to yours:  only 30-some modules reported at the end.

Under my 2.3.4 installation instead, the same thing reports on over
300 modules, so best guess is that trace.py is indeed suffering
breakage introduced in 2.4.  No idea about specifics, though.
From kbk at shore.net  Sat Nov  6 04:08:08 2004
From: kbk at shore.net (Kurt B. Kaiser)
Date: Sat Nov  6 04:08:15 2004
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200411060308.iA6388E3022594@h006008a7bda6.ne.client2.attbi.com>

Patch / Bug Summary
___________________

Patches :  241 open ( -2) /  2683 closed ( +5) /  2924 total ( +3)
Bugs    :  784 open ( -5) /  4567 closed (+22) /  5351 total (+17)
RFE     :  158 open ( +2) /   131 closed ( +0) /   289 total ( +2)

New / Reopened Patches
______________________

New BaseSMTPServer module  (2004-10-30)
       http://python.org/sf/1057417  opened by  Mark D. Roth

chr, ord, unichr documentation updates  (2004-10-31)
       http://python.org/sf/1057588  opened by  Mike Brown

bdist_rpm not able to compile multiple rpm packages  (2004-11-04)
       http://python.org/sf/1060577  opened by  Mihai Ibanescu

Patches Closed
______________

No ValueError for safe_substitute()  (2004-10-29)
       http://python.org/sf/1056967  closed by  bwarsaw

typo in comment (unicodeobject.h)  (2004-10-28)
       http://python.org/sf/1056231  closed by  rhettinger

ref. manual talks of 'sequence' instead of 'iterable'  (2003-10-23)
       http://python.org/sf/829073  closed by  fdrake

Fix for Unix macro expansion in citetitle  (2004-10-26)
       http://python.org/sf/1054715  closed by  fdrake

Remove CoreServices / CoreFoundation dependencies in core  (2004-09-26)
       http://python.org/sf/1035255  closed by  rhettinger

New / Reopened Bugs
___________________

add link in time module to calendar.timegm()  (2004-10-30)
CLOSED http://python.org/sf/1057535  opened by  Jeff Shute

compiler.transformer, "from module import *"  (2004-10-31)
       http://python.org/sf/1057835  opened by  Felix Wiemann

os.utime() doesn't work on WinME  (2004-11-01)
CLOSED http://python.org/sf/1057992  opened by  Raymond Hettinger

test_traceback.py fails on WinME  (2004-11-01)
CLOSED http://python.org/sf/1057993  opened by  Raymond Hettinger

Can't read some http URLs using neither urllib, nor urllib2  (2004-11-01)
       http://python.org/sf/1058059  opened by  Vsevolod Novikov

Blatantly false statement in the Unicode section  (2004-11-01)
CLOSED http://python.org/sf/1058351  opened by  Olivier Verdier

r'\10' as replacement pattern loops in compilation  (2004-11-02)
       http://python.org/sf/1058786  opened by  Nick Maclaren

Legacy requirements: Incorrect comments?  (2004-11-02)
       http://python.org/sf/1058937  opened by  Geoffrey T. Dairiki

install_lib fails under Python 2.1  (2004-11-02)
       http://python.org/sf/1058960  opened by  Geoffrey T. Dairiki

distutil bdist hardcodes the python location  (2004-11-02)
       http://python.org/sf/1059244  opened by  Steve Menard

errors=ignore broken on StreamWriter?  (2004-11-03)
CLOSED http://python.org/sf/1059748  opened by  Wummel

incorrect "'yield' outside function" error  (2004-11-04)
       http://python.org/sf/1060135  opened by  Willem Broekema

os.utime does not work on directories under Windows  (2004-11-04)
CLOSED http://python.org/sf/1060212  opened by  Boyle

Buffer overflow in socketmodule.c  (2004-11-04)
       http://python.org/sf/1060396  opened by  Mihai Ibanescu

Remove .min() and .max() from Decimal  (2004-11-04)
       http://python.org/sf/1060644  reopened by  facundobatista

Remove .min() and .max() from Decimal  (2004-11-04)
       http://python.org/sf/1060644  opened by  Facundo Batista

Error in difflib docs  (2004-11-05)
CLOSED http://python.org/sf/1060825  opened by  Kent Johnson

email.Parser does not recognize attachment  (2004-11-05)
CLOSED http://python.org/sf/1060941  opened by  Martin Muenstermann

Bugs Closed
___________

weakref callback vs gc vs threads  (2004-10-27)
       http://python.org/sf/1055820  closed by  tim_one

add link in time module to calendar.timegm()  (2004-10-31)
       http://python.org/sf/1057535  closed by  jlgijsbers

os.rmtree error handling was always broken  (2004-10-18)
       http://python.org/sf/1048941  closed by  jlgijsbers

minor error in os.access  (2004-10-29)
       http://python.org/sf/1056498  closed by  jlgijsbers

Attempt to run all atexit handlers  (2004-10-22)
       http://python.org/sf/1052242  closed by  montanaro

email.Utils not mentioned  (2004-09-17)
       http://python.org/sf/1030118  closed by  bwarsaw

os.utime() doesn't work on WinME  (2004-11-01)
       http://python.org/sf/1057992  closed by  rhettinger

test_traceback.py fails on WinME  (2004-11-01)
       http://python.org/sf/1057993  closed by  perky

email.Message does not allow iteration  (2004-08-26)
       http://python.org/sf/1017329  closed by  bwarsaw

Blatantly false statement in the Unicode section  (2004-11-02)
       http://python.org/sf/1058351  closed by  perky

File write examples are inadequate  (2002-10-09)
       http://python.org/sf/621057  closed by  fdrake

Clarify trailing comma in func arg list  (2003-09-01)
       http://python.org/sf/798652  closed by  fdrake

Install-time module compilation fails with 2.4b1  (2004-10-18)
       http://python.org/sf/1049003  closed by  loewis

errors=ignore broken on StreamWriter?  (2004-11-03)
       http://python.org/sf/1059748  closed by  lemburg

latex2html convert bug  (2003-12-30)
       http://python.org/sf/867556  closed by  quiver

os.utime does not work on directories under Windows  (2004-11-04)
       http://python.org/sf/1060212  closed by  tim_one

Remove .min() and .max() from Decimal  (2004-11-04)
       http://python.org/sf/1060644  closed by  rhettinger

LaTeX not required  (2004-05-05)
       http://python.org/sf/948517  closed by  fdrake

ambiguity in os.tmpfile  (2004-10-28)
       http://python.org/sf/1056515  closed by  fdrake

Duplicated &lt;link rel  (2004-02-18)
       http://python.org/sf/899423  closed by  fdrake

Error in difflib docs  (2004-11-05)
       http://python.org/sf/1060825  closed by  rhettinger

email.Parser does not recognize attachment  (2004-11-05)
       http://python.org/sf/1060941  closed by  bwarsaw

New / Reopened RFE
__________________

lock-free data structures?  (2004-11-03)
       http://python.org/sf/1059545  opened by  Markus Elfring

From martin at v.loewis.de  Sat Nov  6 11:10:45 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov  6 11:10:35 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path to
	a	release
In-Reply-To: <2m3bzo5xb8.fsf@starship.python.net>
References: <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de>
	<2m3bzo5xb8.fsf@starship.python.net>
Message-ID: <418CA325.50105@v.loewis.de>

Michael Hudson wrote:
> Err, do you mean latex2html?  If so, I'm not at all sure that's
> realistic.

We'll see. What I consider the difficult part of such a project
(parsing the TeX) is already implemented, so we only need the HTML
generation now. That should be straight-forward, even if laborious.

Regards,
Martin
From martin at v.loewis.de  Sat Nov  6 11:12:55 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov  6 11:12:44 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path to	a
	release
In-Reply-To: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
References: <41896727.7030206@v.loewis.de>
	<4188BE21.6040205@interlink.com.au>	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
Message-ID: <418CA3A7.709@v.loewis.de>

Phillip J. Eby wrote:
> I've dabbled in the guts of latex2html before; it's certainly not pretty.

I wouldn't look into the source of latex2html at all. Instead, I would
rewrite it from scratch, worrying only that the output stays the same
(or sufficiently similar).

> IMO a better long term option might be to use the Python docutils and 
> migrate to reStructuredText, since there are a bevy of backends 
> available for latex, PDF, HTML, etc.  They would probably need more work 
> before they'll be suitable to handle the entire doc generation process, 
> though.

I'm not convinced this would be a good idea. Having TeX as the primary
source allows to produce well-formatted printed documentation.

Regards,
Martin
From martin at v.loewis.de  Sat Nov  6 11:14:09 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov  6 11:13:58 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path
	to	a	release
In-Reply-To: <418BC694.9010306@activestate.com>
References: <41896727.7030206@v.loewis.de>	<4188BE21.6040205@interlink.com.au>	<41896727.7030206@v.loewis.de>	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<418BC694.9010306@activestate.com>
Message-ID: <418CA3F1.4050702@v.loewis.de>

Trent Mick wrote:
> What do people think about docbook?

It's been a plan for several years now, and I spend some time into
advancing the existing tools. Now I'm not convinced anymore that
this is desirable - people don't want to learn new markup languages
every day.

Regards,
Martin
From martin at v.loewis.de  Sat Nov  6 11:18:49 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov  6 11:18:39 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path	to	a
	release
In-Reply-To: <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>	<41896727.7030206@v.loewis.de>
	<4188BE21.6040205@interlink.com.au>	<41896727.7030206@v.loewis.de>	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
Message-ID: <418CA509.2050103@v.loewis.de>

Phillip J. Eby wrote:
> More control of what?  I thought that reST was specifically designed to 
> accommodate all of the Python-specific markup we're using in the latex 
> docs.

How do you create a module index and a "global" index in reST?
How do you express document inclusion (in the spirit of \input)?

> As a matter of language expressiveness, as far as I can tell, reST 
> accomodates marking up both short strings and long blocks, with 
> application-specific markup, so I don't really understand why there 
> shouldn't be a largely 1:1 mapping between the markup systems.

It's not about source code display. It is about the other 200
typographic features that we use in the Python documentation.

>  * Conversion could be very time consuming, for the parts that can't be 
> automated

If the conversion cannot be fully automatic, that would indicate that
there will be a loss of information. Likely, this loss is unacceptable.

> Anyway, I thought the whole point behind the docutils initiative was to 
> create a doc format that could be used anywhere Python needs docs, ala 
> Perl's POD.

I'm still uncertain what the point behind the docutils initiative is.

Regards,
Martin
From mwh at python.net  Sat Nov  6 13:58:10 2004
From: mwh at python.net (Michael Hudson)
Date: Sat Nov  6 13:58:12 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path to	a
	release
In-Reply-To: <418BC694.9010306@activestate.com> (Trent Mick's message of
	"Fri, 05 Nov 2004 10:29:40 -0800")
References: <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<418BC694.9010306@activestate.com>
Message-ID: <2mk6sz3zj1.fsf@starship.python.net>

Trent Mick <trentm@ActiveState.com> writes:

> Phillip J. Eby wrote:
>> I've dabbled in the guts of latex2html before; it's certainly not pretty.
>> IMO a better long term option might be to use the Python docutils
>> and migrate to reStructuredText, since there are a bevy of backends
>> available for latex, PDF, HTML, etc.  They would probably need more
>> work before they'll be suitable to handle the entire doc generation
>> process, though.
>
> What do people think about docbook?

Where did this conversation start again?  Is it just that generating
HTML currently involves "Probably the longest Perl script anyone ever
attempted to maintain."?

Suggestions for alternative documentation formats are a common theme
in Python-land; *reasons* for changing format are much harder to
find...

I think it is probably possible to remove Fred from the critical path
of performing a Python release, because due in large part to his work,
building the Python docs just isn't that difficult (heck, my
latex2html install hasn't rotted and died in *years*).  I'm pretty
sure it's possible on the starship, too.  Of course, if Fred doesn't
do the doc bits, someone else has to -- I don't know if Martin or
Anthony want to take on yet another task.  There might be something to
be said for finding someone in a more convenient timezone.

Is the process of scattering files around on creosote automated?

Cheers,
mwh

-- 
  <shapr> ucking keyoar
                                                -- from Twisted.Quotes
From ncoghlan at iinet.net.au  Sat Nov  6 16:09:47 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov  6 16:09:57 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path
	to	a	release
In-Reply-To: <418CA3A7.709@v.loewis.de>
References: <41896727.7030206@v.loewis.de>	<4188BE21.6040205@interlink.com.au>	<41896727.7030206@v.loewis.de>	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<418CA3A7.709@v.loewis.de>
Message-ID: <418CE93B.7010605@iinet.net.au>

Martin v. L?wis wrote:
> Phillip J. Eby wrote:
> 
>> I've dabbled in the guts of latex2html before; it's certainly not pretty.
> 
> 
> I wouldn't look into the source of latex2html at all. Instead, I would
> rewrite it from scratch, worrying only that the output stays the same
> (or sufficiently similar).

Being able to easily build the current docs on a Windows system would be 
convenient. Even a full install of Cygwin doesn't include all the tools 
needed to build by the current process (latex2html is the main offender 
- I believe it *can* be made to work with Cygwin, but it's a separate 
download that requires a few other tweaks, and installation of a couple 
more support tools)

>> IMO a better long term option might be to use the Python docutils and 
>> migrate to reStructuredText, since there are a bevy of backends 
>> available for latex, PDF, HTML, etc.  They would probably need more 
>> work before they'll be suitable to handle the entire doc generation 
>> process, though.
> 
> I'm not convinced this would be a good idea. Having TeX as the primary
> source allows to produce well-formatted printed documentation.

A _reST to TeX converter that added the extra typesetting info might be 
an interesting tool.

So things with simple typesetting needs can be written in _reST, while 
complex typesetting is still possible with all the power of TeX. (I 
don't see how _reST could be given the same typesetting power without 
losing its elegant simplicity).

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From pje at telecommunity.com  Sat Nov  6 17:34:55 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat Nov  6 17:34:09 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path
	to	a release
In-Reply-To: <418CA3A7.709@v.loewis.de>
References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com>

At 11:12 AM 11/6/04 +0100, Martin v. L?wis wrote:
>Phillip J. Eby wrote:
>>IMO a better long term option might be to use the Python docutils and 
>>migrate to reStructuredText, since there are a bevy of backends available 
>>for latex, PDF, HTML, etc.  They would probably need more work before 
>>they'll be suitable to handle the entire doc generation process, though.
>
>I'm not convinced this would be a good idea. Having TeX as the primary
>source allows to produce well-formatted printed documentation.

If what you mean is that TeX itself does better formatting than e.g. the 
docutils PDF backend, I would note that there are two LaTeX backends for 
docutils: one a generic latex backend, and the other that's supposed to 
generate markup suitable for the current Python documentation toolchain.

From pje at telecommunity.com  Sat Nov  6 17:51:20 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat Nov  6 17:50:34 2004
Subject: [Python-Dev] Re: interlocking dependencies on the
	path	to	a release
In-Reply-To: <418CA509.2050103@v.loewis.de>
References: <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com>

At 11:18 AM 11/6/04 +0100, Martin v. L?wis wrote:
>Phillip J. Eby wrote:
>>More control of what?  I thought that reST was specifically designed to 
>>accommodate all of the Python-specific markup we're using in the latex docs.
>
>How do you create a module index and a "global" index in reST?

By adding directives, or using interpreted text, as long as the feature is 
supported by a given output writer.


>How do you express document inclusion (in the spirit of \input)?

There's an "include" directive.  I don't know what you mean by the "spirit 
of \input".


>>As a matter of language expressiveness, as far as I can tell, reST 
>>accomodates marking up both short strings and long blocks, with 
>>application-specific markup, so I don't really understand why there 
>>shouldn't be a largely 1:1 mapping between the markup systems.
>
>It's not about source code display. It is about the other 200
>typographic features that we use in the Python documentation.

I don't get what source code display has to do with it.  I'm pointing out 
that the languages (Latex and reST) have comparable expressiveness in their 
markup facilities, e.g.:

Latex:   \foo{bar}

reST:    `bar`:foo


Latex:   \begin{foo}
          blah blah
          \end{foo}

reST:    .. foo::
             blah blah

The difference is merely that the meaning of reST's equivalents to macros 
and environments (i.e. "interpreted text roles" and "directives") are 
defined using Python code rather than Latex.  Of course, a latex writer 
could still be used to generate latex output, if that is the preferred 
format for printing.

By the way, please don't confuse my answering your questions here with me 
advocating an actual migration or conversion at this point: from the 
dicussion so far and my rechecking the status of the various available 
docutils writers, it's clear to me that the writers aren't yet mature 
enough to handle generating the Python documentation.  But I don't 
currently see any issues on the *reader* end.  That is, I have not yet seen 
any issues raised that rule out reST syntax as a documentation source 
format.  (I'm not saying there aren't any, either, just that I haven't seen 
such an issue raised yet.)

From martin at v.loewis.de  Sat Nov  6 18:35:03 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov  6 18:34:51 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path  to	a
	release
In-Reply-To: <5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com>
References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com>
Message-ID: <418D0B47.3010302@v.loewis.de>

Phillip J. Eby wrote:
> If what you mean is that TeX itself does better formatting than e.g. the 
> docutils PDF backend, I would note that there are two LaTeX backends for 
> docutils: one a generic latex backend, and the other that's supposed to 
> generate markup suitable for the current Python documentation toolchain.

That still might not be sufficient. reST certainly uses a specific 
subset of TeX's capabilities when generating LaTeX, so some features
of TeX may simply be unaccessible when you have to stick to reST
syntax.

Regards,
Martin
From martin at v.loewis.de  Sat Nov  6 18:42:58 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov  6 18:42:47 2004
Subject: [Python-Dev] Re: interlocking dependencies on the  path	to	a
	release
In-Reply-To: <5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com>
References: <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
	<5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com>
Message-ID: <418D0D22.4010405@v.loewis.de>

Phillip J. Eby wrote:
>> How do you create a module index and a "global" index in reST?
> 
> 
> By adding directives, or using interpreted text, as long as the feature 
> is supported by a given output writer.

Ah, so you would make use of extensions; i.e. we would essentially
clone reST.

>> How do you express document inclusion (in the spirit of \input)?
> 
> 
> There's an "include" directive.  I don't know what you mean by the 
> "spirit of \input".

If include means literal inclusion, it is in the spirit of \input.

> The difference is merely that the meaning of reST's equivalents to 
> macros and environments (i.e. "interpreted text roles" and "directives") 
> are defined using Python code rather than Latex.  Of course, a latex 
> writer could still be used to generate latex output, if that is the 
> preferred format for printing.

I did not understand that, in order to use reST in a complex
application, you have to write your own markup extensions, and
then support those in all the backends you care about.

If that is typical, then yes, reST should be able to represent
everything we have in the documentation right now. It just means
that we cannot expect reST to work *out-of-the-box*. Instead, we
first need to implement the missing formatting elements for the
backends we care for. That sounds like a significant project of
its own.

Regards,
Martin
From pje at telecommunity.com  Sat Nov  6 19:45:00 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat Nov  6 19:44:15 2004
Subject: [Python-Dev] Re: interlocking dependencies on the 
	path	to	a release
In-Reply-To: <418D0D22.4010405@v.loewis.de>
References: <5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com>
	<5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
	<5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20041106130940.035910a0@mail.telecommunity.com>

At 06:42 PM 11/6/04 +0100, Martin v. L?wis wrote:
>Phillip J. Eby wrote:
>>>How do you create a module index and a "global" index in reST?
>>
>>By adding directives, or using interpreted text, as long as the feature 
>>is supported by a given output writer.
>
>Ah, so you would make use of extensions; i.e. we would essentially
>clone reST.

If you mean clone the Python extensions to LaTeX, then yes.


>>The difference is merely that the meaning of reST's equivalents to macros 
>>and environments (i.e. "interpreted text roles" and "directives") are 
>>defined using Python code rather than Latex.  Of course, a latex writer 
>>could still be used to generate latex output, if that is the preferred 
>>format for printing.
>
>I did not understand that, in order to use reST in a complex
>application, you have to write your own markup extensions, and
>then support those in all the backends you care about.

Just as was done for LaTeX, yes.  (E.g. the Python latex2html 
postprocessing step, and the Python LaTeX macros.)


>If that is typical, then yes, reST should be able to represent
>everything we have in the documentation right now. It just means
>that we cannot expect reST to work *out-of-the-box*. Instead, we
>first need to implement the missing formatting elements for the
>backends we care for. That sounds like a significant project of
>its own.

Potentially so.  However, there are some significant shortcuts possible for 
many situations.  reST includes an unparameterized macro facility, a 'raw' 
directive that passes markup through to the backend, and a 'role' directive 
to create markup roles for use with interpreted text.  These roles apply 
"class" attributes to the DOM, that then end up as HTML or XML "class" 
attributes, or potentially as LaTeX macros or environments.  For example, 
one might use:

    .. role:: var(emphasis)

to define a text role that could be used to replace \var{foo} with :var:`foo`.

So, I think there are some regularities that can be exploited, such that 
it's not an N*M project, for N features and M backends.  Those features 
that can be implemented in terms of existing reST concepts (other than the 
'raw' passthru) would only need implementation once.  Indeed, if reST 
simply had a suitable directive for defining directives, I think that all 
of the merely typographical aspects of Python documentation markup could be 
defined in reST, as part of the documentation source.

The main main areas of "missing" features in docutils backends that I'm 
aware of would fall under the headings of index generation (all backends), 
and multi-page HTML output with navigation links (HTML backend).

From pje at telecommunity.com  Sat Nov  6 19:48:44 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat Nov  6 19:47:58 2004
Subject: [Python-Dev] Re: interlocking dependencies on the path 
	to	a release
In-Reply-To: <418D0B47.3010302@v.loewis.de>
References: <5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20041106134543.020f4c30@mail.telecommunity.com>

At 06:35 PM 11/6/04 +0100, Martin v. L?wis wrote:
>Phillip J. Eby wrote:
>>If what you mean is that TeX itself does better formatting than e.g. the 
>>docutils PDF backend, I would note that there are two LaTeX backends for 
>>docutils: one a generic latex backend, and the other that's supposed to 
>>generate markup suitable for the current Python documentation toolchain.
>
>That still might not be sufficient. reST certainly uses a specific subset 
>of TeX's capabilities when generating LaTeX, so some features
>of TeX may simply be unaccessible when you have to stick to reST
>syntax.

There's a 'raw' directive that can be used for this:

.. raw:: latex

    \setlength{\parindent}{0pt}

The data will be ignored by non-latex backends.  Similarly, you may use raw 
HTML with 'raw:: html'.

See 
http://docutils.sourceforge.net/docs/ref/rst/directives.html#raw-data-pass-through

From jcarlson at uci.edu  Sun Nov  7 00:05:44 2004
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun Nov  7 00:14:59 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
Message-ID: <20041106102107.F853.JCARLSON@uci.edu>


A recent patch to offer an SMTP server (SocketServer derivative) sparked
the below...

Question:
Does we (and by 'we' I mean those in charge of developing Python) want
to offer both asynchronous (deriving from asyncore, asynchat, etc.) and
synchronous versions of server software in the standard library?


Observations:
There is currently 1 async server, and 5 sync servers in the standard
library.

If we want to offer both sync and async versions, it makes sense to
reduce programmer effort and share as much code as possible.


Idea:
Set up a framework that allows Python programmers to produce async and
sync variants of their servers easily.  I have a partial example
'implementation' below.

It seems to make sense (if such a framework is desired) to place all
such modules in a 'Server' package, and to describe precisely what is
necessary to make such a package usable, used, and added to in a PEP
called something like "A flexible server framework".


Future Python:
This would obviously be a 2.5 feature/module, and I would be willing to
write both the PEP (perhaps after the 1st of January, 2005 as I have a
journal paper write before then) (and afterwards) translate the current
standard library servers to the framework.

Depending on how popular the framework is through the lifetime of Python
2.5 (if it makes it into 2.5), it may make sense to deprecate the
original servers in 2.6 or 2.7 (if ever).  I would be willing to
volunteer to maintain the server package, or at least as many of the
protocols as I understand, for 2.5 and beyond (I don't know much about
XMLRPC, so translation is about as much as I could probably do).


 - Josiah



In the case of 'chatty' protocols like SMTP, POP, etc., where the
following complete, merely a prototype...


class SyncServerBase:
    def send(self, data):
        if data is None:
            self.sock.close()
        else:
            self.sock.write(data)
    def close_when_done(self):
        self.send(None)
    ...

class AsyncServerBase(asyncore.dispatcher):
    def send(self, data):
        #handle buffering with deque, etc.
    def close_when_done(self):
        #toss a None into the queue
    ...

class AsynchatMixin(asynchat.async_chat):
    #perhaps write a better buffered reader and writer...
    limit = None
    def readable(self):
        #handle optional request size limit
        return limit is None or len(self.ac_in_buffer) < self.limit
    def found_terminator(self):
        self.request = self.ac_in_buffer
        self.ac_in_buffer = ''
        self.handle_request()
    ...

class SMTP_Handler:
    def SMTP_HELO(self, cmd):
        ....
    def SMTP_MAIL(self, cmd):
        ...
        self.send(response)
        ...
    ...

class SMTP_Sync(SyncServerBase, SMTP_Handler):
    def handle_request(self):
        #does all of the reading necessary to read from a socket and
        #handle dispatch to SMTP_*


class SMTP_Async(AsyncServerBase, AsynchatMixin, SMTP_Handler):
    terminator = '\r\n'
    def handle_request(self):
        #command line (or email content) is provided in self.request
        #handles dispatch to SMTP_*, reading a line/request is provided
        #in this case by the AsynchatMixin

Etcetera.
From tim.peters at gmail.com  Sun Nov  7 04:31:33 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Sun Nov  7 04:31:41 2004
Subject: [Python-Dev] patch-finalizer vs Zope3
In-Reply-To: <1f7befae041105120727ae399e@mail.gmail.com>
References: <20041104171957.A0C633B8038@smtp.zope.com>
	<1f7befae041105120727ae399e@mail.gmail.com>
Message-ID: <1f7befae04110619317dfe0f4d@mail.gmail.com>

[Tim, on a patch-finalizer consequence]
> OK, that actually wasn't the first test to produce garbage, it was
> just the first time test.py *noticed* gc.garbage wasn't empty.  The
> first test to produce garbage was BuddyCityState, and it produces
> garbage in isolation:
> ...

Man, this sure brings back miserable memories of how hard it could be
to debug leaks due to cycles before Neil added gcmodule.c!  The best
tool we had then was my Cyclops.py, but I stopped keeping that up to
date because there was obviously no more need for it <wink>.

Desktop boxes have gotten a lot more capable since then, and apps have
grown in ambition to match.  Here we ran one measly test from the huge
Zope3 test suite; it ran in an eye blink (maybe two).  From the first
weakref in gc.garbage alone, it turned out that more than 42,000
distinct cycles were reachable, breaking into a bit fewer than 4,000
strongly connected components, the latter ranging in size from 2
objects (almost all a new-style class and its mro) to about 40,000.  A
text file account of these SCCs consumed about 20MB.

So what you do?  I don't know.  These are always solvable, but they're
rarely easy.  I always end up writing a small pile of new graph
analysis code specifically exploiting peculiarities of the case at
hand.  In this case I also changed gcmodule.c a little, to put trash
reachable from resurrected trash weakrefs into gc.garbage too, but
unlike gc.DEBUG_SAVEALL not to also put *all* trash in gc.garbage. 
That was key to figuring out what the original trash cycles were.

"The answer" in this case appears to be in Zope3's
zope/interface/adapter.py's AdapterRegistry.__init__:

class AdapterRegistry(object):
   """Adapter registry
   """

   # Implementation note:
   # We are like a weakref dict ourselves. We can't use a weakref
   # dict because we have to use spec.weakref() rather than
   # weakref.ref(spec) to get weak refs to specs.

   _surrogateClass = Surrogate

   def __init__(self):
       default = self._surrogateClass(Default, self)
       self._default = default
       null = self._surrogateClass(Null, self)
       self._null = null

       # Create separate lookup object and copy it's methods
       surrogates = {Default.weakref(): default, Null.weakref(): null}
       def _remove(k):
           try:
               del surrogates[k]
           except KeyError:
               pass
       lookup = AdapterLookup(self, surrogates, _remove)

The rest doesn't appear to matter.  As the attached smoking gun shows,
a weakref W has an instance of the nested _remove function as its
callback(*).  The _remove function has a cell object, pointing to the
lexically enclosing `surrogates` dict.  That dict in turn has W as a
key.  So there's a cycle, and refcounting alone can't clean it up. 
All of this stuff *was* trash, but because patch-finalizer decides to
call W reachable, the cell and the `surrogates` dict became reachable
too, and none of it got cleaned up (nor, of course, did anything else
reachable from this cycle).

When 2.4b2 looks at this case, it clears W because its referent is
trash, and ignore's W's callback because W is also trash.  Since, had
it been called, the only thing the callback would have done is mutate
other trash, it didn't hurt not to call it.  But it remains a stretch
to believe that "ignoring callbacks on trash weakrefs is harmless" is
a necessary outcome.

Still, whatever else this may or may not imply, it convinces me we
can't adopt a patch-finalizer-ish approach without serious effort at
making leaks "due to it" more easily analyzable.

(*) A weakref W's callback is discoverable after all in Python, just
by calling gc.get_referents(W).  This is something I've rediscovered
about 6 times by now, and it always catches me by surprise <wink>.
-------------- next part --------------
cycle
obj 0:
cycle starts here
    0x3c84f20
    <type 'weakref'>
<weakref at 03C84F20; to 'InterfaceClass' at 012B8118 (IWriteContainer)>
    weakref to <InterfaceClass zope.app.container.interfaces.IWriteContainer>
obj 1:
    0x3c89a30
    <type 'function'>
<function _remove at 0x03C89A30>
obj 2:
    0x3c8b658
    <type 'tuple'>
(<cell at 0x03C8B968: dict object at 0x03C6A620>,)
obj 3:
    0x3c8b968
    <type 'cell'>
<cell at 0x03C8B968: dict object at 0x03C6A620>
obj 4:
    0x3c6a620
    <type 'dict'>
{<weakref at 020628F0; to 'InterfaceClass' at 015748F8 (Default)>: <Surrogate(<InterfaceClass zope.interface.adapter.Default>)>,
 <weakref at 020629C8; to 'InterfaceClass' at 01574A10 (Null)>: <Surrogate(<InterfaceClass zope.interface.adapter.Null>)>,
 <weakref at 03C84F20; to 'InterfaceClass' at 012B8118 (IWriteContainer)>: <Surrogate(<InterfaceClass zope.app.container.interfaces.IWriteContainer>)>,
 <weakref at 03C8C080; to 'InterfaceClass' at 00BD0188 (Interface)>: <Surrogate(<InterfaceClass zope.interface.Interface>)>,
 <weakref at 03C8C278; to 'InterfaceClass' at 012B3D20 (IObjectEvent)>: <Surrogate(<InterfaceClass zope.app.event.interfaces.IObjectEvent>)>,
 <weakref at 03C8C3E0; to 'InterfaceClass' at 011909D8 (IHTTPRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.http.IHTTPRequest>)>,
 <weakref at 03C8C500; to 'InterfaceClass' at 011904D0 (IRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IRequest>)>,
 <weakref at 03C8C620; to 'InterfaceClass' at 01185FC0 (IPublisherRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IPublisherRequest>)>,
 <weakref at 03C8C740; to 'InterfaceClass' at 01185D90 (IPublicationRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IPublicationRequest>)>,
 <weakref at 03C8C860; to 'InterfaceClass' at 00DB5D20 (IPresentationRequest)>: <Surrogate(<InterfaceClass zope.component.interfaces.IPresentationRequest>)>,
 <weakref at 03C8C9C8; to 'InterfaceClass' at 00D99818 (IParticipation)>: <Surrogate(<InterfaceClass zope.security.interfaces.IParticipation>)>,
 <weakref at 03C8CC08; to 'InterfaceClass' at 011903F0 (IApplicationRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IApplicationRequest>)>,
 <weakref at 03C8CD28; to 'InterfaceClass' at 0117A700 (IEnumerableMapping)>: <Surrogate(<InterfaceClass zope.interface.common.mapping.IEnumerableMapping>)>,
 <weakref at 03C8CE48; to 'InterfaceClass' at 0117A540 (IReadMapping)>: <Surrogate(<InterfaceClass zope.interface.common.mapping.IReadMapping>)>,
 <weakref at 03C8CF68; to 'InterfaceClass' at 0117A428 (IItemMapping)>: <Surrogate(<InterfaceClass zope.interface.common.mapping.IItemMapping>)>}
cycle ends here
    0x3c84f20
    <type 'weakref'>
<weakref at 03C84F20; to 'InterfaceClass' at 012B8118 (IWriteContainer)>
    weakref to <InterfaceClass zope.app.container.interfaces.IWriteContainer>



From tim.peters at gmail.com  Sun Nov  7 06:07:37 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Sun Nov  7 06:07:39 2004
Subject: [Python-Dev] patch-finalizer vs Zope3
In-Reply-To: <1f7befae04110619317dfe0f4d@mail.gmail.com>
References: <20041104171957.A0C633B8038@smtp.zope.com>
	<1f7befae041105120727ae399e@mail.gmail.com>
	<1f7befae04110619317dfe0f4d@mail.gmail.com>
Message-ID: <1f7befae04110621071e42e173@mail.gmail.com>

[Tim]
...
>       surrogates = {Default.weakref(): default, Null.weakref(): null}
>       def _remove(k):
>           try:
>               del surrogates[k]
>           except KeyError:
>               pass

FYI, if I fiddle that part to use the same kind of odd spelling
Python's weak dicts stumbled into for other reasons:

        surrogates = {Default.weakref(): default, Null.weakref(): null}
        self._surrogates = surrogates  # new
        def _remove(k, selfref=weakref.ref(self)):  # changed
            self = selfref()  # new
            if self is not None:  # new
                try:
                    del self._surrogates[k]  # changed
                except KeyError:
                    pass

then the Zope3 unit tests don't leave anything in gc.garbage under
2.4b2 + patch-finalizer.  A spattering of tests fail (2 failures + 3
errors), though, due to new

        TypeError: can't pickle weakref objects

exceptions.

Note that in this case the callbacks get invoked but are impotent: 
self isn't strongly reachable from the callbacks, so self is left in
the unreachable set, and so the `selfref` weakref (which does not have
a callback) gets cleared before gc starts breaking cycles:  all the
callbacks see selfref() return None.
From astrand at lysator.liu.se  Sun Nov  7 11:20:47 2004
From: astrand at lysator.liu.se (Peter Astrand)
Date: Sun Nov  7 11:20:54 2004
Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check
	for errors
In-Reply-To: <417EAD88.7080105@v.loewis.de>
References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se>
	<ulldubi9v.fsf@fitlinxx.com>
	<Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se>
	<417E81B1.8080601@v.loewis.de>
	<Pine.GSO.4.51L2.0410262044200.14637@koeberg.lysator.liu.se>
	<417EAD88.7080105@v.loewis.de>
Message-ID: <Pine.GSO.4.51L2.0411071118080.14526@koeberg.lysator.liu.se>

On Tue, 26 Oct 2004, "Martin v. L?wis" wrote:

> Peter Astrand wrote:
> > +                       PySys_WriteStderr("close failed: %s\n", strerror(errno));
>
> This should use HAVE_STRERROR.

Like this?:

diff -u -r2.192 fileobject.c
--- dist/src/Objects/fileobject.c       11 Jun 2004 04:49:03 -0000      2.192
+++ dist/src/Objects/fileobject.c       7 Nov 2004 10:13:04 -0000
@@ -300,12 +300,19 @@
 static void
 file_dealloc(PyFileObject *f)
 {
+       int sts = 0;
        if (f->weakreflist != NULL)
                PyObject_ClearWeakRefs((PyObject *) f);
        if (f->f_fp != NULL && f->f_close != NULL) {
                Py_BEGIN_ALLOW_THREADS
-               (*f->f_close)(f->f_fp);
+               sts = (*f->f_close)(f->f_fp);
                Py_END_ALLOW_THREADS
+               if (sts == EOF)
+#ifdef HAVE_STRERROR
+                       PySys_WriteStderr("close failed: [Errno %d] %s\n", errno, strerror(errno));
+#else
+                       PySys_WriteStderr("close failed: [Errno %d]\n", errno);
+#endif
        }
        PyMem_Free(f->f_setbuf);
        Py_XDECREF(f->f_name);


/Peter ?strand <astrand@lysator.liu.se>

From anthony at interlink.com.au  Sun Nov  7 15:32:35 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sun Nov  7 15:32:53 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <20041106102107.F853.JCARLSON@uci.edu>
References: <20041106102107.F853.JCARLSON@uci.edu>
Message-ID: <418E3203.9060002@interlink.com.au>

Josiah Carlson wrote:
> A recent patch to offer an SMTP server (SocketServer derivative) sparked
> the below...
> 
> Question:
> Does we (and by 'we' I mean those in charge of developing Python) want
> to offer both asynchronous (deriving from asyncore, asynchat, etc.) and
> synchronous versions of server software in the standard library?

Personally, I think that encouraging anyone to develop new software on
top of asyncore/asynchat is a _terrible_ idea.

Anthony

From jhylton at gmail.com  Sun Nov  7 16:09:34 2004
From: jhylton at gmail.com (Jeremy Hylton)
Date: Sun Nov  7 16:09:40 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <418E3203.9060002@interlink.com.au>
References: <20041106102107.F853.JCARLSON@uci.edu>
	<418E3203.9060002@interlink.com.au>
Message-ID: <e8bf7a5304110707093414cfa3@mail.gmail.com>

On Mon, 08 Nov 2004 01:32:35 +1100, Anthony Baxter
<anthony@interlink.com.au> wrote:
> Josiah Carlson wrote:
> > A recent patch to offer an SMTP server (SocketServer derivative) sparked
> > the below...
> >
> > Question:
> > Does we (and by 'we' I mean those in charge of developing Python) want
> > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and
> > synchronous versions of server software in the standard library?
> 
> Personally, I think that encouraging anyone to develop new software on
> top of asyncore/asynchat is a _terrible_ idea.

I agree.

Jeremy
From allison at sumeru.stanford.EDU  Sun Nov  7 17:11:20 2004
From: allison at sumeru.stanford.EDU (Dennis Allison)
Date: Sun Nov  7 17:11:38 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <e8bf7a5304110707093414cfa3@mail.gmail.com>
Message-ID: <Pine.LNX.4.10.10411070805390.25017-100000@sumeru.stanford.EDU>


The question, then, is what base should be used for developing asynchonous
servers?  

On Sun, 7 Nov 2004, Jeremy Hylton wrote:

> On Mon, 08 Nov 2004 01:32:35 +1100, Anthony Baxter
> <anthony@interlink.com.au> wrote:
> > Josiah Carlson wrote:
> > > A recent patch to offer an SMTP server (SocketServer derivative) sparked
> > > the below...
> > >
> > > Question:
> > > Does we (and by 'we' I mean those in charge of developing Python) want
> > > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and
> > > synchronous versions of server software in the standard library?
> > 
> > Personally, I think that encouraging anyone to develop new software on
> > top of asyncore/asynchat is a _terrible_ idea.
> 
> I agree.
> 
> Jeremy

From foom at fuhm.net  Sun Nov  7 17:13:59 2004
From: foom at fuhm.net (James Y Knight)
Date: Sun Nov  7 17:14:05 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <20041106102107.F853.JCARLSON@uci.edu>
References: <20041106102107.F853.JCARLSON@uci.edu>
Message-ID: <087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net>


On Nov 6, 2004, at 6:05 PM, Josiah Carlson wrote:

> A recent patch to offer an SMTP server (SocketServer derivative) 
> sparked
> the below...
>
> Question:
> Does we (and by 'we' I mean those in charge of developing Python) want
> to offer both asynchronous (deriving from asyncore, asynchat, etc.) and
> synchronous versions of server software in the standard library?

I'd like to introduce you to Twisted (http://www.twistedmatrix.com).

More cooperation and code-sharing between Twisted and Python core 
developers would certainly be nice. But I don't think the right 
direction to go for that is writing a bunch of half-assed async servers 
in Python core. I guarantee you it isn't going to be trivial to do 
nicely. There has been a small amount of discussion about contributing 
parts of the Twisted core into Python, but so far no one has 
volunteered to head that project. So, if you're interested, perhaps 
you'd like to take it on. I suspect even that will be a lot of work -- 
you'd have to get consensus on which parts, if any, would be included. 
Figure out acceptable solutions to versioning skew between twisted 
releases and python's included copy. Etc.

James

From arigo at tunes.org  Sun Nov  7 17:27:10 2004
From: arigo at tunes.org (Armin Rigo)
Date: Sun Nov  7 17:51:29 2004
Subject: [Python-Dev] segfault/Py_FatalError at exit with threads
Message-ID: <20041107162710.GA26684@vicky.ecs.soton.ac.uk>

Hi,

Someone familiar with pystate.c and PyGILState_*() should assign himself to
bug http://www.python.org/sf/1061968 -- I don't know all the intricaties of
the locking mecanisms to fix it myself...


Armin
From carribeiro at gmail.com  Sun Nov  7 18:00:26 2004
From: carribeiro at gmail.com (Carlos Ribeiro)
Date: Sun Nov  7 18:00:44 2004
Subject: [Python-Dev] Re: PEP 336, "Make None Callable",
	by Andrew McClelland
In-Reply-To: <41899E5D.2080303@interlink.com.au>
References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net>
	<41899E5D.2080303@interlink.com.au>
Message-ID: <864d3709041107090070849085@mail.gmail.com>

On Thu, 04 Nov 2004 14:13:33 +1100, Anthony Baxter
<anthony@interlink.com.au> wrote:
> 
> > Abstract
> >
> >     None should be a callable object that when called with any
> >     arguments has no side effect and returns None.
> 
> My response to this is simply "yuck". This is a hack, abd I
> don't think it's worthwhile. If you want to test for None, test
> for None. Similarly, if you want a dictionary lookup to default
> to a no-op method, there's a perfectly simple way to spell it now:
> 
>     def defaultNoopMethod(*args, **kwargs): pass
>     somedict.get(key, defaultNoopMethod)
> 
> You can even do this as a one-liner using a lambda if you want.

Just a question, and I sincerely hope it's not a dumb one. Python
already have a noop statement ("pass"). Now, what is being requested
is a standard noop callable. Does it make sense -- both in
theorethical and practical terms -- to allow a statement such as
"pass" to be used in both situations? In other words, does it make
sense to have a "unification" of sorts?

Thanks for any pointers,

-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro@gmail.com
mail: carribeiro@yahoo.com
From martin at v.loewis.de  Sun Nov  7 19:08:08 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun Nov  7 19:08:11 2004
Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check
	for errors
In-Reply-To: <Pine.GSO.4.51L2.0411071118080.14526@koeberg.lysator.liu.se>
References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se>	<ulldubi9v.fsf@fitlinxx.com>	<Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se>	<417E81B1.8080601@v.loewis.de>	<Pine.GSO.4.51L2.0410262044200.14637@koeberg.lysator.liu.se>	<417EAD88.7080105@v.loewis.de>
	<Pine.GSO.4.51L2.0411071118080.14526@koeberg.lysator.liu.se>
Message-ID: <418E6488.6060804@v.loewis.de>

Peter Astrand wrote:
> Like this?:

Yes, look good - please check it in.

Martin
From martin at v.loewis.de  Sun Nov  7 19:20:50 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun Nov  7 19:20:53 2004
Subject: [Python-Dev] Re: PEP 336, "Make None Callable",
	by Andrew McClelland
In-Reply-To: <864d3709041107090070849085@mail.gmail.com>
References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net>	<41899E5D.2080303@interlink.com.au>
	<864d3709041107090070849085@mail.gmail.com>
Message-ID: <418E6782.6060004@v.loewis.de>

Carlos Ribeiro wrote:
> Just a question, and I sincerely hope it's not a dumb one. Python
> already have a noop statement ("pass"). Now, what is being requested
> is a standard noop callable. Does it make sense -- both in
> theorethical and practical terms -- to allow a statement such as
> "pass" to be used in both situations? In other words, does it make
> sense to have a "unification" of sorts?

No. Theoretically, the "pass" statement exists for purely syntactical
purposes: namely, to put a statement where a statement is required, but
no meaningful statement can be found. As a result, "pass" has no
run-time semantics: it is not the case that it is executed and does
nothing, instead, it is simply not executed. The proposed "function
with no effect" would be quite different: it would have a run-time
semantics, which would be to ignore all arguments, and return None.

Practically, it would be very confusing if the keyword "pass" suddenly
started to denote a function. Traditionally, keywords have never
denoted expressions. It might be that True, False, and None become
keywords some day, but these would *only* be use as expressions.
It is unpythonic to give an expression meaning to a statement (just
as assignment is not an expression, either).

The simplest solution to achieve the rationale of the PEP would be
to add a function operator.nop. Of course, writing

def nop(*args):pass

isn't that more difficult than writing

from operator import nop

Regards,
Martin


From jcarlson at uci.edu  Sun Nov  7 19:39:43 2004
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun Nov  7 19:47:33 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net>
References: <20041106102107.F853.JCARLSON@uci.edu>
	<087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net>
Message-ID: <20041107093047.F85C.JCARLSON@uci.edu>


James Y Knight <foom@fuhm.net> wrote:
> 
> On Nov 6, 2004, at 6:05 PM, Josiah Carlson wrote:
> 
> > A recent patch to offer an SMTP server (SocketServer derivative) 
> > sparked
> > the below...
> >
> > Question:
> > Does we (and by 'we' I mean those in charge of developing Python) want
> > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and
> > synchronous versions of server software in the standard library?
> 
> I'd like to introduce you to Twisted (http://www.twistedmatrix.com).
> 
> More cooperation and code-sharing between Twisted and Python core 
> developers would certainly be nice. But I don't think the right 
> direction to go for that is writing a bunch of half-assed async servers 
> in Python core. I guarantee you it isn't going to be trivial to do 
> nicely. There has been a small amount of discussion about contributing 
> parts of the Twisted core into Python, but so far no one has 
> volunteered to head that project. So, if you're interested, perhaps 
> you'd like to take it on. I suspect even that will be a lot of work -- 
> you'd have to get consensus on which parts, if any, would be included. 
> Figure out acceptable solutions to versioning skew between twisted 
> releases and python's included copy. Etc.

It seems as though there has already been discussion of getting portions
of Twisted into the Python standard library.  Being as I have not
developed with Twisted, nor am I really a part of the Twisted community,
I'm not sure that I am really the right person to try to figure out what
parts of Twisted should or should not be included with Python 2.5.

Regardless of the asyncore vs Twisted issue, a proper (meta) framework
should separate out the calls for "what happens to the internal state
when the server receives a reqest like 'foo'", and how the data actually
gets to those calls.  If done correctly, if/when portions of Twisted
make it into the standard library, it shouldn't be terribly difficult to
offer a mechanism to use Twisted as a base (for someone with Twisted and
stdlib Twisted experience), if such a thing is desired.


I suppose the question comes down to; should there be a meta framework
for developing servers in Python that specifies a separation between the
protocol logic and how data gets to that protocol logic?  The desire is
for a mechanism to allow people to create a single version of a server
in their favorite framework (SocketServer, asyncore, Twisted, or other),
and for others to be able to use them in their favorite framework with
little difficulty.


 - Josiah

From pje at telecommunity.com  Sun Nov  7 20:15:57 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun Nov  7 20:15:21 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the
	standard library
In-Reply-To: <20041107093047.F85C.JCARLSON@uci.edu>
References: <087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net>
	<20041106102107.F853.JCARLSON@uci.edu>
	<087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net>
Message-ID: <5.1.1.6.0.20041107135655.020fbd00@mail.telecommunity.com>

At 10:39 AM 11/7/04 -0800, Josiah Carlson wrote:
>It seems as though there has already been discussion of getting portions
>of Twisted into the Python standard library.  Being as I have not
>developed with Twisted, nor am I really a part of the Twisted community,
>I'm not sure that I am really the right person to try to figure out what
>parts of Twisted should or should not be included with Python 2.5.
>
>Regardless of the asyncore vs Twisted issue, a proper (meta) framework
>should separate out the calls for "what happens to the internal state
>when the server receives a reqest like 'foo'", and how the data actually
>gets to those calls.  If done correctly, if/when portions of Twisted
>make it into the standard library, it shouldn't be terribly difficult to
>offer a mechanism to use Twisted as a base (for someone with Twisted and
>stdlib Twisted experience), if such a thing is desired.
>
>I suppose the question comes down to; should there be a meta framework
>for developing servers in Python that specifies a separation between the
>protocol logic and how data gets to that protocol logic?  The desire is
>for a mechanism to allow people to create a single version of a server
>in their favorite framework (SocketServer, asyncore, Twisted, or other),
>and for others to be able to use them in their favorite framework with
>little difficulty.

I believe that this may be an unrealistic goal, even though I have done 
something rather similar myself.  There is a myriad assortment of policy 
issues you'll have to swim upstream past, just to get to the mechanism 
issues.  For example, Twisted assumes its reactor is an interpreter-global, 
maybe even process-global event loop, whereas asyncore and peak.events 
allow multiple event loops.

I've dabbled in integrating Twisted and peak.events, such that code written 
for peak.events can run with or without Twisted, and such that an 
asynchronous server can be run synchronously.  So, it's certainly possible 
to do something like what you're describing, but I'm not sure it's worth 
the effort versus someone simply installing PEAK and/or Twisted.

Now, if the Python standard library included a lightweight task switching 
facility akin to Armin's greenlets, it might be more practical to include 
asynchronous communications facilities in the stdlib.  Essentially, one 
could implement a set of communication primitives that task-switched to a 
global scheduler, or one could supply the scheduler as part of the calls, 
or provide the primitives as methods of a scheduler, etc.  This would 
bypass most of the architectural policy issues.

However, without something like this, you are basically just going to be 
re-inventing either Twisted or peak.events, depending on whether you prefer 
a continuation-passing or generator-pseudothread architecture.

From bob at redivi.com  Sun Nov  7 20:19:09 2004
From: bob at redivi.com (Bob Ippolito)
Date: Sun Nov  7 20:19:17 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <20041107093047.F85C.JCARLSON@uci.edu>
References: <20041106102107.F853.JCARLSON@uci.edu>
	<087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net>
	<20041107093047.F85C.JCARLSON@uci.edu>
Message-ID: <E6A43674-30F1-11D9-9D1F-000A95BA5446@redivi.com>

On Nov 7, 2004, at 13:39, Josiah Carlson wrote:

> I suppose the question comes down to; should there be a meta framework
> for developing servers in Python that specifies a separation between 
> the
> protocol logic and how data gets to that protocol logic?  The desire is
> for a mechanism to allow people to create a single version of a server
> in their favorite framework (SocketServer, asyncore, Twisted, or 
> other),
> and for others to be able to use them in their favorite framework with
> little difficulty.

That's not really practical.  You can't use synchronous protocol code 
in an asynchronous server without threads or Stackless... so what's the 
point?  As for asynchronous frameworks that are compatible with the 
Python license, I'm only aware of asyncore (which people generally feel 
should be deprecated) and Twisted.

I think the best course of action would be to create some stripped down 
derivative of Twisted.  A couple changes off the top of my head would 
make it more suitable for the standard library:

-  Allow generators to be easily used for asynchrony (much easier to 
write synchronous-like code in that style, this has been demonstrated 
by several people)
-  Throw a blocking API on top because some people will want to use it 
in a blocking manner (Twisted's testing framework already has this 
functionality)
-  Allow more than one reactor per process (necessary to support the 
previous, because invariably people will use the blocking API from 
several threads)
-  Bring in some of the standard protocols (SMTP, HTTP, ...)  but not 
the higher-level versions when available (twisted.web) because those 
are so much more volatile.

-bob

From jcarlson at uci.edu  Sun Nov  7 22:37:10 2004
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun Nov  7 22:44:36 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <E6A43674-30F1-11D9-9D1F-000A95BA5446@redivi.com>
References: <20041107093047.F85C.JCARLSON@uci.edu>
	<E6A43674-30F1-11D9-9D1F-000A95BA5446@redivi.com>
Message-ID: <20041107132901.F868.JCARLSON@uci.edu>


Bob Ippolito <bob@redivi.com> wrote:
> That's not really practical.  You can't use synchronous protocol code 
> in an asynchronous server without threads or Stackless... so what's the 
> point?  As for asynchronous frameworks that are compatible with the 
> Python license, I'm only aware of asyncore (which people generally feel 
> should be deprecated) and Twisted.

At least in the standard library right now, there is really one major
difference between sync and async servers; how data gets to and from
functions that implement the logic.


> I think the best course of action would be to create some stripped down 
> derivative of Twisted.  A couple changes off the top of my head would 
> make it more suitable for the standard library:
> 
> -  Allow generators to be easily used for asynchrony (much easier to 
> write synchronous-like code in that style, this has been demonstrated 
> by several people)
> -  Throw a blocking API on top because some people will want to use it 
> in a blocking manner (Twisted's testing framework already has this 
> functionality)
> -  Allow more than one reactor per process (necessary to support the 
> previous, because invariably people will use the blocking API from 
> several threads)
> -  Bring in some of the standard protocols (SMTP, HTTP, ...)  but not 
> the higher-level versions when available (twisted.web) because those 
> are so much more volatile.

This all sounds reasonable to me, though I am certain that I am not the
right man for doing such a conversion.

 - Josiah

From jcarlson at uci.edu  Sun Nov  7 22:53:56 2004
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun Nov  7 23:01:36 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <5.1.1.6.0.20041107135655.020fbd00@mail.telecommunity.com>
References: <20041107093047.F85C.JCARLSON@uci.edu>
	<5.1.1.6.0.20041107135655.020fbd00@mail.telecommunity.com>
Message-ID: <20041107133719.F86E.JCARLSON@uci.edu>


"Phillip J. Eby" <pje@telecommunity.com> wrote:
> 
> I believe that this may be an unrealistic goal, even though I have done 
> something rather similar myself.  There is a myriad assortment of policy 
> issues you'll have to swim upstream past, just to get to the mechanism 
> issues.  For example, Twisted assumes its reactor is an interpreter-global, 
> maybe even process-global event loop, whereas asyncore and peak.events 
> allow multiple event loops.
> 
> I've dabbled in integrating Twisted and peak.events, such that code written 
> for peak.events can run with or without Twisted, and such that an 
> asynchronous server can be run synchronously.  So, it's certainly possible 
> to do something like what you're describing, but I'm not sure it's worth 
> the effort versus someone simply installing PEAK and/or Twisted.

Since most people are saying that this would be ugly (at least in the
realm of Twisted), I'll take back the offer.


> Now, if the Python standard library included a lightweight task switching 
> facility akin to Armin's greenlets, it might be more practical to include 
> asynchronous communications facilities in the stdlib.  Essentially, one 
> could implement a set of communication primitives that task-switched to a 
> global scheduler, or one could supply the scheduler as part of the calls, 
> or provide the primitives as methods of a scheduler, etc.  This would 
> bypass most of the architectural policy issues.

Re: greenlets

If someone were ambitious, one could write an implementation of
greenlets using standard threads for platforms without an optimized
greenlets implementation.

 - Josiah

From allison at sumeru.stanford.EDU  Mon Nov  8 00:33:35 2004
From: allison at sumeru.stanford.EDU (Dennis Allison)
Date: Mon Nov  8 00:33:45 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <20041107133719.F86E.JCARLSON@uci.edu>
Message-ID: <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>


It might be worthwhile to list what the existing problems are with the
asyncore and asynchat libraries.  There is also the problem that many
applications, Zope among them, depend upon them.


From andrew-pythondev at puzzling.org  Mon Nov  8 01:19:01 2004
From: andrew-pythondev at puzzling.org (Andrew Bennetts)
Date: Mon Nov  8 01:19:27 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>
References: <20041107133719.F86E.JCARLSON@uci.edu>
	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>
Message-ID: <20041108001901.GB477@frobozz>

On Sun, Nov 07, 2004 at 03:33:35PM -0800, Dennis Allison wrote:
> 
> It might be worthwhile to list what the existing problems are with the
> asyncore and asynchat libraries.  There is also the problem that many
> applications, Zope among them, depend upon them.

One such list is here:
    http://mail.zope.org/pipermail/zope3-dev/2002-October/003235.html

-Andrew.

From martin at v.loewis.de  Mon Nov  8 06:50:25 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon Nov  8 06:50:27 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <20041108001901.GB477@frobozz>
References: <20041107133719.F86E.JCARLSON@uci.edu>	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>
	<20041108001901.GB477@frobozz>
Message-ID: <418F0921.5040905@v.loewis.de>

Andrew Bennetts wrote:
> One such list is here:
>     http://mail.zope.org/pipermail/zope3-dev/2002-October/003235.html

I fail to see any of these as problematic:

1. invokes readable/writable in each round, thereby not preserving
    state; presumably ok for select and poll, but bad (wasting
    performance for kqueue).
    I can't see this as a problem: asyncore does use select/poll,
    not kqueue. In doing so, it first processes all ready file
    descriptors before going to the next round, so in the next
    round, it needs to check all dispatchers again.
    There seems to be an implicit assertion that anything that uses
    select/poll must be evil, and the only true API is kqueue.
    I can't claim to understand the rationale for introducing
    kqueue in the first place, but if it is to improve performance,
    then I expect that any performance gained in kqueue over
    select goes away by using Python (e.g. adding an indirection
    in dispatching is probably more expensive than what kqueue
    would have saved).

2. ties together protocol and transport. So what? I don't
    want to use SMTP over UDP, or X.25. TCP, possibly together
    with TLS, is just fine.

3. tied to sockets. Again: so what? I can't follow the assertion
    that you cannot use pyOpenSSL because of that, but then,
    I haven't used pyOpenSSL. The only possible interpretation
    is that pyOpenSSL does not expose "raw" (pollable) socket
    objects, which would really sound like a limitation in
    pyOpenSSL, not in asyncore.

a. Cannot use NT I/O completion ports. Again, what's wrong
    with select? Performance? I'd really like to see a Python
    application where the speed gain from IOCP is significant
    compared to using select. The limit of 64 sockets is serious,
    but Python bumps it to 512.

b. cannot do the the TCP transfer code in the C networking
    core. I don't really understand that point. What is the
    TCP transfer code?

Regards,
Martin
From ncoghlan at iinet.net.au  Mon Nov  8 10:51:14 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Mon Nov  8 10:51:18 2004
Subject: [Python-Dev] Re: PEP 336, "Make None Callable",
	by Andrew McClelland
In-Reply-To: <418E6782.6060004@v.loewis.de>
References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net>	<41899E5D.2080303@interlink.com.au>	<864d3709041107090070849085@mail.gmail.com>
	<418E6782.6060004@v.loewis.de>
Message-ID: <418F4192.7020306@iinet.net.au>

Martin v. L?wis wrote:
> The simplest solution to achieve the rationale of the PEP would be
> to add a function operator.nop. Of course, writing
> 
> def nop(*args):pass
> 
> isn't that more difficult than writing
> 
> from operator import nop

Although the def version above has problems with keyword arguments. 
Still, I'd be +0 on

def nop(*args, **kwds): pass

in the operator module, in the spirit of TOOWTDI.

And I meant to add a resounding 'Aye!' to Tim's comment about the 
debugging value of None not being callable - that's an obvious indicator 
of 'I screwed up', whereas silently returning None as proposed in the 
PEP may trigger the error report an arbitrary distance from the original 
mistake.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From jlg at dds.nl  Mon Nov  8 11:07:38 2004
From: jlg at dds.nl (Johannes Gijsbers)
Date: Mon Nov  8 11:05:08 2004
Subject: [Python-Dev] Bug Day in review
Message-ID: <20041108100738.GA5840@surfboy>

Well, Python Bug Day 4 was held yesterday. 12 patches and 10 bugs were
closed. Also, there are some bugs that are almost closable on
http://python.org/moin/PythonBugDayStatus. These need to be reviewed by
a committer.

Things I noticed:

* One one hand, no regressions were introduced, as far as I can tell.
  On the other hand, it's pretty hard to fix a lot of bugs if you're
  trying to keep regressions to a minimum.

* Most people just don't know where to start with fixing the bugs. We
  had a list of 'easy-to-fix' bugs on the second bug day, and it worked
  out pretty well. We should probably prepare one for the next one as
  well.

* The Python bug list may seem pretty large, but it isn't as bad it
  looks. Most of the bugs are either a corner case or a design
  limitation. Both types are hard to fix; there are very few glaringly
  obvious, easy-to-fix bugs left.

* I'm still too much of a newbie developer to run a bug day completely
  by myself. I can help out people with the process of fixing bugs, but
  I can't simultaneously learn about new parts of the Python source.

Conclusions for the next bug day:

* I'm willing to organize another bug day, but I'm not going to do it
  alone. A combination of a more experienced developer (for the hard
  technical decisions/discussions) and me (for introducing new
  developers to the process) would probably work out better.

* There should be a way to mark bugs as "easy-to-fix". I'll try to work
  in this into Roundup somehow, but until we migrate to that, I propose
  that if anyone (committer or otherwise) sees a bug that seems easy to
  fix for a new developer, (s)he should add it to the PythonBugDayStatus
  page.

* Right before a release probably isn't the best time to hold a bug day.

Cheers,

Johannes
From ncoghlan at iinet.net.au  Mon Nov  8 13:40:53 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Mon Nov  8 13:40:59 2004
Subject: [Python-Dev] Pre-PEP: Executing modules inside packages with '-m'
Message-ID: <418F6955.5000509@iinet.net.au>

A little later than planned, but I have a draft of the PEP to extend the 
-m switch to cover modules inside packages.

I will note that the SF reference implementation linked to by the PEP is 
marginally out of date - under certain circumstances (e.g. "python 
-mpychecker.checker") the current patch on SF can overwrite the C-level 
argv[0] which I've realised may not be a great idea given the existence 
of the Py_GetArgcArgv API (the PEP text describes the planned workaround).

Regards,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
-------------- next part --------------
PEP: XXX
Title: Executing modules inside packages with '-m'
Version: $Revision: 1.4 $
Last-Modified: $Date: 2003/09/22 04:51:50 $
Author: Nick Coghlan <ncoghlan@email.com>,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 16-Oct-2004
Python-Version: 2.5
Post-History: 16-Oct-2004


Abstract
========

This PEP defines semantics for executing modules inside packages as
scripts with the ``-m`` command line switch.

The proposed semantics are that the containing package be imported prior
to execution of the script.


Rationale
=========

Python 2.4 adds the command line switch ``-m`` to allow modules to be
located using the Python module namespace for execution as scripts.
The motivating examples were standard library modules such as ``pdb``
and ``profile``.

A number of users and developers have requested extension of the
feature to also support running modules located inside packages.
One example provided is pychecker's ``pychecker.checker`` module.
This capability was left out of the Python 2.4 implementation
because the appropriate semantics are not entirely clear.

The opinion on python-dev was that it was better to postpone the
extension to Python 2.5, and go through the PEP process to help
make sure we got it right.


Scope of this proposal
==========================

In Python 2.4, a module located using ``-m`` is executed just as if its
filename had been provided on the command line.  The goal of this PEP is
to get as close as possible to making that statement also hold true for
modules inside packages.

Prior discussions suggest it should be noted that this PEP is **not**
about any of the following:
- changing the idiom for making Python modules also useful as scripts.

- lifting the restriction of ``-m`` to modules of type PY_SOURCE or
  PY_COMPILED (i.e. ``.py``, ``.pyc``, ``.pyo``,``.pyw``).

- addressing the problem of ``-m`` not understanding zip imports or
  Python's sys.metapath.

The issues listed above are considered orthogonal to the specific
feature addressed by this PEP.


Current Behaviour
=================

Before describing the new semantics, it's worth covering the existing
semantics for Python 2.4 (as they are currently defined only by the
source code).

When ``-m`` is used on the command line, it immediately terminates the
option list (like ``-c``).  The argument is interpreted as the name of
a top-level Python module (i.e. one which can be found on ``sys.path``).

If the module is found, and is of type ``PY_SOURCE`` or ``PY_COMPILED``, then
the command line is effectively reinterpreted from ``python <options> -m
<module> <args>`` to ``python <options> <filename> <args>``. This includes
setting ``sys.argv[0]`` correctly (some scripts rely on this -
Python's own ``regrtest.py`` is one example).

If the module is not found, or is not of the correct type, an error
is printed.


Proposed Semantics
==================

The semantics proposed are fairly simple: if ``-m`` is used to execute
a module inside a package as a script, then the containing package is
imported before executing the module in accordance with the semantics
for a top-level module.

This is necessary due to the way Python's import machinery locates
modules inside packages.  A package may modify its own __path__ variable
during initialisation.  Accordingly, the only way for Python to reliably
locate the module is by importing the containing package and inspecting
its __path__ variable.

Note that the package is *not* imported into the ``__main__`` module's
namespace.  The effects of these semantics that will be visible to the
executed module are:

- the containing package will be in sys.modules

- any external effects of the package initialisation (e.g. installed
  import hooks)


Reference Implementation
========================

A reference implementation is available on SourceForge [1]_.  In this
implementation , if the ``-m`` switch fails to locate the requested
module at the top level, it effectively reinterprets the command
from ``python -m <script>`` to ``python -m execmodule <script>``.
(There is one caveat: when reinterpreted in this way, ``sys.argv[0]``
may not actually contain the filename of ``execmodule``. This only
affects ``execmodule`` itself, not the requested module).

``execmodule`` is a proposed standard library module that contains a single
function (also called ``execmodule``).  When invoked as a script, this
module finds and executes the module supplied as the first argument.  It
adjusts ``sys.argv`` by deleting ``sys.argv[0]`` and replacing the new
``sys.argv[0]`` with the module's filename instead of its Python name.

The function ``execmodule`` is like ``execfile``, but uses the Python
module namespace to locate the script instead of the filesystem. It
has an additional optional argument ``set_argv0`` which causes the
filename of the located module to be written to ``sys.argv[0]`` before
the module is executed.

A hybrid C/Python implementation is used as the Python module is much
more flexible and extensible than the equivalent C code would be. It
also allows the ``execmodule`` function to be made available.

The Python code for ``execmodule`` has also been posted as a
cookbook recipe for Python 2.4 [2]_.


Open Issues
===========

- choosing a name for the standard library module containing ``execmodule``.
  The reference implementation uses ``execmodule``.  An alternative name
  proposed on python-dev is ``runpy``.


Alternatives
============

The main alternative implementation considered ignored packages'
__path__ variables, and looked only in the main package directory.
However, this approach did not meet the main goal of the ``-m`` switch
- to allow the full Python namespace to be used to locate modules
for execution.


References
==========

.. [1] Native ``-m`` execmodule support
   (http://www.python.org/peps/pep-0009.html)

.. [2] execmodule Python Cookbook Recipe
   (http://www.python.org/peps/pep-0001.html)



Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:
From ncoghlan at iinet.net.au  Mon Nov  8 14:15:27 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Mon Nov  8 14:15:35 2004
Subject: [Python-Dev] Pre-PEP: Executing modules inside packages with '-m'
In-Reply-To: <418F6955.5000509@iinet.net.au>
References: <418F6955.5000509@iinet.net.au>
Message-ID: <418F716F.1030304@iinet.net.au>

Oops - forgot to update the links properly. Correct links are:


[1] Native ``-m`` execmodule support
http://sourceforge.net/tracker/?func=detail&aid=1043356&group_id=5470&atid=305470

[2] execmodule Python Cookbook Recipe
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From aleaxit at yahoo.com  Mon Nov  8 14:52:09 2004
From: aleaxit at yahoo.com (Alex Martelli)
Date: Mon Nov  8 14:52:23 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <418F0921.5040905@v.loewis.de>
References: <20041107133719.F86E.JCARLSON@uci.edu>	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>
	<20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de>
Message-ID: <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>


On 2004 Nov 08, at 06:50, Martin v. L?wis wrote:

> a. Cannot use NT I/O completion ports. Again, what's wrong
>    with select? Performance? I'd really like to see a Python
>    application where the speed gain from IOCP is significant

Isn't it the case that, on Windows, select can only use sockets while 
IOCP would also enable the use of other things such as files?  I 
remember that something like that used to be the case, but it's been a 
while since I did Windows, so my memory may be off or currently 
obsolete.  But if it's accurate it seems to me that -- on Windows -- 
select does not allow smooth integration of non-socket sources while 
IOCP might.


Alex

From exarkun at divmod.com  Mon Nov  8 16:34:34 2004
From: exarkun at divmod.com (exarkun@divmod.com)
Date: Mon Nov  8 16:34:39 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <418F0921.5040905@v.loewis.de>
Message-ID: <20041108153434.14104.15431039.divmod.quotient.1818@ohm>

On Mon, 08 Nov 2004 06:50:25 +0100, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin@v.loewis.de> wrote:
>Andrew Bennetts wrote:
> > One such list is here:
> >     http://mail.zope.org/pipermail/zope3-dev/2002-October/003235.html
> 
> I fail to see any of these as problematic:

  Before continuing this discussion, it might be useful to define the goals for whatever ultimately ends up in the standard library.  I believe there are a lot of differing implicit assumptions held by different posters on this topic which will make it very difficult to reach anything resembling concensus if not first resolved.

  Here are a few questions to get started (I'm sure there are more to be considered):

  What level of code re-use is desired?  Should a protocol implementation be portable between different frameworks (Remind anyone of PEP 333)?  Should new classes be required for different transports?  eg, a class for SMTP/TCP, a class for SMTP/OpenSSL-TLS, a class for SMTP/TLSLite-TLS, a class For SMTP/TLSLite-SSL.

  What level of performance is desirable?  Is the aim something comparable to the threading and forking based servers already in the stdlib?  Is it "sky's the limit"?  Should it be platform independent?

  Who is the target audience?  Will this be something beginning network programmers should be able to pick up and use reasonably easy?  Is that more important than allowing more complex software to be expressed easily?  (Most of the rest of these questions are basically subsets of this one)

  That said...

> 
> 1. invokes readable/writable in each round, thereby not preserving
>     state; presumably ok for select and poll, but bad (wasting
>     performance for kqueue).
>     I can't see this as a problem: asyncore does use select/poll,
>     not kqueue. In doing so, it first processes all ready file
>     descriptors before going to the next round, so in the next
>     round, it needs to check all dispatchers again.
>     There seems to be an implicit assertion that anything that uses
>     select/poll must be evil, and the only true API is kqueue.
>     I can't claim to understand the rationale for introducing
>     kqueue in the first place, but if it is to improve performance,
>     then I expect that any performance gained in kqueue over
>     select goes away by using Python (e.g. adding an indirection
>     in dispatching is probably more expensive than what kqueue
>     would have saved).

  KQueue's performance benefits over select() are absolutely noticable from Python.  We're talking orders of magnitude stuff here.  select() is fine for small servers, and poll() is even a little better, but unless you want to impose harsh, arbitrary limits on the scalability of servers (and certain kinds of clients, eg aggregators) developed in Python, these shouldn't be the only two options.  Even if KQueue isn't supported directly by the standard library, the mechanism should be amenable to support of it by third parties - this means an easily overridden notification call as well as supporting assumptions from the rest of the library.  Calling a function once per connection per iteration should be avoided if at all possible (and it is possible).

> 
> 2. ties together protocol and transport. So what? I don't
>     want to use SMTP over UDP, or X.25. TCP, possibly together
>     with TLS, is just fine.

  Keep in mind there is no support for SSL servers in the standard library (this is still true, right?  I admit I haven't been paying much attention to this lately).  So right off the bat, you _can't_ tie the transport to the protocol if you want to write an SSL server.  It needs to be trivially parameterizable by a third party.  Even ignoring that, there are several choices for SSL implementations in Python.  If it can be avoided, I don't see any reason to require a separate class for each of them for each protocol.

> 
> 3. tied to sockets. Again: so what? I can't follow the assertion
>     that you cannot use pyOpenSSL because of that, but then,
>     I haven't used pyOpenSSL. The only possible interpretation
>     is that pyOpenSSL does not expose "raw" (pollable) socket
>     objects, which would really sound like a limitation in
>     pyOpenSSL, not in asyncore.

  The point isn't "tied to sockets" but "tied to the socket module".  For a slightly less obvious example of why this is a drawback, consider that PyOpenSSL connections raise SSL.Error, not socket.error.  So if you want to use PyOpenSSL, you don't just have to find and replace all of the uses of socket.socket(), but all of the uses of socket.error as well.  Even worse, the behavior of SSL sockets doesn't entirely match the behavior of non-SSL sockets - select() can tell you a socket is ready for reading, and then a read can return '' because the available bytes were consumed entirely by SSL protocol-level negotiation.  asyncore will interpret this to mean the socket has been closed by the peer and call handle_close()!  So you really need an almost entirely different dispatcher to deal with SSL, and now we're back to the point above of having lots of different classes, one for each transport.

> 
> a. Cannot use NT I/O completion ports. Again, what's wrong
>     with select? Performance? I'd really like to see a Python
>     application where the speed gain from IOCP is significant
>     compared to using select. The limit of 64 sockets is serious,
>     but Python bumps it to 512.

  Performance, yes.  As Alex Martelli mentioned before me, IOCP also supports non-socket handles.  stdin/stdout, for example, as well as handles to subprocesses.  Again, since IOCP provides speedups comparable to KQueue when compared to select(), the speedups for large numbers of sockets is totally noticable, even from sluggish old Python ;)  Of course, you'd never notice them if your server could only handle 512 connections, but maybe you'd appreciate not having that limit, too :)

> 
> b. cannot do the the TCP transfer code in the C networking
>     core. I don't really understand that point. What is the
>     TCP transfer code?

  This would involve rewriting all of the socket handling, buffer handling, and even dispatch as an extension module.  Buffer handling is a real win here, since implementing it in Python involves a lot of wasteful string copying.  Itamar (original poster on the zope list) is now employed writing a lot of code like this for Twisted.  He has very high throughput requirements, and they'd be very difficult to achieve without the ability to replace this core component with his own code.  Since Twisted _is_ architected in such a way that this code is all one unit, instead of spread out among dozens, hundreds, or thousands of instances, after rewriting just a few classes, essentially any Twisted application can benefit from his speedup.

  Hope this clears some things up,

  Jp
From FBatista at uniFON.com.ar  Mon Nov  8 18:28:32 2004
From: FBatista at uniFON.com.ar (Batista, Facundo)
Date: Mon Nov  8 18:30:19 2004
Subject: [Python-Dev] Policy about old Python versions
Message-ID: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar>

[Aahz]
#- As soon as the new version is released, if there is no bugfix planned
#- for that issue.  Two versions later (i.e. 2.3 in this case) 
#- even if it
#- looks like it ought to be fixed in 2.1 -- we haven't been 
#- doing bugfixes
#- more than one version old.

[Martin v. L?wis]
#- It is (IMO) fine to close anything now that is fixed in 2.3 
#- and newer.
#- The proper "Resolution" is "Fixed", with a Comment "Fixed in 2.3".

[Anthony Baxter]
#- As for the original query, about what to do with bugs filed against
#- 2.1.x - I'd suggest closing them. There's a vanishingly small chance
#- that there will ever be another 2.1 or 2.2 release.

A bug count from SF:

Group  Quantity
2.1.1      2
2.1.2      3
2.2       20
2.2.1     17
2.2.1c     1
2.2.2     35
2.2.3     24
        -----
         102

Of 769 open bugs, that represents more than a 13%: I think that closing them
or reassigning them to a newer version (if the problem continues) will make
work with bugs a little easier.

So I propose to take the job of getting into each one of these bugs and
comment:

"""
Please, could you verify if this problem persists in Python 2.3.4 or 2.4?

If yes, in which version? Can you provide a test case?

If the problem is solved, from which version?

Note that if you fail to answer in one month, I'll close this bug as "Won't
fix".

Thank you!
"""

If nobody answers in a month, close the bug. The worst problem that we can
have here is to close a bug that still exists.

What do you think of this? If you're ok, I'll do it.


[Trent Mick]
#- Maybe a "Known Issues in Python 2.1" PEP which refers to these bug 
#- numbers and then close the bugs? I don't want to make more work for 
#- people, though.

This idea of compilate all the problems of those past versions in a PEP is
good, but I can not do it: in the best case I can compile all the "once
opened bugs" for those versions, but not verify that each is really a
problem.

A PEP with this kind of uncertainty worths it?


Thanks for everything.

.	Facundo


#- -----Mensaje original-----
#- De: Trent Mick [mailto:trentm@ActiveState.com]
#- Enviado el: Viernes, 05 de Noviembre de 2004 15:26
#- Para: Anthony Baxter
#- CC: Aahz; Batista, Facundo; Python Dev (E-mail)
#- Asunto: Re: [Python-Dev] Policy about old Python versions
#- 
#- 
#- Anthony Baxter wrote:
#- > As for the original query, about what to do with bugs filed against
#- > 2.1.x - I'd suggest closing them. There's a vanishingly 
#- small chance
#- > that there will ever be another 2.1 or 2.2 release.
#- 
#- Maybe a "Known Issues in Python 2.1" PEP which refers to these bug 
#- numbers and then close the bugs? I don't want to make more work for 
#- people, though.
#- 
#- Trent
#- 
#- -- 
#- Trent Mick
#- trentm@activestate.com
#- 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20041108/32052fcc/attachment-0001.html
From jimjjewett at yahoo.com  Mon Nov  8 18:55:02 2004
From: jimjjewett at yahoo.com (Jim J Jewett)
Date: Mon Nov  8 18:55:05 2004
Subject: [Python-Dev] Policy about old Python versions
Message-ID: <20041108175502.48541.qmail@web50701.mail.yahoo.com>

Batista, Facundo wrote:

[Trent Mick]
#- Maybe a "Known Issues in Python 2.1" PEP 
#- which refers to these bug numbers and then 
#- close the bugs? 

Batista, Facundo wrote:

# the best case I can compile all the "once
# opened bugs" for those versions, but not verify
# that each is really a problem.

Could you also indicate which ones were closed
only because they were too old?

In other words, if you Bug nnn was already
closed for other reasons, then either don't
include it on your list at all, or note in
your list that it was closed (instead of just
expiring).

-jJ



		
__________________________________ 
Do you Yahoo!? 
Check out the new Yahoo! Front Page. 
www.yahoo.com 
 

From fdrake at acm.org  Mon Nov  8 18:55:35 2004
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Mon Nov  8 18:55:43 2004
Subject: [Python-Dev] Bug Day in review
In-Reply-To: <20041108100738.GA5840@surfboy>
References: <20041108100738.GA5840@surfboy>
Message-ID: <200411081255.35450.fdrake@acm.org>

On Monday 08 November 2004 05:07 am, Johannes Gijsbers wrote:
 > * There should be a way to mark bugs as "easy-to-fix". I'll try to work
 >   in this into Roundup somehow, but until we migrate to that, I propose

Now *this* sort of functionality is a good reason to migrate to roundup.  I've 
generally not felt that there was any benefit to the migration, but being 
able to add this sort of community-support feature (assuming it actually 
happens) makes the cost of migration much more palatable, since there's an 
actual concrete benefit on the other side.  This makes a lot more sense than 
having to wrap an external document around the SourceForge trackers.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>

From gjc at inescporto.pt  Mon Nov  8 19:42:50 2004
From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro)
Date: Mon Nov  8 19:45:21 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the
	standard library
In-Reply-To: <Pine.LNX.4.10.10411070805390.25017-100000@sumeru.stanford.EDU>
References: <Pine.LNX.4.10.10411070805390.25017-100000@sumeru.stanford.EDU>
Message-ID: <1099939370.561.55.camel@spectrum.inescn.pt>

  Hello,

  I recently felt the need to have a main loop with timeouts and IO
notifications, such as the one found in pygtk's gobject module (wrapping
glib functions).  However, I did not want to add a dependency of pygtk
to this program.  Therefore I implemented a main loop module in python.

  I think that both asyncore and twisted are a bit too high-level.  I
like the low-level event driven approach much better.  It's easier to
understand.  That's why I wrote this.

  There's a unit test, and it seems to work, but this code is still
recent and not very well tested.

http://yang.inescn.pt/~gjc/pymainloop/

  Just so you know this exists...

  Regards.

Dom, 2004-11-07 ?s 08:11 -0800, Dennis Allison escreveu:
> The question, then, is what base should be used for developing asynchonous
> servers?  
> 
> On Sun, 7 Nov 2004, Jeremy Hylton wrote:
> 
> > On Mon, 08 Nov 2004 01:32:35 +1100, Anthony Baxter
> > <anthony@interlink.com.au> wrote:
> > > Josiah Carlson wrote:
> > > > A recent patch to offer an SMTP server (SocketServer derivative) sparked
> > > > the below...
> > > >
> > > > Question:
> > > > Does we (and by 'we' I mean those in charge of developing Python) want
> > > > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and
> > > > synchronous versions of server software in the standard library?
> > > 
> > > Personally, I think that encouraging anyone to develop new software on
> > > top of asyncore/asynchat is a _terrible_ idea.
> > 
> > I agree.
> > 
> > Jeremy
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/gjc%40inescporto.pt
-- 
Gustavo J. A. M. Carneiro
<gjc@inescporto.pt> <gustavo@users.sourceforge.net>
The universe is always one step beyond logic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3086 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20041108/6e607a5d/smime.bin
From FBatista at uniFON.com.ar  Mon Nov  8 19:49:46 2004
From: FBatista at uniFON.com.ar (Batista, Facundo)
Date: Mon Nov  8 19:51:34 2004
Subject: [Python-Dev] Policy about old Python versions
Message-ID: <A128D751272CD411BC9200508BC2194D053C7B7F@escpl.tcp.com.ar>

[Jim J Jewett]

#- # the best case I can compile all the "once
#- # opened bugs" for those versions, but not verify
#- # that each is really a problem.
#- 
#- Could you also indicate which ones were closed
#- only because they were too old?
#- 
#- In other words, if you Bug nnn was already
#- closed for other reasons, then either don't
#- include it on your list at all, or note in
#- your list that it was closed (instead of just
#- expiring).

Yes, in this old-bug-closing round I'll surely detail which bugs were closed
and the reason of each closure.

.	Facundo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20041108/52490e6c/attachment.htm
From jlg at dds.nl  Mon Nov  8 20:38:31 2004
From: jlg at dds.nl (Johannes Gijsbers)
Date: Mon Nov  8 20:44:40 2004
Subject: [Python-Dev] Policy about old Python versions
In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar>
References: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar>
Message-ID: <20041108193831.GA5944@surfboy>

On Mon, Nov 08, 2004 at 02:28:32PM -0300, Batista, Facundo wrote:
> So I propose to take the job of getting into each one of these bugs and
> comment:
> 
> """
> Please, could you verify if this problem persists in Python 2.3.4 or 2.4?
> 
> If yes, in which version? Can you provide a test case?
> 
> If the problem is solved, from which version?
> 
> Note that if you fail to answer in one month, I'll close this bug as "Won't
> fix".
> 
> Thank you!
> """

Sounds good to me, with the following caveat. Before adding this
comment, do a casual check whether the bug has been reproduced by
another developer and/or whether you can reproduce it yourself. Don't
spend too much time on it, but just check whether it's obvious the bug
still exists. Wouldn't want to see bugs like http://python.org/sf/548176
or http://python.org/sf/531205 get closed because of lack of response.

By the way: you know about the 'canned response' facility, right? Saves
you from copy and pasting the same response over and over again.

Cheers,

Johannes
From janssen at parc.com  Mon Nov  8 21:33:32 2004
From: janssen at parc.com (Bill Janssen)
Date: Mon Nov  8 21:33:46 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library 
In-Reply-To: Your message of "Sun, 07 Nov 2004 15:33:35 PST."
	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> 
Message-ID: <04Nov8.123335pst."58617"@synergy1.parc.xerox.com>

> It might be worthwhile to list what the existing problems are with the
> asyncore and asynchat libraries.  There is also the problem that many
> applications, Zope among them, depend upon them.

And Medusa, which is also widely used (though unsupported).  On the
Web-SIG list last year, I brought up bringing something like Medusa
(possibly even Medusa) into the standard library, to replace the
widely unused HTTP servers in there already.

Bill
From p.f.moore at gmail.com  Mon Nov  8 21:49:56 2004
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon Nov  8 21:50:05 2004
Subject: [Python-Dev] Policy about old Python versions
In-Reply-To: <20041108193831.GA5944@surfboy>
References: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar>
	<20041108193831.GA5944@surfboy>
Message-ID: <79990c6b0411081249198ffbbc@mail.gmail.com>

On Mon, 8 Nov 2004 20:38:31 +0100, Johannes Gijsbers <jlg@dds.nl> wrote:

> Sounds good to me, with the following caveat. Before adding this
> comment, do a casual check whether the bug has been reproduced by
> another developer and/or whether you can reproduce it yourself. Don't
> spend too much time on it, but just check whether it's obvious the bug
> still exists. Wouldn't want to see bugs like http://python.org/sf/548176
> or http://python.org/sf/531205 get closed because of lack of response.

FWIW, I tested these two bugs on 2.3 and 2.4b2, and added an
appropriate comment. If there are any other such bugs, it's easy
enough to do this yourself.

Paul
From FBatista at uniFON.com.ar  Mon Nov  8 21:57:13 2004
From: FBatista at uniFON.com.ar (Batista, Facundo)
Date: Mon Nov  8 21:59:07 2004
Subject: [Python-Dev] Policy about old Python versions
Message-ID: <A128D751272CD411BC9200508BC2194D053C7B89@escpl.tcp.com.ar>

[Johannes Gijsbers]

#- Sounds good to me, with the following caveat. Before adding this
#- comment, do a casual check whether the bug has been reproduced by
#- another developer and/or whether you can reproduce it yourself. Don't
#- spend too much time on it, but just check whether it's 
#- obvious the bug
#- still exists. Wouldn't want to see bugs like 
#- http://python.org/sf/548176
#- or http://python.org/sf/531205 get closed because of lack of 
#- response.

Ok.


#- By the way: you know about the 'canned response' facility, 
#- right? Saves
#- you from copy and pasting the same response over and over again.

I know, never used it, good oportunity to learn, ;).

.	Facundo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20041108/56c5223b/attachment.htm
From irmen at xs4all.nl  Mon Nov  8 22:25:54 2004
From: irmen at xs4all.nl (Irmen de Jong)
Date: Mon Nov  8 22:25:56 2004
Subject: [Python-Dev] Bug Day in review
In-Reply-To: <20041108100738.GA5840@surfboy>
References: <20041108100738.GA5840@surfboy>
Message-ID: <418FE462.2010708@xs4all.nl>

Johannes Gijsbers wrote:
> Well, Python Bug Day 4 was held yesterday. 12 patches and 10 bugs were
> closed. Also, there are some bugs that are almost closable on
> http://python.org/moin/PythonBugDayStatus. These need to be reviewed by
> a committer.

I was one of the participants, and I have to say that it was rather
fun to do and contribute some patches/bug comments.
And thanks to Johannes ofcourse for answering all my silly questions.

[ some words about myself: I've ported Python to AmigaDOS waaay back
in the python 1.5 days and so was quite familiar with the code at
that time. But years have passed since then and I've not looked at
the Python code anymore (besides the stdlib), instead I used Python
to create Pyro and more recently, Snakelets. While not being a
'python developer', it was nice to dig around in the Python source
code again on the bug day. New and yet familiar :-) ]

I've only subscribed to python-dev yesterday, and will mainly be
just lurking around to see what's going on.
(The bug-day has sparked my interest so to speak)

> * Most people just don't know where to start with fixing the bugs. We
>   had a list of 'easy-to-fix' bugs on the second bug day, and it worked
>   out pretty well. We should probably prepare one for the next one as
>   well.

That would be very nice indeed, I had some trouble finding things
that 1) I have some real knowledge of and 2) were 'easy-to-fix',
like you're saying. So the 'easy-to-fix' flag for bugs could
help a lot here, to make it easier for newbies like me to find
stuff that we can perhaps fix (or help to fix) during a bug day.
It's also easier for the organizers to find things to
stick on the work list, I guess.

One of the things that still puzzles me, is what you should do
with old bugs that are in groups 'Python 2.1 ... 2.3', but I saw
that there's already a discussion going on about this on this list :)


Bye,

--Irmen de Jong
From tjreedy at udel.edu  Mon Nov  8 22:26:35 2004
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon Nov  8 22:26:33 2004
Subject: [Python-Dev] Re: Policy about old Python versions
References: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar><20041108193831.GA5944@surfboy>
	<79990c6b0411081249198ffbbc@mail.gmail.com>
Message-ID: <cmooa2$thd$1@sea.gmane.org>

side note:

I find stuff like
&gt;&gt;&gt; from email.Utils import parseaddr
much harder to read than the proper
>>> from email.Utils import parseaddr
 http://python.org/sf/531205

Even worse is
 http://&lt;host&gt;:&lt;port&gt;/&lt;path&gt;?&lt;searchpart&gt;
versus something like
scheme>://<authority><path>?<query>
(both quoted from http://python.org/sf/548176)

SourceForge is designed (intentionally, I am sure)
to accept and properly display the latter
and not require chicken-scratch input.

Terry J. Reedy





From t-meyer at ihug.co.nz  Mon Nov  8 22:52:44 2004
From: t-meyer at ihug.co.nz (Tony Meyer)
Date: Mon Nov  8 22:53:31 2004
Subject: [Python-Dev] Re: Policy about old Python versions
In-Reply-To: <ECBA357DDED63B4995F5C1F5CBE5B1E801521491@its-xchg4.massey.ac.nz>
Message-ID: <ECBA357DDED63B4995F5C1F5CBE5B1E86C37CF@its-xchg4.massey.ac.nz>

> I find stuff like
> &gt;&gt;&gt; from email.Utils import parseaddr
> much harder to read than the proper
> >>> from email.Utils import parseaddr
>  http://python.org/sf/531205
> 
> Even worse is  
> http://&lt;host&gt;:&lt;port&gt;/&lt;path&gt;?&lt;searchpart&gt;
> versus something like
> scheme>://<authority><path>?<query>
>(both quoted from http://python.org/sf/548176)
>
> SourceForge is designed (intentionally, I am sure)
> to accept and properly display the latter
> and not require chicken-scratch input.

Actually, this is almost certainly the result of a SF glitch (from around
the start of the year, IIRC).  At that time, all SF trackers (comments,
summary, etc) had '<' converted to &lt; (etc).  There's no way to edit the
existing comments, so the trackers are permanently stuck like that.  The SF
people themselves indicated that they would not be able to fix any, except
manually, and that would only be done where there was a real need (I can't
think of any example of that).

You'll find examples of this in lots of project trackers that are suitably
old.  Python was probably hit harder than some, since the '>' character is
probably reasonably common in comments.

=Tony.Meyer

From greg at electricrain.com  Tue Nov  9 00:10:42 2004
From: greg at electricrain.com (Gregory P. Smith)
Date: Tue Nov  9 00:10:46 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <418F0921.5040905@v.loewis.de>
References: <20041107133719.F86E.JCARLSON@uci.edu>
	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>
	<20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de>
Message-ID: <20041108231042.GA21680@zot.electricrain.com>

On Mon, Nov 08, 2004 at 06:50:25AM +0100, "Martin v. L?wis" wrote:
> Andrew Bennetts wrote:
> >One such list is here:
> >    http://mail.zope.org/pipermail/zope3-dev/2002-October/003235.html
> 
> I fail to see any of these as problematic:
> 
> 1. invokes readable/writable in each round, thereby not preserving
>    state; presumably ok for select and poll, but bad (wasting
>    performance for kqueue).
>    I can't see this as a problem: asyncore does use select/poll,
>    not kqueue. In doing so, it first processes all ready file
>    descriptors before going to the next round, so in the next
>    round, it needs to check all dispatchers again.
>    There seems to be an implicit assertion that anything that uses
>    select/poll must be evil, and the only true API is kqueue.
>    I can't claim to understand the rationale for introducing
>    kqueue in the first place, but if it is to improve performance,
>    then I expect that any performance gained in kqueue over
>    select goes away by using Python (e.g. adding an indirection
>    in dispatching is probably more expensive than what kqueue
>    would have saved).

The complaint about the readable and writable methods is partially the
overhead of doing a method call for each file descriptor every time
thru the loop.  Its inefficient.  Itamar mentions that both select()
and poll() effectively maintain a reusable data structure that could
be maintained directly by the file descriptor event handlers rather
than expensively querying them all each time thru the loop.

In any given iteration thru the loop its typical for only a small
fraction of the file desciptors to change readable/writable state so
why not have them manage their flag directly.  (analogy: a push rather
than a pull of the readable/writable state to the event loop)

-g

From Andrew.MacKeith at ABAQUS.com  Mon Nov  8 21:46:04 2004
From: Andrew.MacKeith at ABAQUS.com (Andrew MacKeith)
Date: Tue Nov  9 06:32:24 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
References: <04Nov8.123335pst."58617"@synergy1.parc.xerox.com>
Message-ID: <418FDB0C.9090305@ABAQUS.com>

Bill Janssen wrote:
>>It might be worthwhile to list what the existing problems are with the
>>asyncore and asynchat libraries.  There is also the problem that many
>>applications, Zope among them, depend upon them.
> 
> 
> And Medusa, which is also widely used (though unsupported).  On the
> Web-SIG list last year, I brought up bringing something like Medusa
> (possibly even Medusa) into the standard library, to replace the
> widely unused HTTP servers in there already.

Unused?
I use them when I need a simple server.

Andrew

> 
> Bill
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/mackeith%40acm.org
> 



From martin at v.loewis.de  Tue Nov  9 06:49:06 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue Nov  9 06:49:08 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>
References: <20041107133719.F86E.JCARLSON@uci.edu>	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>	<20041108001901.GB477@frobozz>
	<418F0921.5040905@v.loewis.de>
	<6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>
Message-ID: <41905A52.2080507@v.loewis.de>

Alex Martelli wrote:
> But if it's accurate it seems to me that -- on Windows -- 
> select does not allow smooth integration of non-socket sources while 
> IOCP might.

I believe this is right. I don't see this as a problem, though - in the
typical asyncore application, what non-socket sources would you have?

Regards,
Martin
From martin at v.loewis.de  Tue Nov  9 07:07:56 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue Nov  9 07:07:56 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <20041108153434.14104.15431039.divmod.quotient.1818@ohm>
References: <20041108153434.14104.15431039.divmod.quotient.1818@ohm>
Message-ID: <41905EBC.2060107@v.loewis.de>

exarkun@divmod.com wrote:
> KQueue's performance benefits over select() are absolutely noticable
> from Python. We're talking orders of magnitude stuff here.  select()
> is fine for small servers

Can you quantify this a bit? How can I tell whether my server is small,
and how do I notice a the difference?

> Keep in mind there is no support for SSL servers in the standard
> library (this is still true, right?

True.

> If it can be avoided, I don't see any reason to require a separate
> class for each of them for each protocol.

I'm not sure it can be avoided. But then, I have not implemented any SSL
server in Python. I would think that
- in SMTP, you *must* reuse the socket, since the SSL modes starts
   only after the STARTTLS command
- in HTTP, you need to be SSL-aware, in order to pass the server
   key to the SSL library, and in order to validate the client
   certificate (if the client choses to send one)
IOW, whether or not SSL is used causes changes beyond the transport,
and specific to the protocol and application logic.

> asyncore will interpret this to mean
> the socket has been closed by the peer and call handle_close()! 

I see.

Regards,
Martin
From pje at telecommunity.com  Tue Nov  9 07:09:34 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue Nov  9 07:09:22 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the
	standard library
In-Reply-To: <41905A52.2080507@v.loewis.de>
References: <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>
	<20041107133719.F86E.JCARLSON@uci.edu>
	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>
	<20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de>
	<6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>
Message-ID: <5.1.1.6.0.20041109010737.02cd1710@mail.telecommunity.com>

At 06:49 AM 11/9/04 +0100, Martin v. L?wis wrote:
>Alex Martelli wrote:
>>But if it's accurate it seems to me that -- on Windows -- select does not 
>>allow smooth integration of non-socket sources while IOCP might.
>
>I believe this is right. I don't see this as a problem, though - in the
>typical asyncore application, what non-socket sources would you have?

Subprocess pipes would be one example, although I suppose you could rule 
that out as "atypical", but then that's just saying that anything you can't 
already do with it isn't typical use for it.  I have, however, written 
event-driven apps that talk to subprocess pipes and sockets at the same 
time.  Just not using asyncore.  :)

From exarkun at divmod.com  Tue Nov  9 08:12:30 2004
From: exarkun at divmod.com (Jp Calderone)
Date: Tue Nov  9 08:12:33 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <41905EBC.2060107@v.loewis.de>
Message-ID: <20041109071230.20649.1802519922.divmod.quotient.445@ohm>

On Tue, 09 Nov 2004 07:07:56 +0100, Martin v. Löwis <martin@v.loewis.de> wrote:
>exarkun@divmod.com wrote:
> > KQueue's performance benefits over select() are absolutely noticable
> > from Python. We're talking orders of magnitude stuff here.  select()
> > is fine for small servers
> 
> Can you quantify this a bit? How can I tell whether my server is small,
> and how do I notice a the difference?
> 

  See http://www.kegel.com/dkftpbench/Poller_bench.html#setup.solaris for some specific numbers, as well as a link to a simple benchmark tool you can try out (if you have a BSD machine).  None of these speak to Python overhead, but even assuming a huge number, like 30ms per event notification (which we choose to apply only to KQueue and not poll, because we're giving poll the benefit of the doubt), KQueue is already a win at 100 sockets.  Most important here is that KQueue (and related mechanisms - IOCP on Windows, and AIO on Linux (if it supported sockets ;)) scale with the number of active sockets, rather than the total number of sockets, as the other schemes do.  A KQueue server with 1000 quiescent clients and 1 active client is paying the same cost as a KQueue server with 1 active client; a poll server is paying between 500x and 1000x as much.

> > Keep in mind there is no support for SSL servers in the standard
> > library (this is still true, right?
> 
> True.
> 
> > If it can be avoided, I don't see any reason to require a separate
> > class for each of them for each protocol.
> 
> I'm not sure it can be avoided. But then, I have not implemented any SSL
> server in Python. I would think that
> - in SMTP, you *must* reuse the socket, since the SSL modes starts
>    only after the STARTTLS command

  SMTP/SSL actually exists; I don't believe it is terribly widely used, but it is an alternative to the STARTTLS ESMTP protocol action.  Let's ignore that for the moment though.  The case of switching between a plain TCP connection and an encrypted SSL connection is an interesting one.

  Say we have an SMTP/TCP server class, SMTP_TCP.  It inherits a buffering, error handling, etc, write() method from some helpful base class.  This version of write() handles socket.errors and deals with short send()s and so forth.  How can we write SMTP_TLS?  If it is a subclass of SMTP_TCP, then it inherits all the wrong socket behavior, because eventually it will need to deal with SSL send()s and recv()s.  We could create a mixin that provides an SSL-aware implementation of the write() method, but this will be wrong for the part of the connection that occurs before STARTTLS is issued.

  What options are there?  There are a few unspeakables (runtime class creation, runtime inheritence graph manipulation, etc), and just a couple that are halfway reasonable: override all of the socket-related methods, add a conditional that calls TCP implementations before STARTTLS and SSL implementations afterwards; or add bound methods as attributes to the instance when STARTTLS arrives, essentially overriding the TCP implementations with the SSL implementations.

  As for the first of these, it's a lot of code duplication, especially when you consider how many protocols have the concept of STARTTLS and will need to do the same thing.  A mixin could help reduce this, although the fragile dependence of this solution on the exact shape of the inheritence tree is something I've had to deal with and would not look forward to encountering again.  At best, it will be obscure.

  For the latter, well, that sounds like a lot of extra work to just duplicate what you'd get by dispatching calls to a 2nd object, and then replacing that object when STARTTLS arrives.  Kind of like...

    def cmd_STARTTLS(self):
        self.transport.write('220 Begin TLS negotiation now\r\n')
        self.transport = SSLConnection(self.transport)

  This strikes me as downright elegent, and would continue to do so even were I to forget what it is being compared to ;)

> - in HTTP, you need to be SSL-aware, in order to pass the server
>    key to the SSL library, and in order to validate the client
>    certificate (if the client choses to send one)
> IOW, whether or not SSL is used causes changes beyond the transport,
> and specific to the protocol and application logic.

  These can all be handled with relative ease.  Since I was so long winded above, I'll try and keep this short:

    def cmd_STARTTLS(self):
        if self.transport.canStartSSL():
            self.transport.write('220 Begin TLS negotiation now\r\n')
            self.transport = SSLConnection(self.transport)
        else:
            self.transport.write('454 TLS not available\r\n')

  The exact spelling of "self.transport.canStartSSL()" can go in any of a number of ways, but that is a different discussion. ;)

  Point being, graceful degradation of functionality is entirely possible, and arguably easier (or at least more logically structured) than the class-based solutions, which would simply present import errors in the absence of necessary SSL support, or would require the equivalent of the "self.transport.canStartSSL()" check somewhere out in the application code, rather than nicely encapsulated within the implementation of STARTTLS itself.

  Another way to put all of this is that by separating the protocol from the transport, the library is required to provide P + T classes, where P is the number of supported protocols and T is the number of supported transports.  By statically defining all permutations of the two, the library is required to provide P * T classes.  Aside from the extra effort required, many users won't be interested in anything near the full permutation space in any single application, and so will be just as happy to create the pairs themselves, as necessary.

  Jp
From lunz at falooley.org  Tue Nov  9 16:18:58 2004
From: lunz at falooley.org (Jason Lunz)
Date: Tue Nov  9 16:19:05 2004
Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the
	standard library
References: <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>
	<20041107133719.F86E.JCARLSON@uci.edu>
	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>
	<20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de>
	<6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>
	<41905A52.2080507@v.loewis.de>
	<5.1.1.6.0.20041109010737.02cd1710@mail.telecommunity.com>
Message-ID: <cmqn52$j1l$1@sea.gmane.org>

pje@telecommunity.com said:
> Subprocess pipes would be one example, although I suppose you could rule 
> that out as "atypical", but then that's just saying that anything you can't 
> already do with it isn't typical use for it.  I have, however, written 
> event-driven apps that talk to subprocess pipes and sockets at the same 
> time.  Just not using asyncore.  :)

I need to combine subprocess pipes and sockets in an async loop too. I
wouldn't call it atypical at all.

Jason

From db3l at fitlinxx.com  Wed Nov 10 00:34:34 2004
From: db3l at fitlinxx.com (David Bolen)
Date: Wed Nov 10 00:34:41 2004
Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the
	standard	library
References: <20041107133719.F86E.JCARLSON@uci.edu>
	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>
	<20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de>
	<6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>
	<41905A52.2080507@v.loewis.de>
Message-ID: <u4qjya96d.fsf@fitlinxx.com>

"Martin v. L?wis" <martin@v.loewis.de> writes:

> Alex Martelli wrote:
> > But if it's accurate it seems to me that -- on Windows -- 
> > select does not allow smooth integration of non-socket sources while
> > IOCP might.
> 
> I believe this is right. I don't see this as a problem, though - in the
> typical asyncore application, what non-socket sources would you have?

On Windows, I miss the ability to mix in serial port I/O the most,
since many of my servers are translators between the network and
serial devices.

-- David

From raymond.hettinger at verizon.net  Wed Nov 10 06:33:25 2004
From: raymond.hettinger at verizon.net (Raymond Hettinger)
Date: Wed Nov 10 06:35:45 2004
Subject: [Python-Dev] Deprecate PyRange_New()
Message-ID: <000001c4c6e6$cda46fa0$6628a044@oemcomputer>

A while back, Tim added a comment that PyRange_New() should be
deprecated in-part because it is filled with problems and is not used
anywhere in the core.

Doesn't anyone mind if I mark it in the C-API docs as deprecated so it
can be phased out later?


Raymond Hettinger

From martin at v.loewis.de  Wed Nov 10 08:36:22 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed Nov 10 08:36:23 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <20041109071230.20649.1802519922.divmod.quotient.445@ohm>
References: <20041109071230.20649.1802519922.divmod.quotient.445@ohm>
Message-ID: <4191C4F6.2040604@v.loewis.de>

Jp Calderone wrote:

> See http://www.kegel.com/dkftpbench/Poller_bench.html#setup.solaris
> for some specific numbers, as well as a link to a simple benchmark
> tool you can try out (if you have a BSD machine).  None of these
> speak to Python overhead, but even assuming a huge number, like 30ms
> per event notification (which we choose to apply only to KQueue and
> not poll, because we're giving poll the benefit of the doubt), KQueue
> is already a win at 100 sockets.  Most important here is that KQueue
> (and related mechanisms - IOCP on Windows, and AIO on Linux (if it
> supported sockets ;)) scale with the number of active sockets, rather
> than the total number of sockets, as the other schemes do.  A KQueue
> server with 1000 quiescent clients and 1 active client is paying the
> same cost as a KQueue server with 1 active client; a poll server is
> paying between 500x and 1000x as much.

Again, my question is: so what? If I have 1000 idle clients, and one
active client, it appears that poll will need 1ms per call (1184 ?s
on a 650 MHz dual Pentium). This is followed by the actual processing,
which will take *much* longer (depending on the nature of processing,
of course). A http server, for example, would need to parse the request,
locate the file, and start sending it back. This might take 10ms and
more.

Perhaps people really build Python servers which have simultaneously
30000 sockets open; I don't know.

> Say we have an SMTP/TCP server class, SMTP_TCP.  It inherits a
> buffering, error handling, etc, write() method from some helpful base
> class.  This version of write() handles socket.errors and deals with
> short send()s and so forth.  How can we write SMTP_TLS?  If it is a
> subclass of SMTP_TCP, then it inherits all the wrong socket behavior,
> because eventually it will need to deal with SSL send()s and recv()s.

I fail to see the problem. If the file object is replaced/rewrapped
into a SSL object, the base class can continue to use
read/write/send/recv - if only the SSL object supported the same
operations (which it really should if switching from plain sockets
to SSL sockets is a common task) (and you seem to be suggesting that
as a good strategy).

My point here is that you just cannot use the very same class for
SMTP, and SMTP+STARTTLS - you indicated that a subclass must be
created. This was my original point: you cannot instantiate the
baseclass for use with TLS, as the baseclass does not implement
the cmd_STARTTLS method.

> These can all be handled with relative ease.  Since I was so long
> winded above, I'll try and keep this short:
> 
> def cmd_STARTTLS(self): 
 >     if self.transport.canStartSSL():
>         self.transport.write('220 Begin TLS negotiation now\r\n') 
>         self.transport = SSLConnection(self.transport) 
 >     else:
>         self.transport.write('454 TLS not available\r\n')
> 
> The exact spelling of "self.transport.canStartSSL()" can go in any of
> a number of ways, but that is a different discussion. ;)

This cannot work, atleast not for the problem I was discussing:
In https, there is no starttls command to be implemented in a
subclass. Instead, the TLS negotiation occurs at connection
establishment, or transparently at any point during the
conversation.

The base class knows nothing about certificates, and client
authentification. So again, we need another class that deals
with these aspects of the protocol. Those aspects are *tied*
to the transport - so the resulting class will be
transport-specific.

> Another way to put all of this is that by separating the protocol
> from the transport, the library is required to provide P + T classes,
> where P is the number of supported protocols and T is the number of
> supported transports.  By statically defining all permutations of the
> two, the library is required to provide P * T classes.  Aside from
> the extra effort required, many users won't be interested in anything
> near the full permutation space in any single application, and so
> will be just as happy to create the pairs themselves, as necessary.

Who has required the library to provide P*T classes? That was a very
foolish design decision, as the users only need a small selection of
combinations. As a library designer, I would have created those
pairs that my users actually need.

Regards,
Martin

From martin at v.loewis.de  Wed Nov 10 08:37:28 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed Nov 10 08:37:27 2004
Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the
	standard	library
In-Reply-To: <u4qjya96d.fsf@fitlinxx.com>
References: <20041107133719.F86E.JCARLSON@uci.edu>	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>	<20041108001901.GB477@frobozz>
	<418F0921.5040905@v.loewis.de>	<6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>	<41905A52.2080507@v.loewis.de>
	<u4qjya96d.fsf@fitlinxx.com>
Message-ID: <4191C538.4050209@v.loewis.de>

David Bolen wrote:
> On Windows, I miss the ability to mix in serial port I/O the most,
> since many of my servers are translators between the network and
> serial devices.

Do you know whether IOCP could be used with the serial ports?

Regards,
Martin
From walter at livinglogic.de  Wed Nov 10 12:38:26 2004
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Wed Nov 10 12:38:29 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <1f7befae0411051406711618cf@mail.gmail.com>
References: <41864572.4000806@livinglogic.de>	
	<16774.45652.835103.327656@montanaro.dyndns.org>	
	<4187765D.3000705@livinglogic.de>
	<4187CA1C.9050508@livinglogic.de>	
	<1f7befae04110210537c35e514@mail.gmail.com>	
	<418A269B.6010207@livinglogic.de>
	<1f7befae0411051406711618cf@mail.gmail.com>
Message-ID: <4191FDB2.2000002@livinglogic.de>

Tim Peters wrote:
> [Walter D?rwald]
> 
>>I guess it's not worth it to try to fix doctest/trace to
>>provide sourcecode for doctest code, but IMHO trace should
>>be able to survive a doctest.
> 
> Sure!  The assert also triggers for "@test" on my box, BTW -- and also
> did in 2.3.4.

This seems to be fixed now.

> [...]
>>The trace call looked like this:
>>./python ../../../trace.py --count --summary --missing Lib/test/regrtest.py
>>with ../../../trace.py being the trace.py from current
>>CVS with the assert statement removed.
>>
>>So am I doing something wrong or is trace.py broken?
> 
> Sorry, I don't know.  trace.py is like voting to me -- I'm highly in
> favor of it, but never have time for it <0.5 wink>.  I only dropped in
> here to explain the source of the synthesized doctest "file name".
> 
> FWIW, doing what you did with current CVS Python on Windows, I get
> results similar to yours:  only 30-some modules reported at the end.
> 
> Under my 2.3.4 installation instead, the same thing reports on over
> 300 modules, so best guess is that trace.py is indeed suffering
> breakage introduced in 2.4.  No idea about specifics, though.

Using regrtest.py's tracing option instead of trace.py gives better
results (see http://styx.livinglogic.de/~walter/brokentrace2.txt):
656 covered modules (including a strange /tmp/tmp6ccfd4/t5/string.py)

There seems to be something wrong with the way trace.py starts
the script.

BTW, I'd like to have an option the specify the coverdir in regrtest.py,
as now finding the coverage file when you have the name of the Python
file is more complicated. But I guess this has to wait until 2.4 is
out the door.

Bye,
    Walter D?rwald

From aahz at pythoncraft.com  Wed Nov 10 15:38:36 2004
From: aahz at pythoncraft.com (Aahz)
Date: Wed Nov 10 15:38:39 2004
Subject: [Python-Dev] Deprecate PyRange_New()
In-Reply-To: <000001c4c6e6$cda46fa0$6628a044@oemcomputer>
References: <000001c4c6e6$cda46fa0$6628a044@oemcomputer>
Message-ID: <20041110143836.GB28182@panix.com>

On Wed, Nov 10, 2004, Raymond Hettinger wrote:
>
> A while back, Tim added a comment that PyRange_New() should be
> deprecated in-part because it is filled with problems and is not used
> anywhere in the core.
> 
> Doesn't anyone mind if I mark it in the C-API docs as deprecated so it
> can be phased out later?

+1

+1 on emitting a compiler warning in 2.5 (a bit late for 2.4, I think).
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

WiFi is the SCSI of the 21st Century -- there are fundamental technical
reasons for sacrificing a goat.  (with no apologies to John Woods)
From aahz at pythoncraft.com  Wed Nov 10 15:49:00 2004
From: aahz at pythoncraft.com (Aahz)
Date: Wed Nov 10 15:49:01 2004
Subject: [Python-Dev] What is docutils?
In-Reply-To: <418CA509.2050103@v.loewis.de>
References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
	<418CA509.2050103@v.loewis.de>
Message-ID: <20041110144859.GC28182@panix.com>

[Irregular side-note: could people please trim the Cc: line when
responding to python-dev posts?]

On Sat, Nov 06, 2004, "Martin v. L?wis" wrote:
>
> I'm still uncertain what the point behind the docutils initiative is.

It's very simple: to provide roughly 80% of the power of DocBook in a
human-readable text format.

Everything else comes from the uses people desire to put that capability
to.  An eventual goal is to make reST the standard markup for Python
comments, but it's actually not advanced enough for that purpose yet.
(That's my opinion, but of course docutils itself is mostly documented
using reST.)  People are using reST for many different other purposes,
though: writing books, wiki markup, web pages (see the recent
www.python.org partial changeover), and so on.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

WiFi is the SCSI of the 21st Century -- there are fundamental technical
reasons for sacrificing a goat.  (with no apologies to John Woods)
From phd at mail2.phd.pp.ru  Wed Nov 10 16:32:31 2004
From: phd at mail2.phd.pp.ru (Oleg Broytmann)
Date: Wed Nov 10 16:32:34 2004
Subject: [Python-Dev] Reply, Group-Reply,
	List-Reply (was: What is docutils?)
In-Reply-To: <20041110144859.GC28182@panix.com>
References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
	<418CA509.2050103@v.loewis.de> <20041110144859.GC28182@panix.com>
Message-ID: <20041110153230.GC22306@phd.pp.ru>

On Wed, Nov 10, 2004 at 09:49:00AM -0500, Aahz wrote:
> [Irregular side-note: could people please trim the Cc: line when
> responding to python-dev posts?]

   python-dev@ is almost always exactly in the Cc header. That because
people use Group-Reply (Reply-to-All) command in their mailers.
List-Reply is very unusual command in MUAs.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd@phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.
From barry at python.org  Wed Nov 10 17:17:09 2004
From: barry at python.org (Barry Warsaw)
Date: Wed Nov 10 17:17:19 2004
Subject: [Python-Dev] Reply, Group-Reply, List-Reply (was: What is
	docutils?)
In-Reply-To: <20041110153230.GC22306@phd.pp.ru>
References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au>
	<41896727.7030206@v.loewis.de>
	<5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com>
	<5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com>
	<418CA509.2050103@v.loewis.de> <20041110144859.GC28182@panix.com>
	<20041110153230.GC22306@phd.pp.ru>
Message-ID: <1100103429.20402.15.camel@geddy.wooz.org>

On Wed, 2004-11-10 at 10:32, Oleg Broytmann wrote:
> On Wed, Nov 10, 2004 at 09:49:00AM -0500, Aahz wrote:
> > [Irregular side-note: could people please trim the Cc: line when
> > responding to python-dev posts?]
> 
>    python-dev@ is almost always exactly in the Cc header. That because
> people use Group-Reply (Reply-to-All) command in their mailers.
> List-Reply is very unusual command in MUAs.

The best you can do is go to your Mailman option's page for python-dev
and select "Yes" for "Avoid duplicate copies of messages?".  The effect
of this is to have Mailman suppress sending you a list copy if your
email address is explicitly named in the recipient headers.  It's not
ideal, but it's all Mailman can do.  Getting people to trim recipients
is a hopeless battle.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: This is a digitally signed message part
Url : http://mail.python.org/pipermail/python-dev/attachments/20041110/793754d5/attachment.pgp
From foom at fuhm.net  Wed Nov 10 18:39:45 2004
From: foom at fuhm.net (James Y Knight)
Date: Wed Nov 10 18:39:53 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <4191C4F6.2040604@v.loewis.de>
References: <20041109071230.20649.1802519922.divmod.quotient.445@ohm>
	<4191C4F6.2040604@v.loewis.de>
Message-ID: <82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net>

On Nov 10, 2004, at 2:36 AM, Martin v. L?wis wrote:
> I fail to see the problem. If the file object is replaced/rewrapped
> into a SSL object, the base class can continue to use
> read/write/send/recv - if only the SSL object supported the same
> operations (which it really should if switching from plain sockets
> to SSL sockets is a common task) (and you seem to be suggesting that
> as a good strategy).

It can't work completely transparently; SSL has some more complicated 
requirements. I don't think it's really useful to go into SSL's inner 
workings here, but, a major issue is that reading from a socket can 
sometimes require writing to it and vice versa.

> My point here is that you just cannot use the very same class for
> SMTP, and SMTP+STARTTLS - you indicated that a subclass must be
> created. This was my original point: you cannot instantiate the
> baseclass for use with TLS, as the baseclass does not implement
> the cmd_STARTTLS method.

False. STARTTLS is part of the SMTP protocol and is always implemented, 
as shown in JP's message. Therefore, there is no subclass, STARTTLS is 
part of the main SMTP class.

But, the real point is that you can use the same class for SMTP and 
SMTP-over-ssl, which is fortunate, because you need to switch to the 
over-ssl implementation half way through the connection.

This discussion really seems useless to me. Asyncore certainly has some 
serious deficiencies, and arguing each and every one of them is okay 
because alternatively "nobody needs that feature" or "If X worked in 
some way (which it doesn't), then everything would be fine" just seems 
silly. I've only started developing with Twisted fairly recently, so I 
don't know the entire history of its development, but I'm pretty sure 
it wasn't started just for fun...

James
From jcarlson at uci.edu  Wed Nov 10 19:38:06 2004
From: jcarlson at uci.edu (Josiah Carlson)
Date: Wed Nov 10 19:46:46 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net>
References: <4191C4F6.2040604@v.loewis.de>
	<82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net>
Message-ID: <20041110100357.F8BE.JCARLSON@uci.edu>


James Y Knight <foom@fuhm.net> wrote:
> 
> On Nov 10, 2004, at 2:36 AM, Martin v. L?wis wrote:
> > My point here is that you just cannot use the very same class for
> > SMTP, and SMTP+STARTTLS - you indicated that a subclass must be
> > created. This was my original point: you cannot instantiate the
> > baseclass for use with TLS, as the baseclass does not implement
> > the cmd_STARTTLS method.
> 
> False. STARTTLS is part of the SMTP protocol and is always implemented, 
> as shown in JP's message. Therefore, there is no subclass, STARTTLS is 
> part of the main SMTP class.

False.  ESMTP has /optional/ support for STARTTLS, as defined in RFC
3207.  Neither SMTP nor ESMTP compliant mail servers need to implement
STARTTLS, regardless of their compliance to SMTP or ESMTP.


> But, the real point is that you can use the same class for SMTP and 
> SMTP-over-ssl, which is fortunate, because you need to switch to the 
> over-ssl implementation half way through the connection.

You statement is a no-op.  One could implement the greater part of most
any application in a single class, the question is about how much can be
shared.

At the minimal level, self.send() and self.recv() need to have different
implementations for the different with/without SSL variants (assuming
one would merely wrap the bare socket).  No surprise there.  At a higher
level, because SSL-ifying a socket is a blocking operation (requires at
least one round-trip if I remember correctly), discussion about async
SMTP+TLS is pretty moot at this point (unless one uses/abuses tlslite).


> This discussion really seems useless to me. Asyncore certainly has some 
> serious deficiencies, and arguing each and every one of them is okay 
> because alternatively "nobody needs that feature" or "If X worked in 
> some way (which it doesn't), then everything would be fine" just seems 
> silly. I've only started developing with Twisted fairly recently, so I 
> don't know the entire history of its development, but I'm pretty sure 
> it wasn't started just for fun...

Asyncore was written to scratch Sam Rushing's itch.  I'm not up on
Twisted's history.

 - Josiah

From bob at redivi.com  Wed Nov 10 20:16:11 2004
From: bob at redivi.com (Bob Ippolito)
Date: Wed Nov 10 20:16:21 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <20041110100357.F8BE.JCARLSON@uci.edu>
References: <4191C4F6.2040604@v.loewis.de>
	<82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net>
	<20041110100357.F8BE.JCARLSON@uci.edu>
Message-ID: <FB63B6FA-334C-11D9-B23B-000A95BA5446@redivi.com>

On Nov 10, 2004, at 13:38, Josiah Carlson wrote:

> James Y Knight <foom@fuhm.net> wrote:
>>
>> But, the real point is that you can use the same class for SMTP and
>> SMTP-over-ssl, which is fortunate, because you need to switch to the
>> over-ssl implementation half way through the connection.
>
> You statement is a no-op.  One could implement the greater part of most
> any application in a single class, the question is about how much can 
> be
> shared.
>
> At the minimal level, self.send() and self.recv() need to have 
> different
> implementations for the different with/without SSL variants (assuming
> one would merely wrap the bare socket).  No surprise there.  At a 
> higher
> level, because SSL-ifying a socket is a blocking operation (requires at
> least one round-trip if I remember correctly), discussion about async
> SMTP+TLS is pretty moot at this point (unless one uses/abuses tlslite).

Depends on what you mean by blocking.  As far as Twisted 
implementations are concerned, nothing is supposed to block Python code 
from executing unless it has nothing to do and it's sitting on a 
select(...) or equivalent (depends on which reactor you are using).

Such round trip operations are typically done with a Deferred: a 
placeholder for a success value or failure exception (wrapped in a 
Failure instance) that will be available sometime in the future.  When 
you call some function that needs to do a "blocking" operation, it will 
return a Deferred object.  Once you have that Deferred object you can 
attach a chain of success / failure callbacks to it.  When the producer 
of the Deferred object is ready to do so, it will start the callback 
chain by passing the Deferred object a the success or failure value.  
That value (well, the return value of the previous callback in the 
chain) will be passed along the chain until the chain ends.

The added bonus to using Deferreds is that they're not just for 
sockets.  Deferreds can be used to get the return value of a *blocking* 
function by using the "callInThread" mechanism, which dispatches this 
function to a thread from the reactor's thread pool and returns a 
Deferred object that will be passed the return value of that function.  
You can also use them to get the result of some long-running 
computation (that has been broken into small chunks, executing once per 
runloop iteration by way of a timer).

>> This discussion really seems useless to me. Asyncore certainly has 
>> some
>> serious deficiencies, and arguing each and every one of them is okay
>> because alternatively "nobody needs that feature" or "If X worked in
>> some way (which it doesn't), then everything would be fine" just seems
>> silly. I've only started developing with Twisted fairly recently, so I
>> don't know the entire history of its development, but I'm pretty sure
>> it wasn't started just for fun...
>
> Asyncore was written to scratch Sam Rushing's itch.  I'm not up on
> Twisted's history.

I can't speak for its original raison d'etre, but I moved from asyncore 
to Twisted because I ran into limitations in asyncore that Twisted had 
already taken care of.  As an added bonus, there was a larger more 
responsive community behind it, and a lot of existing protocol 
implementations (most if not all of which are more complete than any 
standard library counterpart, if one exists) that came standard with 
Twisted I could piggy-back off of.

Originally I had stayed away from Twisted because it was big and 
complicated in comparison to asyncore, but at some point a light went 
off in my head and I realized that it would make my applications 
SIGNIFICANTLY shorter, had methods for doing the things I needed to do 
that asyncore didn't do well, and solved some problems I didn't even 
know I would have.

-bob

From foom at fuhm.net  Wed Nov 10 20:18:38 2004
From: foom at fuhm.net (James Y Knight)
Date: Wed Nov 10 20:18:57 2004
Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard
	library
In-Reply-To: <20041110100357.F8BE.JCARLSON@uci.edu>
References: <4191C4F6.2040604@v.loewis.de>
	<82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net>
	<20041110100357.F8BE.JCARLSON@uci.edu>
Message-ID: <534381F4-334D-11D9-9244-000A95A50FB2@fuhm.net>


On Nov 10, 2004, at 1:38 PM, Josiah Carlson wrote:
> False.  ESMTP has /optional/ support for STARTTLS, as defined in RFC
> 3207.  Neither SMTP nor ESMTP compliant mail servers need to implement
> STARTTLS, regardless of their compliance to SMTP or ESMTP.

That is irrelevant. My point was, there is no need to make a subclass 
for STARTTLS. There is no architectural reason that it couldn't be in 
the same place as the other supported ESMTP commands. However, the 
original comment, and my response, and your response are all besides 
the main point, which is about SSL vs non-SSL sockets, not about 
whether you should have a separate class for SMTP and SMTP+some 
optional extensions.

>> But, the real point is that you can use the same class for SMTP and
>> SMTP-over-ssl, which is fortunate, because you need to switch to the
>> over-ssl implementation half way through the connection.
>
> You statement is a no-op.  One could implement the greater part of most
> any application in a single class, the question is about how much can 
> be
> shared.
>
> At the minimal level, self.send() and self.recv() need to have 
> different
> implementations for the different with/without SSL variants (assuming
> one would merely wrap the bare socket).  No surprise there.

The point is that the above assumption is a poor one. The protocol 
should not be wrapping the bare socket. With an architecture like 
Twisted, your protocol class doesn't have implementations of send and 
recv. Those belong to the transport, which you call. So, you do not 
have to derive from a different base class for SMTP-over-SSL and 
SMTP-over-TCP, and then do yucky things like switching your class 
halfway through the connection.

> At a higher
> level, because SSL-ifying a socket is a blocking operation (requires at
> least one round-trip if I remember correctly), discussion about async
> SMTP+TLS is pretty moot at this point (unless one uses/abuses tlslite).

That is completely incorrect. OpenSSL works perfectly well in async 
mode. Twisted implements async SSL __right now__.

Again, I do not think this is the appropriate place to be giving SSL 
lessons, but, yes, at least one roundtrip is required to setup an SSL 
session. But, this does __not__ mean your code has to block waiting for 
the roundtrip to complete. OpenSSL returns with an error code if it 
needs more read/write data to complete an operation. You get to take 
that information and feed it to select to wait for the condition to be 
fulfilled, and call OpenSSL again. This works.

James

From skip at pobox.com  Wed Nov 10 20:57:41 2004
From: skip at pobox.com (Skip Montanaro)
Date: Wed Nov 10 20:57:49 2004
Subject: [Python-Dev] Re: [development doc updates]
In-Reply-To: <20041110154700.5763718E89A@grendel.fdrake.net>
References: <20041110154700.5763718E89A@grendel.fdrake.net>
Message-ID: <16786.29365.808646.948039@montanaro.dyndns.org>


    Fred> Lots of changes to how tables are actually formatted, with much of
    Fred> the styling moved from the HTML to the CSS stylesheet.

    Fred> Tables also look nicer in supporting browsers.

Any idea what browsers those would be?

    Fred> Please review the tables and report any strange presentations; be
    Fred> sure to include the specific browser (including platform and
    Fred> version!) in your report.

Using Firefox 1.0PR on a Solaris system I checked out

    http://www.python.org/dev/doc/devel/lib/boolean.html

There are no left, right or top borders.  A PNG is at

    http://www.musi-cal.com/~skip/table.png

Skip
From fdrake at acm.org  Wed Nov 10 21:33:23 2004
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Wed Nov 10 21:33:33 2004
Subject: [Python-Dev] Re: [development doc updates]
In-Reply-To: <16786.29365.808646.948039@montanaro.dyndns.org>
References: <20041110154700.5763718E89A@grendel.fdrake.net>
	<16786.29365.808646.948039@montanaro.dyndns.org>
Message-ID: <200411101533.23469.fdrake@acm.org>

On Wednesday 10 November 2004 02:57 pm, Skip Montanaro wrote:
 >     Fred> Lots of changes to how tables are actually formatted, with much
 > of Fred> the styling moved from the HTML to the CSS stylesheet.
 >
 >     Fred> Tables also look nicer in supporting browsers.
 >
 > Any idea what browsers those would be?

All graphical browsers I tested with until I booted a Windows machine.  ;-)

Specifically:  Mozilla 1.8a4 on Linux, Opera 7.21 on Linux, Galeon 1.2.13 (yet 
another Gecko wrapper) on Linux.

Things looked fine on links, lynx, and w3m as well, though I don't expect they 
looked better than before.  ;-)

 >     Fred> Please review the tables and report any strange presentations;
 > be Fred> sure to include the specific browser (including platform and
 > Fred> version!) in your report.
 >
 > Using Firefox 1.0PR on a Solaris system I checked out
 >
 >     http://www.python.org/dev/doc/devel/lib/boolean.html
 >
 > There are no left, right or top borders.  A PNG is at

That's right.  I liked that better myself; fewer extra lines.  The lines below 
the header and at the bottom of the table are slightly heavier than the other 
lines as well.

I'm surprised that the first and last columns aren't centered, and that the 
heading for the second column is.  I'll need to review more of the actual 
tables in the documents, though the tables I used are quite similar to this 
one.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>

From bac at OCF.Berkeley.EDU  Thu Nov 11 00:47:42 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Thu Nov 11 00:47:57 2004
Subject: [Python-Dev] python-dev Summary for 2004-10-01 through 2004-10-15
	[draft]
Message-ID: <4192A89E.1090305@ocf.berkeley.edu>

No, that subject does not have a typo; this is for the first half of October. 
Have two huge projects due at the end of the quarter and they are killing me. 
Plus I have doctoral apps to prep for and do (and suggestions on good schools 
in the US or even Canada would be great).

Anyway, expect major delays in getting more Summaries out of the door until I 
am done with this quarter in December.

-----------------------------------

=====================
Summary Announcements
=====================
Sorry for the very late and short summary but I am in extreme crunch mode right 
now with school and preparing for applications to doctorate programs (any 
recommendations for schools with a good CS dept. and language research would be 
appreciated).  Don't expect things to pick up again until the quarter is over 
(Dec. 9th is my last final).


=========
Summaries
=========
----------------------
New module: subprocess
----------------------
Peter ?strand wrote `PEP 324`_ to come up with a platform-independent way to 
create processes.  Originally entitled popen5, it has now been added to the 
stdlib as subprocess.

.. _PEP 324: http://www.python.org/peps/pep-0324.html

Contributing threads:
   - `subprocess - updated popen5 module <>`__
   - `python/dist/src/Lib subprocess.py,NONE,1.1 <>`__
   - `Wither docs for the subprocess module? <>`__

--------------------
Python 2.4.b1 is out
--------------------
`Python 2.4b1`__ has been released.  As usual please download it, run the test 
suite, and test it with your own code.

Contributing threads:
   - `2.4b1 release,	this Friday (AU time) -- imminent TRUNK FREEZING <>`__
   - `RELEASED Python 2.4, beta 1 <>`__
   - `TRUNK UNFROZEN (release is done) <>`__

--------------------------------------
We need a replacement for ConfigParser
--------------------------------------
David Goodger brought forward two patches for ConfigParser he wanted to apply. 
  This quickly brought up the point that ConfigParser's API is not exactly 
optimal.  So the decision was made to field new designs.  If you happen to have 
your own code lying around that handles .ini files nicely then get it cleaned 
up and consider proposing at as a replacement.

Some ideas have already been floated on the list but will have to wait for the 
next summary for mention/discussion.

Contributing threads:
   - `ConfigParser patches <>`__


===============
Skipped Threads
===============
- Guidelines for Logging Usage
- Status of undocumented distutils features in 2.4?
- Toward Python's future article
- test_pep263.py is "-kb" in CVS
- python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109
- Python on Windows 64bits?
- About the Wise installer on Windows
- Patch wanted
- python/dist/src/Objects unicodeobject.c, 2.228, 2.229
- Minor cookielib changes for 2.4b1
- proposed struct module format code addition
- Cyclic GC issues
     Just don't have the time to summarize this, but a in-depth discussion of 
garbage collection involving extension code and using non-Python structs.
- python-dev Summary for 2004-09-16 through 2004-09-30 [draft]
     Lead to a discussion of executing modules in packages using the new '-m' 
command-line flag.
From db3l at fitlinxx.com  Thu Nov 11 02:28:46 2004
From: db3l at fitlinxx.com (David Bolen)
Date: Thu Nov 11 02:28:54 2004
Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the
	standard	library
References: <20041107133719.F86E.JCARLSON@uci.edu>
	<Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU>
	<20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de>
	<6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com>
	<41905A52.2080507@v.loewis.de> <u4qjya96d.fsf@fitlinxx.com>
	<4191C538.4050209@v.loewis.de>
Message-ID: <uvfcdnpgx.fsf@fitlinxx.com>

"Martin v. L?wis" <martin@v.loewis.de> writes:

> David Bolen wrote:
> > On Windows, I miss the ability to mix in serial port I/O the most,
> > since many of my servers are translators between the network and
> > serial devices.
> 
> Do you know whether IOCP could be used with the serial ports?

I normally use overlapped I/O with my own WaitForMultipleObjects loop,
but IOCP should work just as well, since access to a serial port is
just another system handle (the port is opened with CreateFile just
like normal files) under Windows and you just use Read/WriteFile to
process the data stream.

IMHO, Windows is arguably more consistent across multiple types of
objects than Unix's "everything is a file" approach.  With something
like Windows WaitForMultipleObjects you can include handles to system
objects for files, semaphores, events, processes and threads, timers,
etc... many of which are concepts that don't fall neatly into a
select() call under Unix.

The rub of course is that sockets are the odd man out, and really feel
bolted on afterwards in Windows (you can read/write to them as file
handles, but not select them directly with the Wait* operations).  If
you're just staying with Windows, you can map sockets to a compatible
object to use with other system objects, but going in through select()
locks you in to sockets only.

For my part, in terms of integrating non-socket sources, I'd probably
be just as happy if Python permitted it's own Queue.Queue to be
selectable, since then I could just have a separate thread with a
native Win32 wait object loop, pumping information into the queue.
But the basic nature of the fundamantal threading constructs Queue is
built on would probably make that difficult, if possible at all.

-- David

From exarkun at divmod.com  Thu Nov 11 03:17:40 2004
From: exarkun at divmod.com (Jp Calderone)
Date: Thu Nov 11 03:17:59 2004
Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the
	standard	library
In-Reply-To: <uvfcdnpgx.fsf@fitlinxx.com>
Message-ID: <20041111021740.20649.160888204.divmod.quotient.2232@ohm>

On 10 Nov 2004 20:28:46 -0500, David Bolen <db3l@fitlinxx.com> wrote:
>"Martin v. L�wis" <martin@v.loewis.de> writes:
> 
> [snip]
> 
> For my part, in terms of integrating non-socket sources, I'd probably
> be just as happy if Python permitted it's own Queue.Queue to be
> selectable, since then I could just have a separate thread with a
> native Win32 wait object loop, pumping information into the queue.
> But the basic nature of the fundamantal threading constructs Queue is
> built on would probably make that difficult, if possible at all.

  This isn't very difficult.  See _UnixWaker/_Win32Waker at in http://cvs.twistedmatrix.com./cvs/trunk/twisted/internet/default.py?view=markup&rev=12283&root=Twisted

  Building a "selectable Queue" on top of the "wakeUp" primitive is a handful of lines at most.  Of course, with the deferToThread() function already built on top of it, the selectable Queue concept doesn't seem quite as attractive anymore.

  Jp
From kbk at shore.net  Thu Nov 11 05:01:21 2004
From: kbk at shore.net (Kurt B. Kaiser)
Date: Thu Nov 11 05:01:27 2004
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200411110401.iAB41LE3007062@h006008a7bda6.ne.client2.attbi.com>

Patch / Bug Summary
___________________

Patches :  244 open ( +3) /  2698 closed (+15) /  2942 total (+18)
Bugs    :  771 open (-13) /  4595 closed (+28) /  5366 total (+15)
RFE     :  157 open ( -1) /   134 closed ( +3) /   291 total ( +2)

New / Reopened Patches
______________________

add missing __all__ to pickletools.py  (2004-11-07)
CLOSED http://python.org/sf/1061679  opened by  George Yoshida

fix for bug 1055168 (pdb not entering the right frame)  (2004-11-06)
CLOSED http://python.org/sf/1061767  opened by  Ilya Sandler

use a new -m option in pdb and profile invocations  (2004-11-06)
CLOSED http://python.org/sf/1061780  opened by  Ilya Sandler

Source code encoding in IDLE console  (2004-11-07)
       http://python.org/sf/1061803  opened by  Hye-Shik Chang

[ 1061857 ] Error and omission in logging docs  (2004-11-07)
CLOSED http://python.org/sf/1061882  opened by  Jeroen Vloothuis

fix for bug 878275 class MyTest(unittest.TestSuite)  (2004-11-07)
CLOSED http://python.org/sf/1061904  opened by  Irmen de Jong

Relative to bug #1061857  -- logging docs  (2004-11-07)
CLOSED http://python.org/sf/1061924  opened by  Rodrigo Dias Arruda Senra

[ 971872 ] An apparent bug in help  (2004-11-07)
CLOSED http://python.org/sf/1061931  opened by  Jeroen Vloothuis

fix for 988120 httplib.py not updating length correctly  (2004-11-07)
CLOSED http://python.org/sf/1061941  opened by  Irmen de Jong

fix for 764437 AF_UNIX socket special linux socket names  (2004-11-07)
       http://python.org/sf/1062014  opened by  Irmen de Jong

Fix for bug issue #1038693 -- bad link in Tkinter docs  (2004-11-07)
CLOSED http://python.org/sf/1062018  opened by  Rodrigo Dias Arruda Senra

Half-fix for 1027771 - DOM insertBefore  (2004-11-07)
CLOSED http://python.org/sf/1062033  opened by  Rodrigo Dias Arruda Senra

fix for 1016880 urllib.urlretrieve silently truncates dwnld  (2004-11-07)
       http://python.org/sf/1062060  opened by  Irmen de Jong

deque pickling problems  (2004-11-08)
CLOSED http://python.org/sf/1062279  opened by  Dima Dorfman

set pickling problems  (2004-11-08)
CLOSED http://python.org/sf/1062353  opened by  Dima Dorfman

Pickle breakage with reduction of recursive structures  (2004-11-08)
       http://python.org/sf/1062277  opened by  Dima Dorfman

distutils and python 1.5.2  (2004-11-09)
CLOSED http://python.org/sf/1063059  opened by  Davide Alberani

Tkinter clipboard_get method (new in Tk 8.4)  (2004-11-10)
       http://python.org/sf/1063914  opened by  Martin Franklin

Patches Closed
______________

add missing __all__ to pickletools.py  (2004-11-06)
       http://python.org/sf/1061679  closed by  tim_one

fix for bug 1055168 (pdb not entering the right frame)  (2004-11-07)
       http://python.org/sf/1061767  closed by  jlgijsbers

use a new -m option in pdb and profile invocations  (2004-11-07)
       http://python.org/sf/1061780  closed by  rhettinger

[ 1061857 ] Error and omission in logging docs  (2004-11-07)
       http://python.org/sf/1061882  closed by  jlgijsbers

fix for bug 878275 class MyTest(unittest.TestSuite)  (2004-11-07)
       http://python.org/sf/1061904  closed by  jlgijsbers

Check for NULL returns in compile.c:com_import_stmt  (2004-09-10)
       http://python.org/sf/1025636  closed by  jhylton

Relative to bug #1061857  -- logging docs  (2004-11-07)
       http://python.org/sf/1061924  closed by  jlgijsbers

[ 971872 ] An apparent bug in help  (2004-11-07)
       http://python.org/sf/1061931  closed by  jlgijsbers

fix for 988120 httplib.py not updating length correctly  (2004-11-07)
       http://python.org/sf/1061941  closed by  jhylton

Update HTTPRespnse.length  (2004-07-10)
       http://python.org/sf/988642  closed by  jhylton

Fix for bug issue #1038693 -- bad link in Tkinter docs  (2004-11-07)
       http://python.org/sf/1062018  closed by  jlgijsbers

Half-fix for 1027771 - DOM insertBefore  (2004-11-07)
       http://python.org/sf/1062033  closed by  jlgijsbers

deque pickling problems  (2004-11-08)
       http://python.org/sf/1062279  closed by  rhettinger

set pickling problems  (2004-11-08)
       http://python.org/sf/1062353  closed by  rhettinger

distutils and python 1.5.2  (2004-11-09)
       http://python.org/sf/1063059  closed by  loewis

New / Reopened Bugs
___________________

test_socket fails  (2004-11-06)
       http://python.org/sf/1061429  opened by  TPJ

spelling error in win installer  (2004-11-06)
CLOSED http://python.org/sf/1061457  opened by  Simon Dahlbacka

Manual typesets bit-shift operators as guillemet  (2004-11-06)
       http://python.org/sf/1061770  opened by  Brion Vibber

Error and omission in logging docs  (2004-11-07)
CLOSED http://python.org/sf/1061857  opened by  Kent Johnson

"k" specifier in PyArg_ParseTuple incomplete documentated  (2004-11-07)
       http://python.org/sf/1061920  opened by  Gustavo J. A. M. Carneiro

threads: segfault or Py_FatalError at exit  (2004-11-07)
CLOSED http://python.org/sf/1061968  opened by  Armin Rigo

test_subprocess fails on winXP  (2004-11-07)
CLOSED http://python.org/sf/1061991  opened by  Miki Tebeka

locale.getdefaultlocale() returns wrong encoding  (2004-11-08)
CLOSED http://python.org/sf/1062261  opened by  Andreas Jung

pwent objects from the pwd module ar e not pickle-able  (2004-11-08)
       http://python.org/sf/1062708  opened by  Keith Dart

not enough information in SGMLParserError?  (2004-11-09)
       http://python.org/sf/1063229  opened by  Alan Ezust

test_subprocess 2.4b2 fails on tru64  (2004-11-10)
       http://python.org/sf/1063571  opened by  roadkill

Modules/zipimport.c does not compile on solaris  (2004-11-08)
CLOSED http://python.org/sf/1062495  opened by  Niki W. Waibel

optparse docs mildly mangled  (2004-11-10)
       http://python.org/sf/1063757  opened by  Michael Hudson

IDLE aborts on any error when started with "-n"  (2004-11-10)
       http://python.org/sf/1063840  opened by  Miki Tebeka

asyncore should handle ECONNRESET in send  (2004-11-10)
       http://python.org/sf/1063924  opened by  Florent Guillaume

Random core dumps  (2004-11-10)
       http://python.org/sf/1063937  opened by  munder12

Bugs Closed
___________

spelling error in win installer  (2004-11-06)
       http://python.org/sf/1061457  closed by  rhettinger

Why does Python link to Foundation?  (2004-09-24)
       http://python.org/sf/1034277  closed by  bcannon

time module returns wrong timezone under windows  (2004-10-24)
       http://python.org/sf/1053539  closed by  bcannon

incorrect "'yield' outside function" error  (2004-11-04)
       http://python.org/sf/1060135  closed by  rhettinger

compiler.transformer, "from module import *"  (2004-10-31)
       http://python.org/sf/1057835  closed by  mwh

pdb.set_trace should go up one more frame  (2004-10-27)
       http://python.org/sf/1055168  closed by  jlgijsbers

Buffer overflow in socketmodule.c  (2004-11-04)
       http://python.org/sf/1060396  closed by  jhylton

test_unicode_file fails on win2k  (2004-07-12)
       http://python.org/sf/989338  closed by  loewis

2.4b1 Win32 installer eliminates 2.3 assoc  (2004-10-20)
       http://python.org/sf/1050486  closed by  loewis

Error and omission in logging docs  (2004-11-07)
       http://python.org/sf/1061857  closed by  jlgijsbers

Implicit close() should check for errors  (2004-05-24)
       http://python.org/sf/959379  closed by  astrand

subprocess on Windows: extra windows  (2004-10-29)
       http://python.org/sf/1057061  closed by  astrand

An apparent bug in help  (2004-06-12)
       http://python.org/sf/971872  closed by  jlgijsbers

httplib.HTTPResponse.read() not updating the length correctl  (2004-07-09)
       http://python.org/sf/988120  closed by  jhylton

subprocess on Windows 2: unexpected failure  (2004-10-29)
       http://python.org/sf/1057052  closed by  astrand

threads: segfault or Py_FatalError at exit  (2004-11-07)
       http://python.org/sf/1061968  closed by  arigo

subprocess fails for args="...", executable="..."  (2004-10-29)
       http://python.org/sf/1056441  closed by  astrand

test_subprocess fails on winXP  (2004-11-07)
       http://python.org/sf/1061991  closed by  tebeka

bad link in Tkinter docs  (2004-10-01)
       http://python.org/sf/1038693  closed by  jlgijsbers

file.next() info hidden  (2004-09-29)
       http://python.org/sf/1036626  closed by  jlgijsbers

In DOM Node Objects, add more explanations for insertBefore  (2004-09-14)
       http://python.org/sf/1027771  closed by  jlgijsbers

httplib throws a TypeError when the target host disconnects  (2002-08-06)
       http://python.org/sf/591349  closed by  jlgijsbers

locale.getdefaultlocale() returns wrong encoding  (2004-11-08)
       http://python.org/sf/1062261  closed by  jlgijsbers

senddigest error  (2004-09-01)
       http://python.org/sf/1020605  closed by  facundobatista

Memory leakage in SAX with exception  (2002-07-31)
       http://python.org/sf/589149  closed by  facundobatista

Include/pyport.h: Bad LONG_BIT assumption on non-glibc sys  (2004-09-07)
       http://python.org/sf/1023838  closed by  cdgregorr

test_xrange fails on osf1 v5.1b  (2004-09-06)
       http://python.org/sf/1022813  closed by  dharma_roadkill

Modules/zipimport.c does not compile on solaris  (2004-11-08)
       http://python.org/sf/1062495  closed by  rhettinger

New / Reopened RFE
__________________

Remove troublesome assert in trace.py  (2004-11-07)
CLOSED http://python.org/sf/1062190  opened by  Jp Calderone

RFE Closed
__________

lock-free data structures?  (2004-11-03)
       http://python.org/sf/1059545  closed by  tim_one

optparse error method doesn't print option info  (2004-07-29)
       http://python.org/sf/1000439  closed by  nestler

Remove troublesome assert in trace.py  (2004-11-07)
       http://python.org/sf/1062190  closed by  rhettinger

From leogah at spamcop.net  Wed Nov 10 22:37:53 2004
From: leogah at spamcop.net (Richard Brodie)
Date: Thu Nov 11 05:23:53 2004
Subject: [Python-Dev] Re: [development doc updates]
Message-ID: <000301c4c76d$89518400$af0189c3@oemcomputer>

> Fred L. Drake, Jr. wrote

> I'm surprised that the first and last columns aren't centered, and that
the
> heading for the second column is.

It seems to me that:

.realtable th {text-align: inherit} is more specific than
.center {text-align: center} ,

so the second rule needs strengthening.

Richard Brodie.

From fdrake at acm.org  Thu Nov 11 05:42:06 2004
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Thu Nov 11 05:42:30 2004
Subject: [Python-Dev] Re: [development doc updates]
In-Reply-To: <000301c4c76d$89518400$af0189c3@oemcomputer>
References: <000301c4c76d$89518400$af0189c3@oemcomputer>
Message-ID: <200411102342.06528.fdrake@acm.org>

On Wednesday 10 November 2004 04:37 pm, Richard Brodie wrote:
 > It seems to me that:
 >
 > .realtable th {text-align: inherit} is more specific than
 > .center {text-align: center} ,
 >
 > so the second rule needs strengthening.

Bingo!  I need the second rule in its simple form elsewhere, but I can 
actually remove the first completely since I set the alignment-controlling 
class for all cells in a .realtable.

The first of those rules was added while I was trying to make <col> useful for 
column layout, but browsers haven't implemented that sufficiently to do so 
yet.

I've tested the fix with my browsers and committed to CVS; an updated version 
of the documentation should be online later this evening.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>

From whisper at oz.net  Thu Nov 11 21:58:48 2004
From: whisper at oz.net (whisper@oz.net)
Date: Thu Nov 11 21:58:51 2004
Subject: [Python-Dev] @boo @hiss def NewDeclSyntaxSux():
Message-ID: <4193D288.7020405@oz.net>

Hmm... wander off for a bit and come back to find that Python has taken 
a big step towards being an Ugly Language[tm].

I can hardly wait for the advent of $ and # as some sort of syntactial 
marker.

Being in beta, it's obviously too late to do anything about it, but I 
think this is going to turn out to be a REALLY BAD MOVE.

David LeBlanc
Seattle, WA USA
From aahz at pythoncraft.com  Thu Nov 11 22:38:45 2004
From: aahz at pythoncraft.com (Aahz)
Date: Thu Nov 11 22:38:48 2004
Subject: [Python-Dev] @boo @hiss def NewDeclSyntaxSux():
In-Reply-To: <4193D288.7020405@oz.net>
References: <4193D288.7020405@oz.net>
Message-ID: <20041111213845.GA13333@panix.com>

On Thu, Nov 11, 2004, whisper@oz.net wrote:
>
> Hmm... wander off for a bit and come back to find that Python has taken 
> a big step towards being an Ugly Language[tm].
> 
> I can hardly wait for the advent of $ and # as some sort of syntactial 
> marker.
> 
> Being in beta, it's obviously too late to do anything about it, but I 
> think this is going to turn out to be a REALLY BAD MOVE.

While I agree with you, please remember that python-dev is a mailing list
where language should stay at the professional level.  If you want to
emote, please use comp.lang.python instead.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

WiFi is the SCSI of the 21st Century -- there are fundamental technical
reasons for sacrificing a goat.  (with no apologies to John Woods)
From tim.peters at gmail.com  Fri Nov 12 05:29:03 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov 12 05:29:06 2004
Subject: [Python-Dev] Code coverage tool updated
In-Reply-To: <4191FDB2.2000002@livinglogic.de>
References: <41864572.4000806@livinglogic.de>
	<16774.45652.835103.327656@montanaro.dyndns.org>
	<4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de>
	<1f7befae04110210537c35e514@mail.gmail.com>
	<418A269B.6010207@livinglogic.de>
	<1f7befae0411051406711618cf@mail.gmail.com>
	<4191FDB2.2000002@livinglogic.de>
Message-ID: <1f7befae04111120293d9c489b@mail.gmail.com>

[Walter D?rwald]
> Using regrtest.py's tracing option instead of trace.py gives better
> results (see http://styx.livinglogic.de/~walter/brokentrace2.txt):

Cool!

> 656 covered modules (including a strange /tmp/tmp6ccfd4/t5/string.py)

test_pkg.py creates the weird string.py.  Don't know why.

> There seems to be something wrong with the way trace.py starts
> the script.
>
> BTW, I'd like to have an option the specify the coverdir in regrtest.py,
> as now finding the coverage file when you have the name of the Python
> file is more complicated. But I guess this has to wait until 2.4 is
> out the door.

I don't know anything about this, but there's no reason to avoid
changing regrtest.py just because 2.4 is in beta.  The arguments to
regrtest.py aren't documented, and it's an internal module to help us
test Python anyway.  So if you can do better 2.4 testing by changing
it now, change it now.
From tim.peters at gmail.com  Fri Nov 12 05:51:15 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov 12 05:51:17 2004
Subject: [Python-Dev] Deprecate PyRange_New()
In-Reply-To: <000001c4c6e6$cda46fa0$6628a044@oemcomputer>
References: <000001c4c6e6$cda46fa0$6628a044@oemcomputer>
Message-ID: <1f7befae04111120514932c693@mail.gmail.com>

[Raymond Hettinger]
> A while back, Tim added a comment that PyRange_New() should be
> deprecated in-part because it is filled with problems and is not used
> anywhere in the core.

And because it's undocumented.  The only place it's even mentioned is
in the NEWS for 2.3a1:

- PyRange_New() now raises ValueError if the fourth argument is not 1.
  This is part of the removal of deprecated features of the xrange
  object.

> Doesn't anyone mind if I mark it in the C-API docs as deprecated so it
> can be phased out later?

+1

I don't object to *documenting* a PyRange_New() API function, BTW, but
the current implementation is too broken to fix with less than heroic
effort.  If PyRange_New() needs to exist in the C API, then it should
have the same (lo, hi, step) signature as the builtin range(), for
which we have correct but non-obvious error-checking C code.  The
funky (start, len, step, reps) signature of the current PyRange_New()
is much more difficult to check for errors (e.g., detecting C long
multiplication overflow in portable C is a sub-problem), and the
current signature is too surprising regardless (because it's so
different from the builtin range()).
From FBatista at uniFON.com.ar  Fri Nov 12 13:48:23 2004
From: FBatista at uniFON.com.ar (Batista, Facundo)
Date: Fri Nov 12 13:50:19 2004
Subject: [Python-Dev] Floating point -> string conversions
Message-ID: <A128D751272CD411BC9200508BC2194D053C7BC0@escpl.tcp.com.ar>

There's a thread in the standard list (with this very Subject) that talks
about implicit coercing with "%d" in strings replacement.

jfouhy posted this example:

>>> pow(2, 31)
2147483648L
>>> '%d' % 2147483647.0              # python will convert to int
'2147483647'
>>> '%d' % 2147483648.0              # too big for an int, so error.
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: int argument required
>>> '%d' % long(2147483648.0)        # but yet, no trouble accepting a long.
'2147483648'
>>> '%d' % int(2147483648.0)         # and int() converts to long anyway
'2147483648'


I think that here's a bug, but don't know where:

Should %d accept only integer values? So this is a bug:

>>> '%d' % 2147483647.0
'2147483647'

Should %d accept also floats, and make the conversion? So there're two bugs:

- In the doc (it's not explicited that it should accept a other data types)

- In this behaviour difference:

>>> '%d' % 2147483647.0
'2147483647'
>>> '%d' % 2147483648.0
Traceback (most recent call last):
...
TypeError: int argument required


I really don't know about opening a bug, because I don't know which to open,
:p

.	Facundo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20041112/929d3980/attachment.html
From arigo at tunes.org  Fri Nov 12 14:12:36 2004
From: arigo at tunes.org (Armin Rigo)
Date: Fri Nov 12 14:19:37 2004
Subject: [Python-Dev] struct module: add OverflowErrors now?
Message-ID: <20041112131236.GA29540@vicky.ecs.soton.ac.uk>

Hi,

The struct module is too lazy in checking for overflows (it only checks some
cases on some architectures).  See patch #1038854.  The question is: can we
make it strict this late in the 2.4 release process?  This *will* break some
user code; the proof is that it breaks some standard library code, e.g.
gzip.py, which uses a signed typecode when it really means an unsigned one.

Fixing it in 2.4 would be nice, but on the other hand we could postpone it for
the 2.5 trunk; people trying 2.5 early will see the problem quickly.  We could
add warnings for 2.4 but that looks overkill.  Or we could just go ahead and
break existing buggy code with 2.4b3.  Opinions?


Armin
From pinard at iro.umontreal.ca  Fri Nov 12 15:51:51 2004
From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard)
Date: Fri Nov 12 15:51:53 2004
Subject: [Python-Dev] struct module: add OverflowErrors now?
In-Reply-To: <20041112131236.GA29540@vicky.ecs.soton.ac.uk>
References: <20041112131236.GA29540@vicky.ecs.soton.ac.uk>
Message-ID: <20041112145151.GA16397@alcyon.progiciels-bpi.ca>

> The struct module is too lazy in checking for overflows [...].

Agreed.  Python is usually on the side of not letting errors go through
silently, and as we learn to comfortably rely on this behaviour, I found
the weakness of `struct' in this area a bit, huh, surprising. :-)

-- 
Fran?ois Pinard   http://pinard.progiciels-bpi.ca
From aahz at pythoncraft.com  Fri Nov 12 16:41:53 2004
From: aahz at pythoncraft.com (Aahz)
Date: Fri Nov 12 16:42:00 2004
Subject: [Python-Dev] struct module: add OverflowErrors now?
In-Reply-To: <20041112131236.GA29540@vicky.ecs.soton.ac.uk>
References: <20041112131236.GA29540@vicky.ecs.soton.ac.uk>
Message-ID: <20041112154153.GB5470@panix.com>

On Fri, Nov 12, 2004, Armin Rigo wrote:
> 
> The struct module is too lazy in checking for overflows (it only
> checks some cases on some architectures).  See patch #1038854.  The
> question is: can we make it strict this late in the 2.4 release
> process?  This *will* break some user code; the proof is that it
> breaks some standard library code, e.g. gzip.py, which uses a signed
> typecode when it really means an unsigned one.

+1 on fixing it now, assuming you can find an appropriate reviewer.
Bugfixing is allowed during beta, and relying on this behavior is a bug.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

WiFi is the SCSI of the 21st Century -- there are fundamental technical
reasons for sacrificing a goat.  (with no apologies to John Woods)
From Scott.Daniels at Acm.Org  Fri Nov 12 21:33:29 2004
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Fri Nov 12 21:32:12 2004
Subject: [Python-Dev] Re: Code coverage tool updated
In-Reply-To: <4191FDB2.2000002@livinglogic.de>
References: <41864572.4000806@livinglogic.de>		<16774.45652.835103.327656@montanaro.dyndns.org>		<4187765D.3000705@livinglogic.de>	<4187CA1C.9050508@livinglogic.de>		<1f7befae04110210537c35e514@mail.gmail.com>		<418A269B.6010207@livinglogic.de>	<1f7befae0411051406711618cf@mail.gmail.com>
	<4191FDB2.2000002@livinglogic.de>
Message-ID: <cn36k7$jjg$1@sea.gmane.org>

Walter D?rwald wrote:
>... There seems to be something wrong with the way trace.py starts
> the script.

I'm starting to loook at writing a dinky trace.py note/document as a
result of bug day, and I have seen two problems so far.  First, the
assert you removed, and second in class CoverageResults:

class CoverageResults:
     def __init__(self, counts=None, calledfuncs=None, infile=None,
                  callers=None, outfile=None):
         ...
           try:
               counts, calledfuncs, callers = \
                       pickle.load(open(self.infile, 'rb'))
               self.update(self.__class__(counts, calledfuncs, callers))
         ...

I suspect switching infile and callers in the parameter list will
give better results.  I don;t know enough about trace yet to build
a test for this problem.

-- Scott David Daniels
Scott.Daniels@Acm.Org

From bac at OCF.Berkeley.EDU  Fri Nov 12 22:28:13 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Fri Nov 12 22:28:25 2004
Subject: [Python-Dev] Floating point -> string conversions
In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7BC0@escpl.tcp.com.ar>
References: <A128D751272CD411BC9200508BC2194D053C7BC0@escpl.tcp.com.ar>
Message-ID: <41952AED.4080900@ocf.berkeley.edu>

Batista, Facundo wrote:
> There's a thread in the standard list (with this very Subject) that 
> talks about implicit coercing with "%d" in strings replacement.
> 
> jfouhy posted this example:
> 
>  >>> pow(2, 31)
> 2147483648L
>  >>> '%d' % 2147483647.0              # python will convert to int
> '2147483647'
>  >>> '%d' % 2147483648.0              # too big for an int, so error.
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> TypeError: int argument required
>  >>> '%d' % long(2147483648.0)        # but yet, no trouble accepting a 
> long.
> '2147483648'
>  >>> '%d' % int(2147483648.0)         # and int() converts to long anyway
> '2147483648'
> 
> 
> I think that here's a bug, but don't know where:
> 

I say it is in the conversion from float not trying it as a long to make it 
work.  I expect the conversion types in string interpolation to convert the 
argument as needed to meet the conversion type.  Since a float can safely be 
converted to a long that is what should be done since in the eyes of Python int 
== long.

Just think of how printf() handles it; it just interprets the argument as what 
the conversion type says it is.  I think Python should do the same.

-Brett
From python at ce.anoka.k12.mn.us  Fri Nov 12 23:22:02 2004
From: python at ce.anoka.k12.mn.us (Mike Tuller)
Date: Fri Nov 12 23:21:57 2004
Subject: [Python-Dev] parse xml
Message-ID: <46D2E28D-34F9-11D9-93D6-0003938E6950@ce.anoka.k12.mn.us>

I am just starting with python. I have a problem where I want to take a 
document that is in xml and take the items and enter them into a mysql 
database. The database part I know, I just need some hints on where to 
look to be able to parse something like this.


                         <dict>
                                 <key>_name</key>
                                 <string>VLC</string>
                                 <key>info</key>
                                 <string>VLC media player 0.7.2, 
Copyright (c) 1996-2004 VideoLAN.</string>
                                 <key>lastModified</key>
                                 <date>2004-05-21T15:04:12Z</date>
                                 <key>path</key>
                                 <string>/Applications/VLC.app</string>
                                 <key>version</key>
                                 <string>0.7.2</string>
                         </dict>
                         <dict>
                                 <key>_name</key>
                                 <string>WeatherPop</string>
                                 <key>lastModified</key>
                                 <date>2003-06-19T05:35:22Z</date>
                                 <key>path</key>
                                 
<string>/Applications/WeatherPop.app</string>
                                 <key>version</key>
                                 <string>1.8.1</string>
                         </dict>


etc.etc.etc.

I want to be able to get the values for name and version for each 
application. I don't need any other information.

Can someone give me some ideas on where to look for more information on 
doing something like this?



Mike

From just at letterror.com  Fri Nov 12 23:38:35 2004
From: just at letterror.com (Just van Rossum)
Date: Fri Nov 12 23:38:32 2004
Subject: [Python-Dev] parse xml
In-Reply-To: <46D2E28D-34F9-11D9-93D6-0003938E6950@ce.anoka.k12.mn.us>
Message-ID: <r01050400-1026-9819104A34FB11D9848D003065D5E7E4@[10.0.0.23]>

Mike Tuller wrote:

> I am just starting with python.

Oops, wrong list. This list is for development OF Python, not WITH.

That said, your data below looks a lot like plist. If it is, have a look
at:

http://cvs.sourceforge.net/viewcvs.py/*checkout*/python/python/dist/src/
Lib/plat-mac/plistlib.py

(unwrap url). You need Python 2.3 to use this module.

Just
From rwgk at cci.lbl.gov  Sat Nov 13 05:20:03 2004
From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve)
Date: Sat Nov 13 05:20:09 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
Message-ID: <200411130420.iAD4K3K2006945@boa.lbl.gov>

Proposal:

  >>> sys.gettickeraccumulation.__doc__
  'getgettickeraccumulation() -> current count of bytecodes processed by the interpreter.'

Target environments:

  Python-based systems with (many) extension modules.

Motivation:

  A fast, easy, and non-intrusive method for estimating how much time
  is spent in Python code and how much time in extension modules.

  E.g. if numarray is used it is often difficult to estimate how much
  time is spent in the numarray extension and how much in
  application-specific Python code. Is it worth investing time
  reimplementing the application-specific code in C or C++? If most of
  the time is already spent in the numarray extension the answer will
  be no.

Method:

  Determination of "micro-seconds per tick:"

    time.time() / sys.gettickeraccumulation() * 1.e6

  A "tick" is the processing of one "Python virtual instruction."
  See also "setcheckinterval" here:
     http://www.python.org/doc/current/lib/module-sys.html

  A pure Python program will spent the bulk of the time interpreting
  bytecode. This will lead to a small value for "micro-seconds per
  tick." In contrast, if almost all the time is spent in extension
  modules, a calculation may run a long time without increasing
  sys.gettickeraccumulation(). For example:

                     time      ticks  time/tick in micro-seconds
      0.00% Python  0.008         20    405.908
     10.00% Python  0.072     444680      0.163
     20.00% Python  0.138     889340      0.155
     30.00% Python  0.203    1334000      0.152
    ...
    100.00% Python  0.659    4446620      0.148

  With higher resolution:

                     time      ticks  time/tick in micro-seconds
      0.00% Python  6.758         20 337893.748
      0.10% Python  0.728     444680      1.638
      0.20% Python  0.795     889340      0.894
      0.30% Python  0.837    1334000      0.627
      0.40% Python  0.923    1778660      0.519
      0.50% Python  0.978    2223320      0.440
      0.60% Python  1.091    2667980      0.409
      0.70% Python  1.111    3112640      0.357
      0.80% Python  1.191    3557300      0.335
      0.90% Python  1.236    4001960      0.309
      1.00% Python  1.315    4446620      0.296

  In real applications we observed times/tick of around 10 on the same
  platform, indicating that significant performance increases could
  only be achieved through extensive recoding in a compiled language.
  On the other hand if we see values smaller than 1 we know that
  significant performance increases are achievable with a reasonable
  effort.

Implementation:

  A full patch based on Python-2.4b2 is attached. The essence is this
  additional line in Python/ceval.c:

+			_Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker;

  Note that _Py_Ticker and _Py_CheckInterval exist already in the
  Python distribution. The impact of the additional code on the runtime
  performance of a pure Python program is minute. On Xeon/Linux the
  factor is smaller than 1.00005. The factor is even smaller if
  extension modules are used.

  The full patch, the patched files, and a complete Python distribution
  including the patched files can be found here:

    http://cci.lbl.gov/~rwgk/python/Python-2.4b2_ticker_patch
    http://cci.lbl.gov/~rwgk/python/Python-2.4b2_ticker_patched_files.tar.gz
    http://cci.lbl.gov/~rwgk/python/Python-2.4b2_ticker.tar.gz


diff -u -r Python-2.4b2/Include/ceval.h Python-2.4b2_ticker/Include/ceval.h
--- Python-2.4b2/Include/ceval.h	2004-10-10 19:40:35.000000000 -0700
+++ Python-2.4b2_ticker/Include/ceval.h	2004-11-12 18:16:28.000000000 -0800
@@ -68,6 +68,11 @@
 
 /* this used to be handled on a per-thread basis - now just two globals */
 PyAPI_DATA(volatile int) _Py_Ticker;
+#ifndef HAVE_LONG_LONG
+PyAPI_DATA(volatile long) _Py_TickerAccumulation;
+#else
+PyAPI_DATA(volatile PY_LONG_LONG) _Py_TickerAccumulation;
+#endif
 PyAPI_DATA(int) _Py_CheckInterval;
 
 /* Interface for threads.
diff -u -r Python-2.4b2/Objects/longobject.c Python-2.4b2_ticker/Objects/longobject.c
--- Python-2.4b2/Objects/longobject.c	2004-09-19 23:14:54.000000000 -0700
+++ Python-2.4b2_ticker/Objects/longobject.c	2004-11-12 18:20:53.000000000 -0800
@@ -38,6 +38,7 @@
 
 #define SIGCHECK(PyTryBlock) \
 	if (--_Py_Ticker < 0) { \
+		_Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker; \
 		_Py_Ticker = _Py_CheckInterval; \
 		if (PyErr_CheckSignals()) { PyTryBlock; } \
 	}
diff -u -r Python-2.4b2/PC/os2emx/python24.def Python-2.4b2_ticker/PC/os2emx/python24.def
--- Python-2.4b2/PC/os2emx/python24.def	2004-10-10 19:40:50.000000000 -0700
+++ Python-2.4b2_ticker/PC/os2emx/python24.def	2004-11-12 18:16:47.000000000 -0800
@@ -743,6 +743,7 @@
   "_Py_CheckRecursionLimit"
   "_Py_CheckInterval"
   "_Py_Ticker"
+  "_Py_TickerAccumulation"
 
 ; From python24_s.lib(compile)
   "PyCode_New"
diff -u -r Python-2.4b2/Python/ceval.c Python-2.4b2_ticker/Python/ceval.c
--- Python-2.4b2/Python/ceval.c	2004-10-10 19:40:50.000000000 -0700
+++ Python-2.4b2_ticker/Python/ceval.c	2004-11-12 18:22:55.000000000 -0800
@@ -373,6 +373,7 @@
 	pendinglast = j;
 
 	_Py_Ticker = 0;
+	_Py_TickerAccumulation = 0;
 	things_to_do = 1; /* Signal main loop */
 	busy = 0;
 	/* XXX End critical section */
@@ -476,6 +477,11 @@
    per thread, now just a pair o' globals */
 int _Py_CheckInterval = 100;
 volatile int _Py_Ticker = 100;
+#ifndef HAVE_LONG_LONG
+volatile long _Py_TickerAccumulation = 0;
+#else
+volatile PY_LONG_LONG _Py_TickerAccumulation = 0;
+#endif
 
 PyObject *
 PyEval_EvalCode(PyCodeObject *co, PyObject *globals, PyObject *locals)
@@ -776,6 +782,7 @@
                                    a try: finally: block uninterruptable. */
                                 goto fast_next_opcode;
                         }
+			_Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker;
 			_Py_Ticker = _Py_CheckInterval;
 			tstate->tick_counter++;
 #ifdef WITH_TSC
diff -u -r Python-2.4b2/Python/sysmodule.c Python-2.4b2_ticker/Python/sysmodule.c
--- Python-2.4b2/Python/sysmodule.c	2004-08-12 11:19:17.000000000 -0700
+++ Python-2.4b2_ticker/Python/sysmodule.c	2004-11-12 18:51:14.000000000 -0800
@@ -442,6 +442,20 @@
 "getcheckinterval() -> current check interval; see setcheckinterval()."
 );
 
+static PyObject *
+sys_gettickeraccumulation(PyObject *self, PyObject *args)
+{
+#ifndef HAVE_LONG_LONG
+	return PyInt_FromLong(_Py_TickerAccumulation + _Py_CheckInterval - _Py_Ticker);
+#else
+	return PyLong_FromLongLong(_Py_TickerAccumulation + _Py_CheckInterval - _Py_Ticker);
+#endif
+}
+
+PyDoc_STRVAR(gettickeraccumulation_doc,
+"gettickeraccumulation() -> current count of bytecodes processed by the interpreter."
+);
+
 #ifdef WITH_TSC
 static PyObject *
 sys_settscdump(PyObject *self, PyObject *args)
@@ -763,6 +777,8 @@
 	 setcheckinterval_doc}, 
 	{"getcheckinterval",	sys_getcheckinterval, METH_NOARGS,
 	 getcheckinterval_doc}, 
+	{"gettickeraccumulation", sys_gettickeraccumulation, METH_NOARGS,
+	 gettickeraccumulation_doc}, 
 #ifdef HAVE_DLOPEN
 	{"setdlopenflags", sys_setdlopenflags, METH_VARARGS, 
 	 setdlopenflags_doc},
From anthony at interlink.com.au  Sat Nov 13 05:34:20 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sat Nov 13 05:34:44 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <200411130420.iAD4K3K2006945@boa.lbl.gov>
References: <200411130420.iAD4K3K2006945@boa.lbl.gov>
Message-ID: <41958ECC.7080100@interlink.com.au>

Ralf W. Grosse-Kunstleve wrote:
> Proposal:
> 
>   >>> sys.gettickeraccumulation.__doc__
>   'getgettickeraccumulation() -> current count of bytecodes processed by the interpreter.'

This doesn't offer enough to be worth rolling into the 2.4 code.

Anthony


-- 
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.
From johahn at home.se  Thu Nov 11 22:50:21 2004
From: johahn at home.se (Johan Hahn)
Date: Sat Nov 13 05:35:40 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
Message-ID: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>

Hi

As far as I can tell from the archive, this has not been discussed before.
This is the second time in less than a week that I have stumbled over the rather 
clumsy syntax of extracting some elements of a sequence and at the same time 
remove those from the sequence:
>>> L = 'a b 1 2 3'.split(' ')
>>> a,b,L = L[0], L[1], L[2:]

I think it would be nice if the following was legal:
>>> a,b,*L = 'a b 1 2 3'.split(' ')
>>> a, b, L
('a', 'b', ['1', '2', '3'])

Today, if the number of variables on both sides of the equal sign doesn't match, 
an exception is raised (for google reference):
ValueError: unpack list of wrong size 

This new syntax is very similar to the special parameter in function definitions that 
catches all excess arguments, as in def func(p, --> *args <--, **kw):, and so the 
semantics should be what everyone expects. 
Then, if we leave this limiting analogy with the *args parameter and allow the 
catch-the-rest variable to be anywhere in the left-hand side, we arrive at a splice 
syntax that reminds me a little of how prolog deals with lists:
>>> a,*b,c = 'a b 1 2 3'.split(' ')
>>> *a,b,c = b.split(' ')
>>> a,b,c
(['b'], '1', '2')

I believe this would make a nice addition to the language.
(Please cc me in response as I'm not subscribed to py-dev.)

...johahn

From anthony at interlink.com.au  Sat Nov 13 05:35:30 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sat Nov 13 05:35:42 2004
Subject: [Python-Dev] struct module: add OverflowErrors now?
In-Reply-To: <20041112131236.GA29540@vicky.ecs.soton.ac.uk>
References: <20041112131236.GA29540@vicky.ecs.soton.ac.uk>
Message-ID: <41958F12.7040406@interlink.com.au>

Armin Rigo wrote:
> Hi,
> 
> The struct module is too lazy in checking for overflows (it only checks some
> cases on some architectures).  See patch #1038854.  The question is: can we
> make it strict this late in the 2.4 release process?  This *will* break some
> user code; the proof is that it breaks some standard library code, e.g.
> gzip.py, which uses a signed typecode when it really means an unsigned one.
> 
> Fixing it in 2.4 would be nice, but on the other hand we could postpone it for
> the 2.5 trunk; people trying 2.5 early will see the problem quickly.  We could
> add warnings for 2.4 but that looks overkill.  Or we could just go ahead and
> break existing buggy code with 2.4b3.  Opinions?

I think I'd prefer to delay it until 2.5. This is going to break a _lot_
of code.

Adding DeprecationWarnings for 2.4 would be more appropriate, imho.




-- 
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.
From foom at fuhm.net  Sat Nov 13 08:02:23 2004
From: foom at fuhm.net (James Y Knight)
Date: Sat Nov 13 08:02:37 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
Message-ID: <F80C7DCC-3541-11D9-AA57-000A95A50FB2@fuhm.net>


On Nov 11, 2004, at 4:50 PM, Johan Hahn wrote:
> As far as I can tell from the archive, this has not been discussed 
> before.
> [...]
> I think it would be nice if the following was legal:
> >>> a,b,*L = 'a b 1 2 3'.split(' ')
> >>> a, b, L
> ('a', 'b', ['1', '2', '3'])

Oh, this has been discussed before. It's come up quite a number of 
times, in fact. But, it's been rejected every time. FWIW [which is very 
little ;)], I agree: this would be nice. I'm strongly for consistency 
and orthogonality: there should be few distinct concepts in the 
language each of which works in many places. Allowing * only in an 
function call but not in other tuple-constructing situations violates 
this concept, IMO. It would have been great if this had been added at 
the same time as * in function calls. Adding it now may be more trouble 
than it's worth, though.

Two threads on the topic, with rejections by Guido:
http://mail.python.org/pipermail/python-dev/2002-November/030349.html
http://mail.python.org/pipermail/python-dev/2004-August/046684.html

Guido's rejection from 
http://mail.python.org/pipermail/python-dev/2004-August/046794.html:
> For the record, let me just say that I'm -1 on adding this feature
> now.  It's only a minor convenience while it's a fairly big change to
> the parser and code generator, but most of all, Python's growth needs
> to be reduced.  If we were to add every "nice" feature that was
> invented (or present in other languages), Python would lose one of its
> main charms, which is that it is really a rather simple language that
> can be learned and put to production quickly.
>
> So while I appreciate the idea (which BTW has been proposed before) I
> think it's not worth adding now, and if you don't hear from me again
> on this topic it's because I haven't changed my mind...

James

From rwgk at cci.lbl.gov  Sat Nov 13 10:19:19 2004
From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve)
Date: Sat Nov 13 10:19:28 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
Message-ID: <200411130919.iAD9JJxP009427@boa.lbl.gov>

> > Proposal:
> > 
> >   >>> sys.gettickeraccumulation.__doc__
> >   'getgettickeraccumulation() -> current count of bytecodes processed by the interpreter.'
> 
> This doesn't offer enough to be worth rolling into the 2.4 code.

I could easily provide a similar patch based on the mainline CVS.
Ralf
From anthony at interlink.com.au  Sat Nov 13 11:52:54 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sat Nov 13 11:53:08 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <200411130919.iAD9JJxP009427@boa.lbl.gov>
References: <200411130919.iAD9JJxP009427@boa.lbl.gov>
Message-ID: <4195E786.1090207@interlink.com.au>

Ralf W. Grosse-Kunstleve wrote:
>>This doesn't offer enough to be worth rolling into the 2.4 code.
> 
> 
> I could easily provide a similar patch based on the mainline CVS.

I'm not sure we're understanding each other here. What I was saying
is that if this were to go into Python, it should go in after 2.4
final is out, and the release24-maint branch has been cut.

Anthony

-- 
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.
From martin at v.loewis.de  Sat Nov 13 12:41:12 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov 13 12:41:09 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
Message-ID: <4195F2D8.6010208@v.loewis.de>

Johan Hahn wrote:
> Hi
> 
> As far as I can tell from the archive, this has not been discussed before.
> This is the second time in less than a week that I have stumbled over the rather 
> clumsy syntax of extracting some elements of a sequence and at the same time 
> remove those from the sequence:
> 
>>>>L = 'a b 1 2 3'.split(' ')
>>>>a,b,L = L[0], L[1], L[2:]

As James says, this has been discussed before. Typically, you don't need
the rest list, so you write

a,b = 'a b 1 2 3'.split(' ')[:2]

If you do need the rest list, it might be best to write

L = 'a b 1 2 3'.split(' ')
a = L.pop(0)
b = L.pop(0)

Whether this is more efficient than creating a new list depends on the
size of the list; so you might want to write

L = 'a b 1 2 3'.split(' ')
a = L[0]
b = L[1]
del L[:2]

This is entirely different from functions calls, which you simply cannot
spread over several statements.

Regards,
Martin
From martin at v.loewis.de  Sat Nov 13 12:48:15 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov 13 12:48:16 2004
Subject: [Python-Dev] [development doc updates]
In-Reply-To: <20041113064452.B36C518E89A@grendel.fdrake.net>
References: <20041113064452.B36C518E89A@grendel.fdrake.net>
Message-ID: <4195F47F.5050504@v.loewis.de>

Fred L. Drake wrote:
> These changes are experimental at this point, and may be changed
> further or abandoned.  I'm very interested in getting feedback on
> these changes; if people like them, they can be put into place for
> Python 2.4.

I like the changes.

> Please direct feedback to the Python Documentation SIG mailing list:

I would have, but my mail to this list was rejected.

Regards,
Martin
From ncoghlan at iinet.net.au  Sat Nov 13 13:05:23 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 13 13:05:30 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
Message-ID: <4195F883.8030309@iinet.net.au>

Johan Hahn wrote:
> Hi
> 
> As far as I can tell from the archive, this has not been discussed before.
> This is the second time in less than a week that I have stumbled over the rather 
> clumsy syntax of extracting some elements of a sequence and at the same time 
> remove those from the sequence:
> 
>>>>L = 'a b 1 2 3'.split(' ')
>>>>a,b,L = L[0], L[1], L[2:]
> 
> 
> I think it would be nice if the following was legal:
> 
>>>>a,b,*L = 'a b 1 2 3'.split(' ')
>>>>a, b, L
> 
> ('a', 'b', ['1', '2', '3'])

Hmm - I just had a thought about this. Is it worth adding a "len" 
argument to list.pop? (The idea was inspired by Martin's use of list.pop 
to handle the above case).

With that change, the above example would become:

a, b = L.pop(0, 2)

At the moment, list.pop is described as equivalent to:

x = L[i]; del L[i]; return x

with this change, it would be:

x = L[i:i+n]; del L[i:i+n]; return x

By default, n = 1, so the standard behaviour of list.pop is preserved.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From martin at v.loewis.de  Sat Nov 13 14:03:15 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov 13 14:03:11 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <4195F883.8030309@iinet.net.au>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
	<4195F883.8030309@iinet.net.au>
Message-ID: <41960613.1000307@v.loewis.de>

Nick Coghlan wrote:
> x = L[i:i+n]; del L[i:i+n]; return x
> 
> By default, n = 1, so the standard behaviour of list.pop is preserved.

This default would actually change the standard behaviour: whereas it
now returns a single element, it would then return a list containing
the single element.

Regards,
Martin
From johahn2003 at home.se  Sat Nov 13 14:16:41 2004
From: johahn2003 at home.se (Johan Hahn)
Date: Sat Nov 13 14:20:07 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME><4195F883.8030309@iinet.net.au>
	<41960613.1000307@v.loewis.de>
Message-ID: <033201c4c983$045a1dd0$0d00a8c0@MAINFRAME>

Martin v. L?wis wrote:
> Nick Coghlan wrote:
> > x = L[i:i+n]; del L[i:i+n]; return x
> > 
> > By default, n = 1, so the standard behaviour of list.pop is preserved.
> 
> This default would actually change the standard behaviour: whereas it
> now returns a single element, it would then return a list containing
> the single element.

Too bad.
I would have liked list.pop(i, n=1, step=1) to be the same as 

x = L[i:i+n:step]; del L[i:i+n:step]; return x

Having pop()/pop(i) return an element and pop(i, n)/pop(i, n, step) return a 
list is a no-no, or is it?

...johahn
From ncoghlan at iinet.net.au  Sat Nov 13 14:41:25 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 13 14:41:33 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <41960613.1000307@v.loewis.de>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
	<4195F883.8030309@iinet.net.au> <41960613.1000307@v.loewis.de>
Message-ID: <41960F05.60006@iinet.net.au>

Martin v. L?wis wrote:
> Nick Coghlan wrote:
> 
>> x = L[i:i+n]; del L[i:i+n]; return x
>>
>> By default, n = 1, so the standard behaviour of list.pop is preserved.
> 
> 
> This default would actually change the standard behaviour: whereas it
> now returns a single element, it would then return a list containing
> the single element.

Ah, good point.

I'd see two possible fixes to that:

a) have n=0 be the default, and mean 'give me the element, not a list 
with 1 element" (L.pop(0,1) would then mean "give me a list containing 
only the first element"). That's a little magical for my taste, though.

b) have a new method called 'extract' or 'poplist' or 'popslice' or 
similar (with the behaviour given above)

Actually, if we went with the b) option, and the name 'popslice', I 
would suggest the following signature:

list.popslice(start=0, end, step=1)

i.e. L.popslice(a, b, c) is to L[a:b:c] as L.pop(a) is to L[a]

And, returning once again to the OP's example, we would have:

a, b = list.popslice(2)

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Sat Nov 13 14:50:30 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 13 14:50:36 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <41960F05.60006@iinet.net.au>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>	<4195F883.8030309@iinet.net.au>
	<41960613.1000307@v.loewis.de> <41960F05.60006@iinet.net.au>
Message-ID: <41961126.7060101@iinet.net.au>

Nick Coghlan wrote:
> Actually, if we went with the b) option, and the name 'popslice', I 
> would suggest the following signature:
> 
> list.popslice(start=0, end, step=1)

"Single argument is end value" was modelled on range(), FWIW.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Sat Nov 13 15:04:44 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 13 15:04:51 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <41961126.7060101@iinet.net.au>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>	<4195F883.8030309@iinet.net.au>	<41960613.1000307@v.loewis.de>
	<41960F05.60006@iinet.net.au> <41961126.7060101@iinet.net.au>
Message-ID: <4196147C.8050303@iinet.net.au>

Nick Coghlan wrote:
> Nick Coghlan wrote:
> 
>> Actually, if we went with the b) option, and the name 'popslice', I 
>> would suggest the following signature:
>>
>> list.popslice(start=0, end, step=1)
> 
> 
> "Single argument is end value" was modelled on range(), FWIW.

Heh - turns out there is an actual slice() constructor with that 
start/end/step signature (I've never used it, so I didn't even know it 
was there).

Anyway, *if* we do anything about this (cleaner unpacking & removal of 
part of a list), I think 'popslice' or something similar is the right 
way to address it (rather than monkeying with syntax, which would only 
address slices at the start of the list anyway).

Any of the "a, b, *c = L" proponents want to come up with a candidate 
'list.popslice' patch for 2.5?

The implementation shouldn't be too hard - the "x = L[a:b:c]; del 
L[a:b:c]; return x" description should suffice for an implementation, 
too (using the C API, rather than actual Python, though).

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From jjl at pobox.com  Sat Nov 13 15:37:53 2004
From: jjl at pobox.com (John J Lee)
Date: Sat Nov 13 15:38:36 2004
Subject: [Python-Dev] Minor cookielib changes for 2.4b1
In-Reply-To: <002601c4b2de$98ed7840$e841fea9@oemcomputer>
References: <002601c4b2de$98ed7840$e841fea9@oemcomputer>
Message-ID: <Pine.LNX.4.58.0411131431010.6308@alice>

Sorry for not replying earlier: I've not been well.

On Fri, 15 Oct 2004, Raymond Hettinger wrote:

> > John J Lee wrote:
> > > The patch contains uncontroversial changes to cookielib and
> associated
> > > modules, documentation and tests. Would be unfortunate not to have
> these
> > > in 2.4.0.
> 
> [Martin]
> > What precisely makes these changes uncontroversial? The fact that no
> > controversion has been developed?

My attempt at time-travel was unsuccessful, sorry.  :-)


> I've looked over the patch and think most of it should wait until Py2.5.
> Refactorings of this magnitude should be done early in the development
> cycle to allow time to discover and fix issues if something gets mucked
> up.  

Hadn't realised how late it was.


> A couple of the doc changes should probably go in for Py2.4b2 (a note on
> thread safety and an example of how to display debugging information).
> If no one objects, I'll put those in after the beta.  Also there is a
> minor readability fix (using escape sequences) for the test suite that
> is reasonable for Py2.4.

Thanks, Raymond.


John
From rwgk at cci.lbl.gov  Sat Nov 13 18:16:05 2004
From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve)
Date: Sat Nov 13 18:16:15 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
Message-ID: <200411131716.iADHG59n012286@boa.lbl.gov>

> it should go in after 2.4
> final is out, and the release24-maint branch has been cut.

OK, I'll wait until 2.4 is out and submit another patch based on the
mainline CVS through the SourceForge system.

> if this were to go into Python

- What speaks against making the ticker accumulation available
  at the script level?

- Is there an alternative way of achieving the goal of my proposal?
  It was:
  > A fast, easy, and non-intrusive method for estimating how much time
  > is spent in Python code and how much time in extension modules.

Thanks!
        Ralf
From ncoghlan at iinet.net.au  Sun Nov 14 10:45:42 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sun Nov 14 10:45:50 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <41964CF2.2090209@botanicus.net>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>	<4195F883.8030309@iinet.net.au>
	<41960613.1000307@v.loewis.de> <41960F05.60006@iinet.net.au>
	<41964CF2.2090209@botanicus.net>
Message-ID: <41972946.8070207@iinet.net.au>

David Wilson wrote:
> I dislike the use of 'popslice' as it implies somewhere along the lines 
> the use of the slice type, which isn't happening.

If I was writing it, using a slice object as an argument to 
PySequence_GetItem is *exactly* the method I would be using to implement it:

   indices = PySlice_New(start, end, step);
   x = PySequence_GetItem(L, indices);
   PySequence_DelItem(L, indices);
   return x;

Add reference counting, error checking and actually hook this up to the 
list type in the interpreter, and you're basically done.

Heck, it could be argued that list.pop should just accept slices for the 
argument:

   a, b = L.pop(slice(2)) # Pop the first two items
   x, y, z = L.pop(slice(1:6:2)) # Pop items 1, 3 & 5

No new method required in that case - if you provide an index to 
list.pop, it will return a single item, if you provide a slice object, 
it will return a list. (Similar to the way list.__getitem__ currently works)

 > Also, what if at some
 > future date it is decided that all magical syntax should be exposed
 > using friendly method names too? Then you might have a list.slice which
 > work in a much different way.[1]

List slicing is already exposed in Python in a few ways:

   L[:2]
   L.__getitem__(slice(2))
   operator.itemgetter(slice(2))(L)

Adding yet another way to spell it doesn't seem too likely. Given that 
lists *had* a __getslice__ method, and it has since been deprecated in 
favour of slice() objects, I'd be especially surprised to see list.slice 
(or equivalent) making a comeback.

I will note that I only looked into the history of slicing this evening 
(when checking the C API for the sample implementation above). Having 
done so, the trend seems to have been away from having special methods 
for handling slices (__getslice__, __setslice__, __delslice__), and 
towards methods which can handle either a numeric index or a slice 
object (__getitem__, __setitem__, __delitem__).

So maybe "list.pop(slice(2))" is the way to go (a non-optimised version 
could be written fairly easily using the existing internal list 
functions to retrieve and then delete the data. The main trick would be 
to leave the standard path through listpop alone to avoid any 
performance hit to the current users of list.pop() and list.pop(x)).

> Second of all, you cannot have a non-default argument follow a default 
> argument, so your proposed syntax would not work without breaking 
> standard semantics and writing special code to act differently depending 
> on the argument list length, and also take into account if keyword args 
> are used, etc, etc.

The signature I proposed is identical to the existing signatures for the 
builtins range(), xrange() and slice().

It's true that normal Python functions and methods can't have a 
signature like that (it's a syntax error). C extensions (including the 
CPython interpreter) can do whatever they want, though. In this case, I 
think being consistent with slice() is more important than being 
consistent with normal function signatures.

However, the suggestion above (allowing slice objects as arguments to 
list.pop) takes advantage of the existing slice() builtin, so none of 
that non-standard argument handling would need to be duplicated.

> For an extra method, +1, but with a different name (I was thinking 
> 'popitems', but dict.popitem returns a (k, v)).

What do you think of allowing slice objects to be used with list.pop()?

> Oh one last thought. If "*<identifier>" were to be added, then what of 
> "**<identifier>"? I can't think of any use for it, and so the whole 
> scheme might appear confusing to beginners who find they can use ** in 
> function declarations, but not in expressions.

Aye, I don't think it's an accident that "a, b *c = L" has been knocked 
on the head 3 times now. My main gripe with it is that it requires new 
syntax, yet still only supports a small subset of slicing. My idea about 
enhancing list.pop or adding a new method to list to provide the desired 
functionality is aimed at avoiding having the discussion *again* in the 
future (since clearly this is functionality some portion of the user 
community would make use of).

I don't actually need the functionality myself, so I doubt I'll get 
around to providing a sample implementation. But the suggestion is there 
if any of the people that *do* want the feature would like to implement it.

> [1] Exposing magical syntax (eg. indexers) as real methods is the 
> Microsoft way for .NET (or at least, according to a C# book I read). It 
> proves useful in occasions where interoperability with lesser systems is 
> required, for example, a language that does not support custom indexers, 
> or for enabling clients of an object bridge [that does not support 
> cust..] to make full use of your classes.

Python already exposes most things as methods - I found browsing through 
the Language Reference a few months back extremely enlightening in that 
regard (specifically Section 3.3 Special Method Names).

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From mwh at python.net  Sun Nov 14 12:08:37 2004
From: mwh at python.net (Michael Hudson)
Date: Sun Nov 14 12:08:39 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <200411130420.iAD4K3K2006945@boa.lbl.gov> (Ralf W.
	Grosse-Kunstleve's message of "Fri, 12 Nov 2004 20:20:03 -0800 (PST)")
References: <200411130420.iAD4K3K2006945@boa.lbl.gov>
Message-ID: <2m3bzc1ydm.fsf@starship.python.net>

"Ralf W. Grosse-Kunstleve" <rwgk@cci.lbl.gov> writes:

> Proposal:
>
>   >>> sys.gettickeraccumulation.__doc__
>   'getgettickeraccumulation() -> current count of bytecodes processed by the interpreter.'
>

You know about the 'tick_counter' field of PyThreadState?  I don't
think it's accessible to Python currently, but it seems to be intended
for similar purposes...

Cheers,
mwh

-- 
  Hiro dicks about with his computer, naturally.  Being stranded 
  on a life raft in the Pacific is a perfect venue for a hacker.
                                        -- Snow Crash, Neal Stephenson
From ncoghlan at gmail.com  Sun Nov 14 13:30:03 2004
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun Nov 14 13:30:15 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <200411130420.iAD4K3K2006945@boa.lbl.gov>
References: <200411130420.iAD4K3K2006945@boa.lbl.gov>
Message-ID: <41974FCB.4090002@iinet.net.au>

Ralf W. Grosse-Kunstleve wrote:
>   A pure Python program will spent the bulk of the time interpreting
>   bytecode.

Perhaps, perhaps not. A heck of a lot of the Python core isn't written 
in Python - time spent executing builtins, or running methods of builtin 
objects usually doesn't involve the main interpreter loop (we're 
generally in pure C-code at that point).

I'm curious how the suggested feature can provide any information that 
is actually useful for optimisation purposes. Just because a low 
proportion of time is spent in Python code, doesn't mean the Python code 
isn't at fault for poor performance.

As an example, in CPython 2.3 and earlier, this:

   result = ""
   for x in strings:
     result += x

is a lot worse performance-wise than:

   result = "".join(strings)

The first version does spend more time in Python code, but the 
performance killer is actually in the string concatenation C code. So 
the time is spent in the C code, but the fault lies in the Python code 
(In Python 2.4, the latter version is still faster, but the difference 
isn't as dramatic as it used to be).

Knowing "I'm spending x% of the time executing Python code" just isn't 
really all that interesting, unless you know *what* Python code is 
taking all the time. (e.g., if decimal.py spends all its time creating 
new instances of Decimal when running a benchmark, then you know to 
either speed up Decimal construction, or to look for temporary objects 
that can be eliminated, or both. Then you run the benchmark again to 
determine if the changes actually improved the situation)

I'd rather encourage people to write appropriate benchmark scripts and 
execute them using "python -m profile <benchmark>", rather than lead 
them up the garden path with a global "Python/non-Python" percentage 
estimation utility.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From arigo at tunes.org  Sun Nov 14 14:17:08 2004
From: arigo at tunes.org (Armin Rigo)
Date: Sun Nov 14 14:29:37 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <4195F883.8030309@iinet.net.au>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
	<4195F883.8030309@iinet.net.au>
Message-ID: <20041114131708.GA9442@vicky.ecs.soton.ac.uk>

Hello Nick,

On Sat, Nov 13, 2004 at 10:05:23PM +1000, Nick Coghlan wrote:
> At the moment, list.pop is described as equivalent to:
> 
> x = L[i]; del L[i]; return x

Then the documentation is not exact: if it were really equivalent to the above
definition, then we could already do today:

  a, b = L.pop(slice(0,2))

because starting from Python 2.3,

  L[slice(0,2)]

is equivalent to

  L[0:2]

So who's wrong: the documentation of list.pop() or its implementation?  It
would be nicely regular if we could pop slices.


Armin
From rwgk at cci.lbl.gov  Sun Nov 14 20:18:14 2004
From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve)
Date: Sun Nov 14 20:18:19 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
Message-ID: <200411141918.iAEJIEvw023160@boa.lbl.gov>

Nick Coghlan wrote:

> Ralf W. Grosse-Kunstleve wrote:
> >    A pure Python program will spent the bulk of the time interpreting
> >    bytecode.
> 
> Perhaps, perhaps not.

Right. But remember what the actual goal is: we want to answer the
question "Is it worth reimplementing a piece currently written
in Python in C/C++?"

> A heck of a lot of the Python core isn't written 
> in Python - time spent executing builtins, or running methods of builtin 
> objects usually doesn't involve the main interpreter loop (we're 
> generally in pure C-code at that point).

If a piece of Python code leads to heavy use of complex, time-consuming
builtin operations it will be of less benefit to reimplement that code
in C/C++. This is exactly what we want to learn.

> I'm curious how the suggested feature can provide any information that 
> is actually useful for optimisation purposes. Just because a low 
> proportion of time is spent in Python code, doesn't mean the Python code 
> isn't at fault for poor performance.
> 
> As an example, in CPython 2.3 and earlier, this:
> 
>    result = ""
>    for x in strings:
>      result += x
> 
> is a lot worse performance-wise than:
> 
>    result = "".join(strings)
> 
> The first version does spend more time in Python code, but the 
> performance killer is actually in the string concatenation C code. So 
> the time is spent in the C code, but the fault lies in the Python code 
> (In Python 2.4, the latter version is still faster, but the difference 
> isn't as dramatic as it used to be).

Exactly. If you try out my patch and look at time/ticks you will see
immediately that there is no point in reimplementing "".join(strings)
in C/C++. Importantly, you don't have to look at the code to arrive at
this conclusion. The time/tick alone will tell you. This is very
helpful if you are working with third-party code.

> Knowing "I'm spending x% of the time executing Python code" just isn't 
> really all that interesting,

Right. Sorry if I gave the wrong impression that this could be
interesting. It is indeed not. What is interesting is the estimated
benefit of reimplementing a piece of Python in C/C++. This is in
fact highly correlated with the time/tick.

> I'd rather encourage people to write appropriate benchmark scripts and 
> execute them using "python -m profile <benchmark> ",

This approach is impractical/impossible in the real world.
For example, this is the problem prompting me to implement
sys.gettickeraccumulation():

  http://pyquante.sourceforge.net/

- It is not our code, i.e. it is difficult for us to know where
  the time is spent.
- It makes heavy use of Numeric.
- It has a few innermost loops implemented in C.

We are using only some parts of this library.
Question: if we reimplement these parts completely in C++, what speedup
can we expect?
So we run a whole calculation and print the time/tick, which you can do
with less than a one-minute investment:

  print time.time()/sys.gettickeraccumulation()*1.e6

as the last statement of your code. If the printed value is close 0.15
on our reference platform we know that the speedup will be in the
neighborhood of 100. Any value higher than 0.15 indicates that the
expected speedup will less. In our case the value was 0.35, and after
we did the reimplementation in C++ we found a speedup of about 10. We
have other applications with time/tick around 10. Just looking at this
number tells us that there is not much to gain unless we completely
eliminate Python. Bring in the cost for the C++ programmer and the
increased cost of maintaining the C++ code compared to Python, and you
know what we decided (not) to do.

> rather than lead 
> them up the garden path with a global "Python/non-Python" percentage 
> estimation utility.

Please consider that that utility is simply printing
time.time()/sys.gettickeraccumulation(), that my patch is trivial, and
that the runtime penalty is close to non-existing.

Cheers,
        Ralf
From martin at v.loewis.de  Sun Nov 14 20:38:18 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun Nov 14 20:38:20 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <20041114131708.GA9442@vicky.ecs.soton.ac.uk>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>	<4195F883.8030309@iinet.net.au>
	<20041114131708.GA9442@vicky.ecs.soton.ac.uk>
Message-ID: <4197B42A.8020401@v.loewis.de>

Armin Rigo wrote:
> So who's wrong: the documentation of list.pop() or its implementation?  
> It would be nicely regular if we could pop slices.

That's an interesting observation. I was somewhat concerned that this
kind of overloading might be confusion, but given that it is even
already documented (with nobody noticing), I'm now very much in favour
of allowing to pop slices.

Of course, whoever implements the change should make sure that slices
are applicable consistently everwhere the documentation says they should
be.

Regards,
Martin



From kayschluehr at gmx.de  Sun Nov 14 19:27:30 2004
From: kayschluehr at gmx.de (Kay Schluehr)
Date: Sun Nov 14 21:30:24 2004
Subject: [Python-Dev] Int literals and method calls
Message-ID: <4197A392.7080100@gmx.de>

Hi  ,
i wondered why string and bool literals have access methods and one can 
call simply

 >>> "1".__class__
<type 'str'>

 >>> False.__class__
<type 'bool'>

But this won't be true for int literals and float literals except for 
those float literals that are
terminated by a dot:

 >>> 1..__class__
<type 'float'>

The expression below raises an error:

 >>> 1.__class__
Traceback (  File "<interactive input>", line 1
    1.__class__
              ^
SyntaxError: invalid syntax

So it seems to be a parser-problem, related to the ambiguity of the 
terminating dot?
Could this be patched?

Thanks
           Kay



From Scott.Daniels at Acm.Org  Sun Nov 14 21:50:00 2004
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Sun Nov 14 21:48:44 2004
Subject: [Python-Dev] Re: Int literals and method calls
In-Reply-To: <4197A392.7080100@gmx.de>
References: <4197A392.7080100@gmx.de>
Message-ID: <cn8gb5$cm0$1@sea.gmane.org>

Kay Schluehr wrote:
> But this won't be true for int literals and float literals ...
> The expression below raises an error:
> 
>  >>> 1.__class__
> Traceback (  File "<interactive input>", line 1
>    1.__class__
>              ^
> SyntaxError: invalid syntax
But the expressions below both work:
     >>> 2 .__class__
     <type 'int'>
     >>> (3).__class__
     <type 'int'>

> So it seems to be a parser-problem, 
To the extent it _is_ a problem, it is a lexer problem.  However,
"In the face of ambiguity, refuse to guess."

-- Scott David Daniels
Scott.Daniels@Acm.Org

From martin at v.loewis.de  Sun Nov 14 22:30:49 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun Nov 14 22:30:52 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <4197A392.7080100@gmx.de>
References: <4197A392.7080100@gmx.de>
Message-ID: <4197CE89.2060801@v.loewis.de>

Kay Schluehr wrote:

>  >>> 1.__class__
> Traceback (  File "<interactive input>", line 1
>    1.__class__
>              ^
> SyntaxError: invalid syntax
> 
> So it seems to be a parser-problem, related to the ambiguity of the 
> terminating dot?

There is no ambiguity. This expression parses as two tokens,
floatnumber (through the second alternative of pointfloat),
followed by identifier. While this is not explicitly specified
in the language reference, Python follows the common "longest
match" approach to tokenization, i.e. you extend each token
from the beginning of the input as much as you can, and then
proceed to the next token.

> Could this be patched?

Perhaps. But it should not, as the current behaviour is
intentional. If you want to take attributeref of an
integer literal, use one of the following forms:

 >>> 2 .__class__
<type 'int'>
 >>> (3).__class__
<type 'int'>

Regards,
Martin
From mike at skew.org  Sun Nov 14 22:49:29 2004
From: mike at skew.org (Mike Brown)
Date: Sun Nov 14 22:49:32 2004
Subject: [Python-Dev] parse xml
In-Reply-To: <46D2E28D-34F9-11D9-93D6-0003938E6950@ce.anoka.k12.mn.us>
	"from Mike Tuller at Nov 12, 2004 04:22:02 pm"
Message-ID: <200411142149.iAELnTIr006016@chilled.skew.org>

Mike Tuller wrote:
> I am just starting with python. I have a problem where I want to take a 
> document that is in xml [...]

You may get any number of helpful answers (zero is a number too, heh)
if you post this to the Python XML SIG list. There are various ways to
go about it, both with Python's built-in XML parsing capabilities, and
with external packages.

See http://mail.python.org/mailman/listinfo/xml-sig/
From greg.ewing at canterbury.ac.nz  Mon Nov 15 05:12:50 2004
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon Nov 15 05:12:59 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <4197A392.7080100@gmx.de>
References: <4197A392.7080100@gmx.de>
Message-ID: <41982CC2.80006@canterbury.ac.nz>

Kay Schluehr wrote:
     1.__class__
>              ^
> SyntaxError: invalid syntax
> 
> Could this be patched?

Why bother? Surely it's not too much trouble to type

   (1).__class__

on the rare occasions when you want to do something
like this.

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+

From michael.walter at gmail.com  Mon Nov 15 05:44:24 2004
From: michael.walter at gmail.com (Michael Walter)
Date: Mon Nov 15 05:44:27 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <41982CC2.80006@canterbury.ac.nz>
References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz>
Message-ID: <877e9a1704111420441c9e683c@mail.gmail.com>

It's confusing/inconsistent.


On Mon, 15 Nov 2004 17:12:50 +1300, Greg Ewing
<greg.ewing@canterbury.ac.nz> wrote:
> Kay Schluehr wrote:
>      1.__class__
> >              ^
> > SyntaxError: invalid syntax
> > 
> > Could this be patched?
> 
> Why bother? Surely it's not too much trouble to type
> 
>    (1).__class__
> 
> on the rare occasions when you want to do something
> like this.
> 
> --
> Greg Ewing, Computer Science Dept, +--------------------------------------+
> University of Canterbury,          | A citizen of NewZealandCorp, a       |
> Christchurch, New Zealand          | wholly-owned subsidiary of USA Inc.  |
> greg@cosc.canterbury.ac.nz         +--------------------------------------+
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/michael.walter%40gmail.com
>
From martin at v.loewis.de  Mon Nov 15 07:16:19 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon Nov 15 07:16:21 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <877e9a1704111420441c9e683c@mail.gmail.com>
References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz>
	<877e9a1704111420441c9e683c@mail.gmail.com>
Message-ID: <419849B3.9020406@v.loewis.de>

Michael Walter wrote:
> It's confusing/inconsistent.

That is not true. It might be confusing, but it is not inconsistent.

Regards,
Martin
From michael.walter at gmail.com  Mon Nov 15 08:30:15 2004
From: michael.walter at gmail.com (Michael Walter)
Date: Mon Nov 15 08:30:18 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <419849B3.9020406@v.loewis.de>
References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz>
	<877e9a1704111420441c9e683c@mail.gmail.com>
	<419849B3.9020406@v.loewis.de>
Message-ID: <877e9a170411142330661d3a33@mail.gmail.com>

I think it's inconsistent because it works for "list literals" but not
for "integer literals". What do I miss?

Cheers,
Michael


On Mon, 15 Nov 2004 07:16:19 +0100, Martin v. L?wis <martin@v.loewis.de> wrote:
> Michael Walter wrote:
> > It's confusing/inconsistent.
> 
> That is not true. It might be confusing, but it is not inconsistent.
> 
> Regards,
> Martin
>
From gerrit at nl.linux.org  Mon Nov 15 10:54:01 2004
From: gerrit at nl.linux.org (Gerrit)
Date: Mon Nov 15 10:54:28 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <419849B3.9020406@v.loewis.de>
References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz>
	<877e9a1704111420441c9e683c@mail.gmail.com>
	<419849B3.9020406@v.loewis.de>
Message-ID: <20041115095401.GA19806@nl.linux.org>

"Martin v. L?wis" wrote:
> Michael Walter wrote:
> >It's confusing/inconsistent.
> 
> That is not true. It might be confusing, but it is not inconsistent.

It works for complex numbers:

>>> 1j.imag
1.0

...in a rather surprising way:

>>> 3+4j.real
3.0
>>> f() + 5j.real
42.0

I would have expected it to be 0.0, because I would have expected the
'.' to have precedence over the '+', but it hasn't. apparantly.

Or has it?
>>> 3+4j.__class__
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: unsupported operand type(s) for +: 'int' and 'type'

I would call it inconsistent:

    3+4j.real means (3+4j).real
    3+4j.__class__ means 4+(4j).__class__

Enough ignorance from my side? :-)

regards,
Gerrit Holl.

-- 
Weather in Twenthe, Netherlands 15/11 09:25:
	5.0?C Few clouds overcast wind 3.1 m/s SSW (57 m above NAP)
-- 
In the councils of government, we must guard against the acquisition of
unwarranted influence, whether sought or unsought, by the
military-industrial complex. The potential for the disastrous rise of
misplaced power exists and will persist.
    -Dwight David Eisenhower, January 17, 1961
From ncoghlan at gmail.com  Mon Nov 15 11:44:15 2004
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon Nov 15 11:44:25 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <200411141918.iAEJIEvw023160@boa.lbl.gov>
References: <200411141918.iAEJIEvw023160@boa.lbl.gov>
Message-ID: <4198887F.8000100@iinet.net.au>

Ralf W. Grosse-Kunstleve wrote:
>>Knowing "I'm spending x% of the time executing Python code" just isn't 
>>really all that interesting,
> 
> 
> Right. Sorry if I gave the wrong impression that this could be
> interesting. It is indeed not. What is interesting is the estimated
> benefit of reimplementing a piece of Python in C/C++. This is in
> fact highly correlated with the time/tick.

Ah, OK, I think I better understand what you're trying to achieve now. 
Does knowing that the 2.4 profiler splits out C invocations from the 
calling Python methods make a difference here?

The 2.4 profiler is the only one I've made heavy use of - I'd forgotten 
that the 2.3 profiler didn't split out the C functions.

I've also discovered that the profiler documentation in Python 2.4 still 
describes the old behaviour (i.e. C-code time getting charged to the 
calling Python function). (I'll put a bug report on SF for that one)

I've got some sample output comparing the 2.3 and 2.4 profilers below. 
The interesting lines in the 2.4 output are the ones without a filename 
- those are the C functions.

Cheers,
Nick.

Python 2.3 profiler:
==================
$ python /usr/lib/python2.3/profile.py /usr/lib/python2.3/profile.py
usage: profile.py scriptfile [arg] ...
          6 function calls in 0.031 CPU seconds

    Ordered by: standard name

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
         1    0.015    0.015    0.015    0.015 <string>:1(?)
         1    0.000    0.000    0.000    0.000 profile.py:10(?)
         1    0.000    0.000    0.000    0.000 profile.py:103(Profile)
         1    0.000    0.000    0.000    0.000 profile.py:334(fake_code)
         1    0.000    0.000    0.000    0.000 profile.py:344(fake_frame)
         1    0.016    0.016    0.031    0.031 
profile:0(execfile('/usr/lib/pytho
n2.3/profile.py'))
         0    0.000             0.000          profile:0(profiler)
==================

And with Python 2.4:
==================
$ ./python -m profile -s time /usr/lib/python2.3/profile.py
usage: profile.py scriptfile [arg] ...
          10 function calls in 0.048 CPU seconds

    Ordered by: internal time

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
         1    0.032    0.032    0.032    0.032 :0(setprofile)
         1    0.016    0.016    0.016    0.016 :0(execfile)
         1    0.000    0.000    0.000    0.000 profile.py:334(fake_code)
         1    0.000    0.000    0.000    0.000 :0(exit)
         1    0.000    0.000    0.000    0.000 profile.py:10(?)
         1    0.000    0.000    0.000    0.000 profile.py:344(fake_frame)
         1    0.000    0.000    0.000    0.000 :0(hasattr)
         1    0.000    0.000    0.000    0.000 profile.py:103(Profile)
         1    0.000    0.000    0.048    0.048 
profile:0(execfile('/usr/lib/pytho
n2.3/profile.py'))
         1    0.000    0.000    0.048    0.048 <string>:1(?)
         0    0.000             0.000          profile:0(profiler)
==================

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From s.percivall at chello.se  Mon Nov 15 13:31:44 2004
From: s.percivall at chello.se (Simon Percivall)
Date: Mon Nov 15 13:31:48 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <20041115095401.GA19806@nl.linux.org>
References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz>
	<877e9a1704111420441c9e683c@mail.gmail.com>
	<419849B3.9020406@v.loewis.de>
	<20041115095401.GA19806@nl.linux.org>
Message-ID: <4F5FA7BF-3702-11D9-B2C1-0003934AD54A@chello.se>

On 2004-11-15, at 10.54, Gerrit wrote:
>
> It works for complex numbers:
>
>>>> 1j.imag
> 1.0
>
> ...in a rather surprising way:
>
>>>> 3+4j.real
> 3.0
>>>> f() + 5j.real
> 42.0
>
> I would have expected it to be 0.0, because I would have expected the
> '.' to have precedence over the '+', but it hasn't. apparantly.
>
> [...]
>
> I would call it inconsistent:
>
>     3+4j.real means (3+4j).real
>     3+4j.__class__ means 4+(4j).__class__

No, it means 3 + (4j.real) which is the same as 3 + 0.0. Both equals 
3.0.

//Simon

From ncoghlan at gmail.com  Mon Nov 15 14:41:59 2004
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon Nov 15 14:42:13 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <4197B42A.8020401@v.loewis.de>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>	<4195F883.8030309@iinet.net.au>	<20041114131708.GA9442@vicky.ecs.soton.ac.uk>
	<4197B42A.8020401@v.loewis.de>
Message-ID: <4198B227.8080309@iinet.net.au>

Martin v. L?wis wrote:
> Armin Rigo wrote:
> 
> Of course, whoever implements the change should make sure that slices
> are applicable consistently everwhere the documentation says they should
> be.

Random thought: the two uses of "["  and "]" in the sequence docs can 
get a little confusing. . . or maybe I'm just tired ;)

Anyway, the sequence and mutable sequence sections of the documentation 
don't reveal anything other than list.pop(). It seems to be the only 
normal method that accepts an index as an argument. Everything else 
looks to be dealt with via standard subscripts and the associated magic 
methods:

   Retrieval: x = L[i] | x = L.__getitem__(i)
   Setting: L[i] = x | L.__setitem__(i, x)
   Deletion: del L[i] | L.__delitem__(i, x)

So it looks like list.pop simply got left out because it wasn't a magic 
method, and the idea of permitting extended slicing for it just didn't 
come up (well, until now).**

Anyway, if this is implemented, array.pop and UserList.pop should allow 
slices, too, as they are described as working like list.pop.

The other container classes I've checked are either immutable, or don't 
allow indexed access beyond the basics (string, unicode, set, deque).

Cheers,
Nick.

** Some items of possible historical interest:
http://www.python.org/doc/2.3.4/whatsnew/section-slices.html
http://sourceforge.net/tracker/?group_id=5470&atid=305470&aid=400998&func=detail
http://mail.python.org/pipermail/python-dev/2002-May/023874.html

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From nbastin at opnet.com  Mon Nov 15 16:19:08 2004
From: nbastin at opnet.com (Nick Bastin)
Date: Mon Nov 15 16:19:43 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <4198887F.8000100@iinet.net.au>
References: <200411141918.iAEJIEvw023160@boa.lbl.gov>
	<4198887F.8000100@iinet.net.au>
Message-ID: <B247747B-3719-11D9-B482-000D932927FE@opnet.com>


On Nov 15, 2004, at 5:44 AM, Nick Coghlan wrote:

> Ah, OK, I think I better understand what you're trying to achieve now.
> Does knowing that the 2.4 profiler splits out C invocations from the
> calling Python methods make a difference here?
>
> The 2.4 profiler is the only one I've made heavy use of - I'd forgotten
> that the 2.3 profiler didn't split out the C functions.
>
> I've also discovered that the profiler documentation in Python 2.4 
> still
> describes the old behaviour (i.e. C-code time getting charged to the
> calling Python function). (I'll put a bug report on SF for that one)

Doh, I thought I'd found all of the places in the doc that needed to be 
corrected.  If you file an SF bug, or just let me know where, I'll fix 
the documentation to reflect the current state of things.

--
Nick

From fumanchu at amor.org  Mon Nov 15 21:02:28 2004
From: fumanchu at amor.org (Robert Brewer)
Date: Mon Nov 15 21:04:00 2004
Subject: [Python-Dev] os.access versus os.exist
Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net>

It would be awfully nice (on posix platforms, for my use-case) to find
out whether a file is inaccessible due to permission restrictions, or
due to non-existence.

Poking around in posixmodule.c, it seems os.access() uses
posix_access(PyObject *self, PyObject *args), which calls
_access/_waccess as described here:
http://msdn.microsoft.com/library/en-us/vclib/html/_crt__access.2c_._wac
cess.asp


Run-Time Library Reference 	 
_access, _waccess

Determine file-access permission.

int _access( 
   const char *path, 
   int mode 
);
int _waccess( 
   const wchar_t *path, 
   int mode 
);

Parameters

path
    File or directory path.
mode
    Permission setting.

Return Value

Each function returns 0 if the file has the given mode. The function
returns -1 if the named file does not exist or is not accessible in the
given mode; in this case, errno is set as follows:

EACCES
    Access denied: file's permission setting does not allow specified
access.
ENOENT
    Filename or path not found.

-----------------

Here's the current posixmodule.posix_access (rev 2.329):

static PyObject *
posix_access(PyObject *self, PyObject *args)
{
	char *path;
	int mode;
	int res;

#ifdef Py_WIN_WIDE_FILENAMES
	if (unicode_file_names()) {
		PyUnicodeObject *po;
		if (PyArg_ParseTuple(args, "Ui:access", &po, &mode)) {
			Py_BEGIN_ALLOW_THREADS
			/* PyUnicode_AS_UNICODE OK without thread lock
as
			   it is a simple dereference. */
			res = _waccess(PyUnicode_AS_UNICODE(po), mode);
			Py_END_ALLOW_THREADS
			return(PyBool_FromLong(res == 0));
		}
		/* Drop the argument parsing error as narrow strings
		   are also valid. */
		PyErr_Clear();
	}
#endif
	if (!PyArg_ParseTuple(args, "si:access", &path, &mode))
		return NULL;
	Py_BEGIN_ALLOW_THREADS
	res = access(path, mode);
	Py_END_ALLOW_THREADS
	return(PyBool_FromLong(res == 0));
}

-----------------

It seems to me that, if EACCES and ENOENT were made available (in a
future version of Python), I could solve my problem easily. One way
would be to raise OSError within os.access if the result of _waccess ==
ENOENT, but this would cause problems with old code which depends upon
the old behavior. Can we add an optional "error_if_not_found" arg to
os.access? Return the result of _waccess instead of True/False? Make a
new os function with the desired behavior?

The next issue, then, is other platforms: is similar functionality
available in OS/2, unix, etc.?

Finally, if anyone has any immediate workarounds for me ;) answer in
private or on c.l.p.


Robert Brewer
MIS
Amor Ministries
fumanchu@amor.org
From rwgk at cci.lbl.gov  Mon Nov 15 21:28:19 2004
From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve)
Date: Mon Nov 15 21:28:25 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
Message-ID: <200411152028.iAFKSJgq040962@boa.lbl.gov>

Nick Coghlan wrote:
> Ralf W. Grosse-Kunstleve wrote:
> > >Knowing "I'm spending x% of the time executing Python code" just isn't 
> > >really all that interesting,
> >  
> >  Right. Sorry if I gave the wrong impression that this could be
> >  interesting. It is indeed not. What is interesting is the estimated
> >  benefit of reimplementing a piece of Python in C/C++. This is in
> >  fact highly correlated with the time/tick.
> 
> Ah, OK, I think I better understand what you're trying to achieve now. 
> Does knowing that the 2.4 profiler splits out C invocations from the 
> calling Python methods make a difference here?

Yes, this is a very helpful improvement. Thanks for pointing this
out! It gives us a more direct way of pin-pointing the time sinks.
However, I see time.time()/sys.gettickeraccumulation() as a
complementary source of information:

  profiler:
    factor 8 runtime penalty
    very detailed information

  time.time()/sys.gettickeraccumulation():
    no runtime penalty
    global information

As we work on new algorithms we can easily monitor the time/tick
without a runtime penalty. As long as the ratio stays reasonably high
(e.g. because we use numarray-like array operations already coded in
C++) we don't have to do anything. If the ratio goes down significantly
we can run the profiler to do detailed analysis of the new code.

Cheers,
        Ralf
From gvanrossum at gmail.com  Mon Nov 15 21:51:44 2004
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Mon Nov 15 21:51:47 2004
Subject: [Python-Dev] os.access versus os.exist
In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net>
References: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net>
Message-ID: <ca471dc2041115125131738f0b@mail.gmail.com>

> It would be awfully nice (on posix platforms, for my use-case) to find
> out whether a file is inaccessible due to permission restrictions, or
> due to non-existence.

Why can't you solve this by doing a stat() when access() returns False?

Also, in general I recommend against using access() except for setuid
apps that need to test whether the *real* user id has the desired
access; there are many additional reasons why opening a file fails
even though access() returns True; amongst the causes can be race
conditions (the file was changed after you called access() or
conditions like ETXTBSY (you can't write an executable file that is
being executed) or ENOMEM (a kernel resource problem).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
From martin at v.loewis.de  Mon Nov 15 21:58:55 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon Nov 15 21:58:57 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <877e9a170411142330661d3a33@mail.gmail.com>
References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz>	
	<877e9a1704111420441c9e683c@mail.gmail.com>	
	<419849B3.9020406@v.loewis.de>
	<877e9a170411142330661d3a33@mail.gmail.com>
Message-ID: <4199188F.7000906@v.loewis.de>

Michael Walter wrote:
> I think it's inconsistent because it works for "list literals" but not
> for "integer literals". What do I miss?

That tokenization works consistently, using the "maximum match"
strategy.

If you meant to parse

1.__class__

as "<integer 1>" "." "<identifier __class__>", not as "<float 1.0>"
"<identifier __class_>", then, for consistency, you should also parse
the second line of

   s = 100
   prints

as "<keyword print>" "<identifier s>", not as "<identifier prints>".
Since the latter is certainly undesirable, the former must be
followed for consistency.

You easily derive the rule "a space is necessary between keyword
and identifier" from the second example; you should, for consistency,
also derive the rule "a space is necessary between an integer
literal and a dot".

As for "list literals": The Python grammar calls them "displays",
not "literals", as they don't (necessarily) denote a literal value,
e.g. in [1,2,x,y+5].

Regards,
Martin
From martin at v.loewis.de  Mon Nov 15 22:00:41 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon Nov 15 22:00:42 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <419861AB.6020400@gmx.de>
References: <4197A392.7080100@gmx.de> <4197CE89.2060801@v.loewis.de>
	<419861AB.6020400@gmx.de>
Message-ID: <419918F9.3070407@v.loewis.de>

Kay Schluehr wrote:
> Thank You. I agree completely with Your consistency rule, which is actually
> a disambiguation. On the other hand It's not just me who was confused about
> method calls on int-literals and for convenience it should be 
> emphasized, at least
> in the language reference but perhaps from the beginning, in the tutorial.

Patches to the language reference are welcome. I'm not so sure about
patches to the tutorial - this is a fairly infrequent problem, so the
tutorial should not bother the reader with this minor nit (IMO).

Regards,
Martin
From martin at v.loewis.de  Mon Nov 15 22:34:23 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon Nov 15 22:34:24 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <4198B227.8080309@iinet.net.au>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>	<4195F883.8030309@iinet.net.au>	<20041114131708.GA9442@vicky.ecs.soton.ac.uk>	<4197B42A.8020401@v.loewis.de>
	<4198B227.8080309@iinet.net.au>
Message-ID: <419920DF.9070002@v.loewis.de>

Nick Coghlan wrote:
> Anyway, the sequence and mutable sequence sections of the documentation 
> don't reveal anything other than list.pop(). It seems to be the only 
> normal method that accepts an index as an argument. 

This isn't really true:

s.index(x[, i[, j]])  	
    return smallest k such that s[k] == x and i <= k < j  	
s.insert(i, x)  	
    same as s[i:i] = [x]

However, I don't think this naturally extends to slices:
for index, there is no <= relationship for slices (atleast
not a natural one), and for insert, you can use slices as
start- and end-index of a slice.

> Anyway, if this is implemented, array.pop and UserList.pop should allow 
> slices, too, as they are described as working like list.pop.

Right. For UserList, it probably falls out naturally.

Regards,
Martin
From fumanchu at amor.org  Mon Nov 15 22:36:24 2004
From: fumanchu at amor.org (Robert Brewer)
Date: Mon Nov 15 22:37:56 2004
Subject: [Python-Dev] os.access versus os.exist
Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245202@exchange.hqamor.amorhq.net>

I wrote:
> > It would be awfully nice (on posix platforms, for my 
> > use-case) to find out whether a file is inaccessible
> > due to permission restrictions, or due to non-existence.

Guido van Rossum replied:
> Why can't you solve this by doing a stat() when access() 
> returns False?

In my current case, stat() gives an error (even though the file exists):

OSError: [Errno 2] No such file or directory:
u'S:/Scans/Projects/2004/TJ/04TJ0267.jpg'

Running the same stat() call in the interpreter (as opposed to inside my
mod_python app) gives no such error; I get normal stat output, so the
file does exist. I figured since the app was running as LocalSystem it
was a permissions issue. [One quick sanity check later] Yes, if I run
Apache2 under my account, stat() does not error.

Hm. I see now I'm following the wrong issue. It has more to do with how
Windows shares mapped drives between users (it doesn't). If I use the
UNC path, I don't have an issue.

Thanks for pointing me away from the wrong direction. ;)


Robert Brewer
MIS
Amor Ministries
fumanchu@amor.org
From fumanchu at amor.org  Mon Nov 15 22:50:10 2004
From: fumanchu at amor.org (Robert Brewer)
Date: Mon Nov 15 22:51:42 2004
Subject: [Python-Dev] os.access versus os.exist
Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245203@exchange.hqamor.amorhq.net>

Me:
> > > It would be awfully nice (on posix platforms, for my 
> > > use-case) to find out whether a file is inaccessible
> > > due to permission restrictions, or due to non-existence.

Guido van Rossum:
> > Why can't you solve this by doing a stat() when access() 
> > returns False?

Me:
> In my current case, stat() gives an error (even though the 
> file exists):
> 
> OSError: [Errno 2] No such file or directory:
> u'S:/Scans/Projects/2004/TJ/04TJ0267.jpg'
> 
> Running the same stat() call in the interpreter (as opposed 
> to inside my mod_python app) gives no such error; I get normal stat
> output, so the file does exist. I figured since the app was running
> as LocalSystem it was a permissions issue. [One quick sanity check
> later] Yes, if I run Apache2 under my account, stat() does not error.
> 
> Hm. I see now I'm following the wrong issue. It has more to 
> do with how Windows shares mapped drives between users (it doesn't).
> If I use the UNC path, I don't have an issue.

Bah. Spoke too soon. I still have the issue even if I use UNC paths.

In Pythonwin interactive session (my logon):
   os.stat() returns a tuple,
   os.path.exists() returns True, and
   os.access() returns True.

Inside the app (running under LocalSystem on the same Win2k machine):
   os.stat() raises OSError: [Errno 2] No such file or directory,
   os.path.exists() returns False, and
   os.access() returns False.


Robert Brewer
MIS
Amor Ministries
fumanchu@amor.org
From martin at v.loewis.de  Mon Nov 15 22:58:41 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon Nov 15 22:58:42 2004
Subject: [Python-Dev] os.access versus os.exist
In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net>
References: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net>
Message-ID: <41992691.8040708@v.loewis.de>

Robert Brewer wrote:
> It would be awfully nice (on posix platforms, for my use-case) to find
> out whether a file is inaccessible due to permission restrictions, or
> due to non-existence.

That's easy enough: just check the exception you get when you try
opening the file.

Don't trust that a negative answer from access() means that you cannot
open the file, or that a positive answer means that you can. In short,
the implementation of access(2) might lie to you. For example, you might
have write access, but writing fails because you are over quota. Or,
you are running setuid, and the effective uid is different from the
real one.

> Can we add an optional "error_if_not_found" arg to
> os.access? Make a
> new os function with the desired behavior?

Either would be fine. Changing the return value would not, since
you would have to negate the result.

> The next issue, then, is other platforms: is similar functionality
> available in OS/2, unix, etc.?

On Unix, access(2) can return (in errno) these errors:
EACCES, ELOOP, ENAMETOOLONG, ENOENT, ENOTDIR, EROFS, and, optionally,
EFAULT, EINVAL, EIO, ENOMEM, ETXTBSY.

Regards,
Martin
From fumanchu at amor.org  Mon Nov 15 23:16:13 2004
From: fumanchu at amor.org (Robert Brewer)
Date: Mon Nov 15 23:17:45 2004
Subject: [Python-Dev] os.access versus os.exist
Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245204@exchange.hqamor.amorhq.net>

Martin v. L?wis wrote:
> Robert Brewer wrote:
> > It would be awfully nice (on posix platforms, for my 
> > use-case) to find out whether a file is inaccessible
> > due to permission restrictions, or due to non-existence.
> 
> That's easy enough: just check the exception you get when you try
> opening the file.

Brilliant! I'll do that. I didn't think of open() because I don't need to open it--just see if it's there.

/*bonk* self


Robert Brewer
MIS
Amor Ministries
fumanchu@amor.org
From tdelaney at avaya.com  Tue Nov 16 00:04:41 2004
From: tdelaney at avaya.com (Delaney, Timothy C (Timothy))
Date: Tue Nov 16 00:05:12 2004
Subject: [Python-Dev] Int literals and method calls
Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE4A90AF@au3010avexu1.global.avaya.com>

"Martin v. L?wis" wrote:

> If you meant to parse
> 
> 1.__class__
> 
> as "<integer 1>" "." "<identifier __class__>", not as "<float 1.0>"
> "<identifier __class_>"

On a related note ...

All this would just go away if in Python 3.0 (or even earlier ;) floats required something after the decimal point i.e. to get '<float 1.0>' you had to type '1.0' and '1.' by itself was a syntax error.

I've never actually seen the use case for having '1.' parse as '<float 1.0>' although I'm sure there is one.

Tim Delaney
From michael.walter at gmail.com  Tue Nov 16 00:07:45 2004
From: michael.walter at gmail.com (Michael Walter)
Date: Tue Nov 16 00:07:48 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <4199188F.7000906@v.loewis.de>
References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz>
	<877e9a1704111420441c9e683c@mail.gmail.com>
	<419849B3.9020406@v.loewis.de>
	<877e9a170411142330661d3a33@mail.gmail.com>
	<4199188F.7000906@v.loewis.de>
Message-ID: <877e9a1704111515073bc2900d@mail.gmail.com>

I understand, thanks.

- Michael


On Mon, 15 Nov 2004 21:58:55 +0100, Martin v. L?wis <martin@v.loewis.de> wrote:
> Michael Walter wrote:
> > I think it's inconsistent because it works for "list literals" but not
> > for "integer literals". What do I miss?
> 
> That tokenization works consistently, using the "maximum match"
> strategy.
> 
> If you meant to parse
> 
> 1.__class__
> 
> as "<integer 1>" "." "<identifier __class__>", not as "<float 1.0>"
> "<identifier __class_>", then, for consistency, you should also parse
> the second line of
> 
>    s = 100
>    prints
> 
> as "<keyword print>" "<identifier s>", not as "<identifier prints>".
> Since the latter is certainly undesirable, the former must be
> followed for consistency.
> 
> You easily derive the rule "a space is necessary between keyword
> and identifier" from the second example; you should, for consistency,
> also derive the rule "a space is necessary between an integer
> literal and a dot".
> 
> As for "list literals": The Python grammar calls them "displays",
> not "literals", as they don't (necessarily) denote a literal value,
> e.g. in [1,2,x,y+5].
> 
> Regards,
> Martin
>
From ncoghlan at gmail.com  Tue Nov 16 02:34:03 2004
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue Nov 16 02:34:13 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <419920DF.9070002@v.loewis.de>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>	<4195F883.8030309@iinet.net.au>	<20041114131708.GA9442@vicky.ecs.soton.ac.uk>	<4197B42A.8020401@v.loewis.de>
	<4198B227.8080309@iinet.net.au> <419920DF.9070002@v.loewis.de>
Message-ID: <4199590B.1040908@iinet.net.au>

Martin v. L?wis wrote:
> Nick Coghlan wrote:
> 
>> Anyway, the sequence and mutable sequence sections of the 
>> documentation don't reveal anything other than list.pop(). It seems to 
>> be the only normal method that accepts an index as an argument. 
> 
> 
> This isn't really true:
> 
> s.index(x[, i[, j]])     
>    return smallest k such that s[k] == x and i <= k < j     
> s.insert(i, x)     
>    same as s[i:i] = [x]
> 
> However, I don't think this naturally extends to slices:
> for index, there is no <= relationship for slices (atleast
> not a natural one), and for insert, you can use slices as
> start- and end-index of a slice.

Yes. Those two were on my list initially, but then I tried to figure out 
how using a slice would actually *work* for them. At which point, I took 
them back off the list - slice arguments just didn't make any sense.

So I think we're down to two things to implement - list.pop and 
array.pop. As you say, UserList.pop should just be a different way of 
spelling list.pop.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From greg.ewing at canterbury.ac.nz  Tue Nov 16 02:50:53 2004
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue Nov 16 02:51:00 2004
Subject: [Python-Dev] os.access versus os.exist
In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net>
References: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net>
Message-ID: <41995CFD.6030902@canterbury.ac.nz>

Robert Brewer wrote:
> It would be awfully nice (on posix platforms, for my use-case) to find
> out whether a file is inaccessible due to permission restrictions, or
> due to non-existence.

On a posix system, you should be able to find out by
checking the error code from the exception. That would
be better than making a separate call, since it would
avoid any possible race conditions. Don't look before
you leap, etc...

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+

From greg.ewing at canterbury.ac.nz  Tue Nov 16 03:00:53 2004
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue Nov 16 03:01:00 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DE4A90AF@au3010avexu1.global.avaya.com>
References: <338366A6D2E2CA4C9DAEAE652E12A1DE4A90AF@au3010avexu1.global.avaya.com>
Message-ID: <41995F55.8040705@canterbury.ac.nz>

Delaney, Timothy C (Timothy) wrote:
> All this would just go away if in Python 3.0 (or even earlier ;)
 > floats required something after the decimal point i.e. to get
> '<float 1.0>' you had to type '1.0' and '1.' by itself was a
 > syntax error.

I always write float literals that way anyhow, in the
interests of clarity. So I wouldn't mind if this
happened either.

+0

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg@cosc.canterbury.ac.nz	   +--------------------------------------+

From ncoghlan at gmail.com  Tue Nov 16 03:02:39 2004
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue Nov 16 03:02:45 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <200411152028.iAFKSJgq040962@boa.lbl.gov>
References: <200411152028.iAFKSJgq040962@boa.lbl.gov>
Message-ID: <41995FBF.3080207@iinet.net.au>

Ralf W. Grosse-Kunstleve wrote:
> As we work on new algorithms we can easily monitor the time/tick
> without a runtime penalty. As long as the ratio stays reasonably high
> (e.g. because we use numarray-like array operations already coded in
> C++) we don't have to do anything. If the ratio goes down significantly
> we can run the profiler to do detailed analysis of the new code.

That seems like a reasonable approach to me - one could easily make use 
of the ticker information in a test framework. So I'll withdraw my 
objection and give the new function a +0.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at gmail.com  Tue Nov 16 03:22:55 2004
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue Nov 16 03:23:01 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <B247747B-3719-11D9-B482-000D932927FE@opnet.com>
References: <200411141918.iAEJIEvw023160@boa.lbl.gov>	<4198887F.8000100@iinet.net.au>
	<B247747B-3719-11D9-B482-000D932927FE@opnet.com>
Message-ID: <4199647F.7000309@iinet.net.au>

Nick Bastin wrote:
> 
> On Nov 15, 2004, at 5:44 AM, Nick Coghlan wrote:
>> I've also discovered that the profiler documentation in Python 2.4 still
>> describes the old behaviour (i.e. C-code time getting charged to the
>> calling Python function). (I'll put a bug report on SF for that one)
> 
> 
> Doh, I thought I'd found all of the places in the doc that needed to be 
> corrected.  If you file an SF bug, or just let me know where, I'll fix 
> the documentation to reflect the current state of things.

The bug report is on SF as of last night. . . *checks browser history 
for the number*. . . it's bug #1066607.

Section 10.6 is the one on Limitations. Also, the front page of the 
hotshot documentation could probably use a warning that the actual 
profile module copes with C extensions better than hotshot does (I think 
hotshot still has the old "C time goes to the calling Python function" 
behaviour - I haven't checked that recently, though)

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From bob at redivi.com  Tue Nov 16 05:16:21 2004
From: bob at redivi.com (Bob Ippolito)
Date: Tue Nov 16 05:17:06 2004
Subject: [Python-Dev] Int literals and method calls
In-Reply-To: <41995F55.8040705@canterbury.ac.nz>
References: <338366A6D2E2CA4C9DAEAE652E12A1DE4A90AF@au3010avexu1.global.avaya.com>
	<41995F55.8040705@canterbury.ac.nz>
Message-ID: <457ECE6E-3786-11D9-9BE2-000A9567635C@redivi.com>


On Nov 16, 2004, at 4:00 AM, Greg Ewing wrote:

> Delaney, Timothy C (Timothy) wrote:
>> All this would just go away if in Python 3.0 (or even earlier ;)
> > floats required something after the decimal point i.e. to get
>> '<float 1.0>' you had to type '1.0' and '1.' by itself was a
> > syntax error.
>
> I always write float literals that way anyhow, in the
> interests of clarity. So I wouldn't mind if this
> happened either.
>
> +0

I tend to use integer float literals without the trailing zero from the 
interpreter if I want float division to happen without typing "from 
__future__ import division".  Having to type an additional character 
isn't a problem, though.

+0

-bob

From carribeiro at gmail.com  Tue Nov 16 13:13:29 2004
From: carribeiro at gmail.com (Carlos Ribeiro)
Date: Tue Nov 16 13:13:32 2004
Subject: [Python-Dev] Looking for authoritative documentation on packages,
	import & ihooks
In-Reply-To: <864d3709041115075629d15045@mail.gmail.com>
References: <864d3709041115075629d15045@mail.gmail.com>
Message-ID: <864d37090411160413756ff4c5@mail.gmail.com>

I have posted this question on c.l.py yesterday but received no
answers. Given the complexity of the theme, I hope that people will
not mind if I ask it here.

I am looking for information on packages & import hooks, including
simple examples on how to implement a simple import hook. Quick
googling turns out some documents, such as:

-- PEP 302:
   http://www.python.org/peps/pep-0302.html

-- What's new on Python 2.3:
   http://www.python.org/doc/2.3.4/whatsnew/section-pep302.html

-- Import SIG:
   http://www.python.org/sigs/import-sig/

-- PyDoc for the ihooks module:
   http://pydoc.org/2.3/ihooks.html

However, after reading them, I ended up more confused than before.
There are lots of references to the ihooks standard module, but there
is no documentation about it on the standard Python docs. The "Modules
That Needs Docs" Wiki entry
(http://www.python.org/moin/ModulesThatNeedDocs) lists ihooks and
imputils, but noone has volunteered to do it yet (worse, not even a
bug report was filled; this is something that I'm going to do).
Packages are also not documented (there is still a reference to an
essay from 1.5 days on the Python 2.3 docs).

For what I could see, the import system went over several iterations
and patches since the 1.5 release (when packages were added). It's now
hard to understand what is the preferred hooking mechanism as far as
Python 2.4 (and future versions) are concerned. For example: it is not
clear, from a simple reading, if PEP 302 is up-to-date, or if it is
considered the preferred approach for import hooks. For this reason, I
would like to know if there is any document which may be considered
"authoritative" for the import system (besides ihooks.py and
import.c). If *all* that exists is the source code, well, I guess I'll
have to read it. But anyway... pointers to simple examples also are
helpful. Thanks in advance.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro@gmail.com
mail: carribeiro@yahoo.com
From pje at telecommunity.com  Tue Nov 16 16:50:51 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue Nov 16 16:49:48 2004
Subject: [Python-Dev] Looking for authoritative documentation on
	packages, import & ihooks
In-Reply-To: <864d37090411160413756ff4c5@mail.gmail.com>
References: <864d3709041115075629d15045@mail.gmail.com>
	<864d3709041115075629d15045@mail.gmail.com>
Message-ID: <5.1.1.6.0.20041116104453.02dbe740@mail.telecommunity.com>

At 10:13 AM 11/16/04 -0200, Carlos Ribeiro wrote:
>For example: it is not
>clear, from a simple reading, if PEP 302 is up-to-date, or if it is
>considered the preferred approach for import hooks. For this reason, I
>would like to know if there is any document which may be considered
>"authoritative" for the import system (besides ihooks.py and
>import.c). If *all* that exists is the source code, well, I guess I'll
>have to read it. But anyway... pointers to simple examples also are
>helpful. Thanks in advance.

PEP 302 is essentially up-to-date (although not fully implemented; see the 
notes in the PEP), and yes, it's the preferred approach, at least if you're 
1) developing an import hook, 2) you're focused on "location" extensions, 
i.e. importing from places not currently supported, as opposed to adding 
new filetypes or new ways of compiling, etc., and 3) want to interoperate 
well with other import hooks.

Regarding the use of other import hook modules such as ihooks and imputil, 
they have in the past been the subject of vigorous debate as to which is 
the "preferred" way of implementing import hooks.  There is also a 
separately distributed import hook module, 'iu', that inspired the current 
PEP 302 approach.  With respect to these various modules, it is apparently 
a matter of programmer taste and/or goals as to which is the best.

From michael.walter at gmail.com  Tue Nov 16 17:01:30 2004
From: michael.walter at gmail.com (Michael Walter)
Date: Tue Nov 16 17:01:34 2004
Subject: [Python-Dev] PEP 310 Status
Message-ID: <877e9a1704111608015caff2fa@mail.gmail.com>

Hello everyone,

I was just wondering about the status of PEP 310 ("with" statement) -
has there been any concensus/plan to implement it? (I tried to google
the answer, but failed ;-)

How about the potentiel inclusion of user-defined "blocks"? I suppose
this would be only a Python 3000 thing, if ever included?

Cheers,
Michael
From pje at telecommunity.com  Tue Nov 16 17:27:58 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue Nov 16 17:26:55 2004
Subject: [Python-Dev] PEP 310 Status
In-Reply-To: <877e9a1704111608015caff2fa@mail.gmail.com>
Message-ID: <5.1.1.6.0.20041116112416.02705890@mail.telecommunity.com>

At 11:01 AM 11/16/04 -0500, Michael Walter wrote:
>Hello everyone,
>
>I was just wondering about the status of PEP 310 ("with" statement) -
>has there been any concensus/plan to implement it? (I tried to google
>the answer, but failed ;-)

Guido has previously said that "with" is reserved for a use similar to the 
use of the "with" statement in Turbo Pascal and Visual Basic; that is, a 
shortcut for referring to a long expression.  Whether that means PEP 310 
needs a new keyword, or that it needs to play a dual role, I'm not sure.


>How about the potentiel inclusion of user-defined "blocks"? I suppose
>this would be only a Python 3000 thing, if ever included?

With the advent of PEP 318, it's now possible to do block-like things with 
closures and decorators, e.g.:

     @with_lock(mylock)
     def do_something():
         # code here

     do_something()

This is somewhat ugly, however, when used in-line.  On the other hand, this 
could also be viewed as an encouragement to modularize more, in order to 
have more readable code.  :)

From michael.walter at gmail.com  Tue Nov 16 17:35:57 2004
From: michael.walter at gmail.com (Michael Walter)
Date: Tue Nov 16 17:35:58 2004
Subject: [Python-Dev] PEP 310 Status
In-Reply-To: <5.1.1.6.0.20041116112416.02705890@mail.telecommunity.com>
References: <877e9a1704111608015caff2fa@mail.gmail.com>
	<5.1.1.6.0.20041116112416.02705890@mail.telecommunity.com>
Message-ID: <877e9a1704111608352f87aa94@mail.gmail.com>

This doesn't quite work for different use cases of blocks, such as
generating markup using something like "with <tag>: <suite>".

Of course one (could|should|will) argue that this is an attempt to
extend Python's syntax, basically "abusing" with beyond its original
scope. I don't know whether this is a bad thing.

Cheers,
Michael


On Tue, 16 Nov 2004 11:27:58 -0500, Phillip J. Eby
<pje@telecommunity.com> wrote:
> At 11:01 AM 11/16/04 -0500, Michael Walter wrote:
> >Hello everyone,
> >
> >I was just wondering about the status of PEP 310 ("with" statement) -
> >has there been any concensus/plan to implement it? (I tried to google
> >the answer, but failed ;-)
> 
> Guido has previously said that "with" is reserved for a use similar to the
> use of the "with" statement in Turbo Pascal and Visual Basic; that is, a
> shortcut for referring to a long expression.  Whether that means PEP 310
> needs a new keyword, or that it needs to play a dual role, I'm not sure.
> 
> 
> >How about the potentiel inclusion of user-defined "blocks"? I suppose
> >this would be only a Python 3000 thing, if ever included?
> 
> With the advent of PEP 318, it's now possible to do block-like things with
> closures and decorators, e.g.:
> 
>      @with_lock(mylock)
>      def do_something():
>          # code here
> 
>      do_something()
> 
> This is somewhat ugly, however, when used in-line.  On the other hand, this
> could also be viewed as an encouragement to modularize more, in order to
> have more readable code.  :)
> 
>
From pedronis at bluewin.ch  Tue Nov 16 17:42:57 2004
From: pedronis at bluewin.ch (Samuele Pedroni)
Date: Tue Nov 16 17:39:31 2004
Subject: [Python-Dev] PEP 310 Status
In-Reply-To: <877e9a1704111608015caff2fa@mail.gmail.com>
References: <877e9a1704111608015caff2fa@mail.gmail.com>
Message-ID: <419A2E11.8020100@bluewin.ch>

Michael Walter wrote:
> Hello everyone,
> 
> I was just wondering about the status of PEP 310 ("with" statement) -
> has there been any concensus/plan to implement it? (I tried to google
> the answer, but failed ;-)

I suppose someone should write an implementation for it, and start 
championing for it again (And a new keyword may be needed because 'with' 
is preempted). The lingering suspect that a more general solution could 
enable it as application and more has worked a bit against it, also by

> How about the potentiel inclusion of user-defined "blocks"? I suppose
> this would be only a Python 3000 thing, if ever included?
> 

generating enormous discussions about this one other topic that went 
mostly nowhere. Don't know whether having another one is a good idea.
Beforehand someone should write a PEP with a concrete proposal in the 
fullness of its details, or maybe even a set of different PEP/proposals.

There is no self-evident and beautiful solution for this, it seems.
Because of some aspects of Python ( focus on readability, non uniformity 
(think vs Lisp/Smalltalk) e.g. expression vs statement distinction, 
class def suite semantics vs. functions definitions..., generators
design decisions) this will very likely entail some sort of compromise.

Depending of how much expressiveness one wants, because of 
non-uniformity, the "right" solution may be even be macros, not just 
blocks. For blocks things like:

- are we up to blurring a bit the expression vs. statement divide,
- can we live with a bit lisp-looking parenthesized constructs in 
expressions,
- Or limiting things to just _one_ block "capture" statement is enough
- do we want to support class-def-like constructs? interested in 
capturing order? markup construction?
- do we want anonymous function vs. suite semantics?
- what the is interaction with yield?
- or with ideas and likely need of rebinding for lexical-scoped variables
- etc

should be considered/addressed

regards







From sxanth at ceid.upatras.gr  Tue Nov 16 19:51:06 2004
From: sxanth at ceid.upatras.gr (Stelios Xanthakis)
Date: Tue Nov 16 18:56:30 2004
Subject: [Python-Dev] __pycode__ extension
Message-ID: <419A4C1A.5060502@ceid.upatras.gr>


Hi.


I posted a message on c.l.p a couple of days ago about a
python patch which adds a member __pycde__ to functions and
classes. This member is a string which holds the python code of
the function/class. (It works in interactively defined code
and exec'd definitions)

Supposing that: I ported this to 2.4b2 and made it have zero overhead
when python runs in normal mode, is there any chance it would be
considered a candidate for inclusion in mainline python?

If yes, I'd appreciate it if somebody took a look at the patch
because there may be a better way to do it.

Patch vs 2.3.4:
http://students.ceid.upatras.gr/~sxanth/ISYSTEM/python-PIESS.tar.bz2

Guido?


c-ya

Stelios

From aahz at pythoncraft.com  Tue Nov 16 19:45:48 2004
From: aahz at pythoncraft.com (Aahz)
Date: Tue Nov 16 19:45:51 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <419A4C1A.5060502@ceid.upatras.gr>
References: <419A4C1A.5060502@ceid.upatras.gr>
Message-ID: <20041116184548.GA5013@panix.com>

On Tue, Nov 16, 2004, Stelios Xanthakis wrote:
> 
> I posted a message on c.l.p a couple of days ago about a
> python patch which adds a member __pycde__ to functions and
> classes. This member is a string which holds the python code of
> the function/class. (It works in interactively defined code
> and exec'd definitions)
> 
> Supposing that: I ported this to 2.4b2 and made it have zero overhead
> when python runs in normal mode, is there any chance it would be
> considered a candidate for inclusion in mainline python?

There is zero chance it'll go into 2.4.  There's a decent chance it'll
go into 2.5, but if you get any pushback on the feature, be prepared to
write up a full PEP.
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

WiFi is the SCSI of the 21st Century -- there are fundamental technical
reasons for sacrificing a goat.  (with no apologies to John Woods)
From jhylton at gmail.com  Tue Nov 16 20:01:47 2004
From: jhylton at gmail.com (Jeremy Hylton)
Date: Tue Nov 16 20:03:24 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <419A4C1A.5060502@ceid.upatras.gr>
References: <419A4C1A.5060502@ceid.upatras.gr>
Message-ID: <e8bf7a5304111611013ad74af5@mail.gmail.com>

On Tue, 16 Nov 2004 10:51:06 -0800, Stelios Xanthakis
<sxanth@ceid.upatras.gr> wrote:
> 
> Hi.
> 
> I posted a message on c.l.p a couple of days ago about a
> python patch which adds a member __pycde__ to functions and
> classes. This member is a string which holds the python code of
> the function/class. (It works in interactively defined code
> and exec'd definitions)

Functions already have a reference to code objects.  How is this different?

Jeremy
From martin at v.loewis.de  Tue Nov 16 20:16:40 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue Nov 16 20:16:42 2004
Subject: [Python-Dev] os.access versus os.exist
In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245203@exchange.hqamor.amorhq.net>
References: <3A81C87DC164034AA4E2DDFE11D258E3245203@exchange.hqamor.amorhq.net>
Message-ID: <419A5218.9090500@v.loewis.de>

Robert Brewer wrote:
> Inside the app (running under LocalSystem on the same Win2k machine):
>    os.stat() raises OSError: [Errno 2] No such file or directory,
>    os.path.exists() returns False, and
>    os.access() returns False.

LocalSystem cannot access network shares. However, this is OT for
python-dev.

Regards,
Martin
From martin at v.loewis.de  Tue Nov 16 20:20:20 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue Nov 16 20:20:26 2004
Subject: [Python-Dev] Looking for authoritative documentation on packages, 
	import & ihooks
In-Reply-To: <864d37090411160413756ff4c5@mail.gmail.com>
References: <864d3709041115075629d15045@mail.gmail.com>
	<864d37090411160413756ff4c5@mail.gmail.com>
Message-ID: <419A52F4.2020408@v.loewis.de>

Carlos Ribeiro wrote:
> For this reason, I
> would like to know if there is any document which may be considered
> "authoritative" for the import system (besides ihooks.py and
> import.c).

This question is OT for python-dev.

Python is open source, so what is wrong with reading import.c?

Regards,
Martin
From pje at telecommunity.com  Tue Nov 16 20:21:46 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue Nov 16 20:20:43 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <e8bf7a5304111611013ad74af5@mail.gmail.com>
References: <419A4C1A.5060502@ceid.upatras.gr>
	<419A4C1A.5060502@ceid.upatras.gr>
Message-ID: <5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com>

At 02:01 PM 11/16/04 -0500, Jeremy Hylton wrote:
>On Tue, 16 Nov 2004 10:51:06 -0800, Stelios Xanthakis
><sxanth@ceid.upatras.gr> wrote:
> >
> > Hi.
> >
> > I posted a message on c.l.p a couple of days ago about a
> > python patch which adds a member __pycde__ to functions and
> > classes. This member is a string which holds the python code of
> > the function/class. (It works in interactively defined code
> > and exec'd definitions)
>
>Functions already have a reference to code objects.  How is this different?

I believe he means the Python source code; I personally would suggest a 
name like '__source__' instead of '__pycode__' as the attribute name.

Of course, I'd also almost prefer to have the AST than the original 
source...  ;)

From sxanth at ceid.upatras.gr  Tue Nov 16 20:21:41 2004
From: sxanth at ceid.upatras.gr (Stelios Xanthakis)
Date: Tue Nov 16 20:21:49 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <e8bf7a5304111611013ad74af5@mail.gmail.com>
References: <419A4C1A.5060502@ceid.upatras.gr>
	<e8bf7a5304111611013ad74af5@mail.gmail.com>
Message-ID: <Pine.GSO.4.61.0411162114260.1618@zenon.ceid.upatras.gr>



On Tue, 16 Nov 2004, Jeremy Hylton wrote:

> Functions already have a reference to code objects.  How is this different?

This is the python-code text.

>>> def foo ():
...     # comment -- instead of doc
...     print 'santa'
...
>>> print foo.__pycode__
def foo():
     # comment -- instead of doc
     print 'santa'


This is fun as I explained in the post at c.l.p
because with a function:

def save(FILE):
     for a in globals():
         if type(eval(a)) == FunctionType:
             FILE.write (eval(a).__pycode__)

We can save the entire state in a file
and continue from there next time.

Stelios

From sxanth at ceid.upatras.gr  Tue Nov 16 20:26:19 2004
From: sxanth at ceid.upatras.gr (Stelios Xanthakis)
Date: Tue Nov 16 20:26:25 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com>
References: <419A4C1A.5060502@ceid.upatras.gr>
	<419A4C1A.5060502@ceid.upatras.gr>
	<5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com>
Message-ID: <Pine.GSO.4.61.0411162124350.1642@zenon.ceid.upatras.gr>



On Tue, 16 Nov 2004, Phillip J. Eby wrote:

>
> I believe he means the Python source code; I personally would suggest a name 
> like '__source__' instead of '__pycode__' as the attribute name.
>

Yes. I'm terrible at choosing names for things :)

> Of course, I'd also almost prefer to have the AST than the original source...

If you have the python source you can get AST.
AST OTOH discards comments and other information.

Stelios

From jhylton at gmail.com  Tue Nov 16 20:35:46 2004
From: jhylton at gmail.com (Jeremy Hylton)
Date: Tue Nov 16 20:35:51 2004
Subject: [Python-Dev] Looking for authoritative documentation on packages,
	import & ihooks
In-Reply-To: <419A52F4.2020408@v.loewis.de>
References: <864d3709041115075629d15045@mail.gmail.com>
	<864d37090411160413756ff4c5@mail.gmail.com>
	<419A52F4.2020408@v.loewis.de>
Message-ID: <e8bf7a5304111611354643e892@mail.gmail.com>

On Tue, 16 Nov 2004 20:20:20 +0100, Martin v. L?wis <martin@v.loewis.de> wrote:
> Carlos Ribeiro wrote:
> > For this reason, I
> > would like to know if there is any document which may be considered
> > "authoritative" for the import system (besides ihooks.py and
> > import.c).
> 
> This question is OT for python-dev.
> 
> Python is open source, so what is wrong with reading import.c?

import.c isn't a document that purports to describe the interface that
import will support over time.  import.c has surely got good clues,
but I don't find it particularly easy to read.  It's a snapshot of
code that may or may not implement that interface correctly.

The question seems OT to me.

Jeremy
From pedronis at bluewin.ch  Tue Nov 16 20:40:35 2004
From: pedronis at bluewin.ch (Samuele Pedroni)
Date: Tue Nov 16 20:37:30 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com>
References: <419A4C1A.5060502@ceid.upatras.gr>	<419A4C1A.5060502@ceid.upatras.gr>
	<5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com>
Message-ID: <419A57B3.3070600@bluewin.ch>

Phillip J. Eby wrote:

> At 02:01 PM 11/16/04 -0500, Jeremy Hylton wrote:
> 
>> On Tue, 16 Nov 2004 10:51:06 -0800, Stelios Xanthakis
>> <sxanth@ceid.upatras.gr> wrote:
>> >
>> > Hi.
>> >
>> > I posted a message on c.l.p a couple of days ago about a
>> > python patch which adds a member __pycde__ to functions and
>> > classes. This member is a string which holds the python code of
>> > the function/class. (It works in interactively defined code
>> > and exec'd definitions)
>>
>> Functions already have a reference to code objects.  How is this 
>> different?
> 
> 
> I believe he means the Python source code; I personally would suggest a 
> name like '__source__' instead of '__pycode__' as the attribute name.
> 
> Of course, I'd also almost prefer to have the AST than the original 
> source...  ;)

a prerequiste for that is likely the ast-branch, because the old
"A"STs are not a reasonable format to be mandated for all python
implementations.


From fperez528 at yahoo.com  Tue Nov 16 20:38:52 2004
From: fperez528 at yahoo.com (Fernando Perez)
Date: Tue Nov 16 20:39:06 2004
Subject: [Python-Dev] Re: __pycode__ extension
References: <419A4C1A.5060502@ceid.upatras.gr>
	<20041116184548.GA5013@panix.com>
Message-ID: <cndl0c$cr8$1@sea.gmane.org>

Aahz wrote:

> On Tue, Nov 16, 2004, Stelios Xanthakis wrote:
>> 
>> I posted a message on c.l.p a couple of days ago about a
>> python patch which adds a member __pycde__ to functions and
>> classes. This member is a string which holds the python code of
>> the function/class. (It works in interactively defined code
>> and exec'd definitions)
>> 
>> Supposing that: I ported this to 2.4b2 and made it have zero overhead
>> when python runs in normal mode, is there any chance it would be
>> considered a candidate for inclusion in mainline python?
> 
> There is zero chance it'll go into 2.4.  There's a decent chance it'll
> go into 2.5, but if you get any pushback on the feature, be prepared to
> write up a full PEP.

Well, not necessarily pushback, but I'd like a clarification at least.  What
kind of memory overhead does this introduce?  If every function running around
is holding a full copy of all its source, is this overhead potentially
significant?  What happens with decorators, which modify functions but are not
explicit source-level transformations?

Since this is already fairly straightforward to implement via inspect, I'd like
to see a pretty strong justification for its real need before seeing it go in. 
It smells a bit of unnecessary bloat to me, but perhaps I'm missing something
obvious, so I'd be happy to be educated.

Regards,

f

From tim.peters at gmail.com  Tue Nov 16 20:45:24 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Tue Nov 16 20:45:27 2004
Subject: [Python-Dev] Looking for authoritative documentation on packages,
	import & ihooks
In-Reply-To: <419A52F4.2020408@v.loewis.de>
References: <864d3709041115075629d15045@mail.gmail.com>
	<864d37090411160413756ff4c5@mail.gmail.com>
	<419A52F4.2020408@v.loewis.de>
Message-ID: <1f7befae0411161145182e417c@mail.gmail.com>

[Carlos Ribeiro]
...
>> For this reason, I would like to know if there is any document which may
>> be considered "authoritative" for the import system (besides ihooks.py
>> and import.c).

[Martin v. L?wis]
> This question is OT for python-dev.
>
> Python is open source, so what is wrong with reading import.c?

I've tried.  I usually stop when the pool of blood from my eyeballs
rises above my ankles <0.9 wink>.

I'm not sure this is OT for python-dev.  There are many cases
involving circular and kinda-circular imports, with and without
packages, where the Python docs say nothing about what must, should,
or can happen.  So we still get bug reports like this recent one,
where nobody has a definitive answer:

    http://www.python.org/sf/992389

That case doesn't involve import hooks, just the meaning of package
imports and what you can and can't "get away with" in __init__.py
files.  In the presence of user-defined import hooks too, reaching the
clarity of mud in this case would be a noteworthy accomplishment.
From martin at v.loewis.de  Tue Nov 16 20:58:45 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue Nov 16 20:58:48 2004
Subject: [Python-Dev] Looking for authoritative documentation on packages, 
	import & ihooks
In-Reply-To: <1f7befae0411161145182e417c@mail.gmail.com>
References: <864d3709041115075629d15045@mail.gmail.com>	<864d37090411160413756ff4c5@mail.gmail.com>	<419A52F4.2020408@v.loewis.de>
	<1f7befae0411161145182e417c@mail.gmail.com>
Message-ID: <419A5BF5.4090201@v.loewis.de>

Tim Peters wrote:
> I'm not sure this is OT for python-dev.  There are many cases
> involving circular and kinda-circular imports, with and without
> packages, where the Python docs say nothing about what must, should,
> or can happen.

Not sure what "this" is, here. The import machinery certainly is
on-topic. The question "where do I find more documentation" is
off-topic. The question "I have this and that change to the
documentation, do people agree it is correct" would be on-topic.

Regards,
Martin
From tim.peters at gmail.com  Tue Nov 16 21:05:01 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Tue Nov 16 21:05:03 2004
Subject: [Python-Dev] Looking for authoritative documentation on packages,
	import & ihooks
In-Reply-To: <419A5BF5.4090201@v.loewis.de>
References: <864d3709041115075629d15045@mail.gmail.com>
	<864d37090411160413756ff4c5@mail.gmail.com>
	<419A52F4.2020408@v.loewis.de>
	<1f7befae0411161145182e417c@mail.gmail.com>
	<419A5BF5.4090201@v.loewis.de>
Message-ID: <1f7befae04111612057efb142@mail.gmail.com>

[Tim Peters]
>> I'm not sure this is OT for python-dev.  There are many cases
>> involving circular and kinda-circular imports, with and without
>> packages, where the Python docs say nothing about what must, should,
>> or can happen.

[Martin v. L?wis]
> Not sure what "this" is, here.

Ya, and I'm not sure what "OT" meant anywhere -- "on" and "off" both
start with "O" in English <wink>.

> The import machinery certainly is on-topic. The question "where do I find more
> documentation" is off-topic.

Could be, but the "sorry, there isn't more" answer seems on-topic
anyway:  developers need to be reminded about areas that remain
ill-defined.

> The question "I have this and that change to the documentation, do people
> agree it is correct" would be on-topic.

Yup, that too.
From pje at telecommunity.com  Tue Nov 16 21:19:34 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue Nov 16 21:18:32 2004
Subject: [Python-Dev] Looking for authoritative documentation on
	packages,  import & ihooks
In-Reply-To: <419A5BF5.4090201@v.loewis.de>
References: <1f7befae0411161145182e417c@mail.gmail.com>
	<864d3709041115075629d15045@mail.gmail.com>
	<864d37090411160413756ff4c5@mail.gmail.com>
	<419A52F4.2020408@v.loewis.de>
	<1f7befae0411161145182e417c@mail.gmail.com>
Message-ID: <5.1.1.6.0.20041116150836.02b32cc0@mail.telecommunity.com>

At 08:58 PM 11/16/04 +0100, Martin v. L?wis wrote:
>Tim Peters wrote:
>>I'm not sure this is OT for python-dev.  There are many cases
>>involving circular and kinda-circular imports, with and without
>>packages, where the Python docs say nothing about what must, should,
>>or can happen.
>
>Not sure what "this" is, here. The import machinery certainly is
>on-topic. The question "where do I find more documentation" is
>off-topic.

That was not what the OP asked; the clear intent of the request was what 
documentation should be considered *authoritative* on the matter, as well 
as what approaches were to be *preferred*.  IMO those sound like items that 
request the judgment and/or "blessing" of the Python development team, in 
the absence of any clearly authoritative documents or statements like 
"such-and-such is the new approach, so-and-so is the old".

Further, the OP:

1. Requested info on python-list first
2. Googled for, and read, the appropriate documentation
3. Raised issues regarding the documentation, and stated an intent to file 
bug reports with more specifics

ISTM these good faith efforts should give someone a little bit more benefit 
of the doubt in a borderline case...  not that this particular case seemed 
at all borderline to me.

From fumanchu at amor.org  Tue Nov 16 22:59:24 2004
From: fumanchu at amor.org (Robert Brewer)
Date: Tue Nov 16 23:00:58 2004
Subject: [Python-Dev] os.access versus os.exist
Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245214@exchange.hqamor.amorhq.net>

Martin v. L?wis wrote:
> Robert Brewer wrote:
> > Inside the app (running under LocalSystem on the same Win2k 
> machine):
> >    os.stat() raises OSError: [Errno 2] No such file or directory,
> >    os.path.exists() returns False, and
> >    os.access() returns False.
> 
> LocalSystem cannot access network shares. However, this is OT for
> python-dev.

Completely understood about LocalSystem. But IMO, os.stat should raise a more specific and correct error. As you pointed out, calling open() raises "IOError: [Errno 13] Permission denied"; it would be nice if stat did the same (in my example). The platform-specific API call (for example, Windows _stati64) doesn't provide this--it would require another test inside posix_do_stat to return a more appropriate error. But it would be a nice touch for us end-users.

In my opinion, "no such file" is the wrong error message and is therefore a bug. But I can accept others disagreeing, since os.stat() simply wraps Windows' _stat* calls (at least, in my example).


Robert Brewer
MIS
Amor Ministries
fumanchu@amor.org
From mwh at python.net  Tue Nov 16 23:46:02 2004
From: mwh at python.net (Michael Hudson)
Date: Tue Nov 16 23:46:06 2004
Subject: [Python-Dev] Looking for authoritative documentation on
	packages, import & ihooks
In-Reply-To: <1f7befae0411161145182e417c@mail.gmail.com> (Tim Peters's
	message of "Tue, 16 Nov 2004 14:45:24 -0500")
References: <864d3709041115075629d15045@mail.gmail.com>
	<864d37090411160413756ff4c5@mail.gmail.com>
	<419A52F4.2020408@v.loewis.de>
	<1f7befae0411161145182e417c@mail.gmail.com>
Message-ID: <2mu0rpza39.fsf@starship.python.net>

Tim Peters <tim.peters@gmail.com> writes:

> [Carlos Ribeiro]
> ...
>>> For this reason, I would like to know if there is any document which may
>>> be considered "authoritative" for the import system (besides ihooks.py
>>> and import.c).
>
> [Martin v. L?wis]
>> This question is OT for python-dev.
>>
>> Python is open source, so what is wrong with reading import.c?
>
> I've tried.  I usually stop when the pool of blood from my eyeballs
> rises above my ankles <0.9 wink>.

I'm glad to hear this isn't just me :-)

Cheers,
mwh
(officially scared of Python/import.c for some time now)

-- 
  <dash> if python is an orchestra, overloaded operators are
        "miscellaneous percussion"              -- from Twisted.Quotes
From kbk at shore.net  Wed Nov 17 06:28:08 2004
From: kbk at shore.net (Kurt B. Kaiser)
Date: Wed Nov 17 06:28:14 2004
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200411170528.iAH5S8E3026597@h006008a7bda6.ne.client2.attbi.com>

Patch / Bug Summary
___________________

Patches :  246 open ( +2) /  2699 closed ( +1) /  2945 total ( +3)
Bugs    :  770 open ( -1) /  4614 closed (+19) /  5384 total (+18)
RFE     :  156 open ( -1) /   135 closed ( +1) /   291 total ( +0)

New / Reopened Patches
______________________

httplib: allowing stream-type body part in requests  (2004-11-12)
       http://python.org/sf/1065257  opened by  Alessandro Forghieri

Fix pydoc crashing on unicode strings  (2004-11-14)
       http://python.org/sf/1065986  opened by  Cherniavsky Beni

Non-ascii in non-unicode __credits__ in Lib/pydoc.py  (2004-08-15)
       http://python.org/sf/1009389  reopened by  loewis

fix for 1067728: Better handling of float arguments to seek  (2004-11-17)
       http://python.org/sf/1067760  opened by  Robert Church

Patches Closed
______________

Fix for os.path.expanduser for Win2000  (2002-11-26)
       http://python.org/sf/643943  closed by  jlgijsbers

Fix various x86_64 build issues  (2004-10-20)
       http://python.org/sf/1050475  closed by  loewis

New / Reopened Bugs
___________________

IDLE doesn't start again when saving empty key set  (2004-11-11)
CLOSED http://python.org/sf/1064535  opened by  Hernan Martinez Foffani

Parser module asts don't match grammar  (2004-11-11)
CLOSED http://python.org/sf/1064873  opened by  logistix

calendar day/month name lookup too slow  (2004-11-12)
CLOSED http://python.org/sf/1065388  opened by  Guido van Rossum

sre_parse group limit check missing with 'python -O'  (2004-11-12)
CLOSED http://python.org/sf/1065427  opened by  Sam Rushing

pprint.pformat requires __repr__  (2004-11-12)
CLOSED http://python.org/sf/1065456  opened by  Dave Kuhlman

Typo about PyErr_WriteUnraisable()  (2004-11-14)
       http://python.org/sf/1066036  opened by  Raise L. Sail

datetime.replace method description error  (2004-11-14)
CLOSED http://python.org/sf/1066438  opened by  Brian Underwood

special methods become static  (2004-11-14)
CLOSED http://python.org/sf/1066490  opened by  Kevin Quick

SegFault in interactive mode when cursor is over a non-ASCII  (2004-11-15)
       http://python.org/sf/1066545  opened by  theIgel

test_pwd fails on 64bit system (Opteron)  (2004-11-15)
       http://python.org/sf/1066546  opened by  Miki Tebeka

"Limitations" section of profiler docs is incorrect  (2004-11-15)
       http://python.org/sf/1066607  opened by  Nick Coghlan

Obsolete info in Tutorial 9.1  (2004-11-15)
       http://python.org/sf/1067018  opened by  Kent Johnson

Small typo  (2004-11-15)
       http://python.org/sf/1067023  opened by  Scott Miller

win32con missing codes VK_VOLUME_MUTE, VK_BROWSER_BACK, ...  (2004-11-15)
       http://python.org/sf/1067153  opened by  Jeff Connelly aka shellreef

Incorrect length of unicode strings using .encode('utf-8')  (2004-11-16)
CLOSED http://python.org/sf/1067294  opened by  Ed Schofield

urllib fails with multiple ftps  (2004-11-16)
       http://python.org/sf/1067702  opened by  Russell Owen

Overflow error seek()ing with float values > (2 ** 31) - 1  (2004-11-17)
       http://python.org/sf/1067728  opened by  Robert Church

wininst --install-script leaves residual files on C:  (2004-11-16)
       http://python.org/sf/1067732  opened by  J Livingston

Overflow error seek()ing with float values > (2 ** 31) - 1  (2004-11-17)
CLOSED http://python.org/sf/1067756  opened by  Robert Church

Bugs Closed
___________

IDLE aborts on any error when started with "-n"  (2004-11-10)
       http://python.org/sf/1063840  closed by  kbk

Konqueror can't render docs because of malformed HTML  (2004-05-28)
       http://python.org/sf/962442  closed by  fdrake

Manual typesets bit-shift operators as guillemet  (2004-11-06)
       http://python.org/sf/1061770  closed by  fdrake

Fix Decimal's .min() and .max() docs regarding NaNs  (2004-11-04)
       http://python.org/sf/1060644  closed by  facundobatista

IDLE doesn't start again when saving empty key set  (2004-11-11)
       http://python.org/sf/1064535  closed by  kbk

Legacy requirements: Incorrect comments?  (2004-11-02)
       http://python.org/sf/1058937  closed by  dairiki

Parser module asts don't match grammar  (2004-11-11)
       http://python.org/sf/1064873  closed by  bcannon

optparse docs mildly mangled  (2004-11-10)
       http://python.org/sf/1063757  closed by  gward

test_subprocess 2.4b2 fails on tru64  (2004-11-10)
       http://python.org/sf/1063571  closed by  astrand

difflib HtmlDiff() extra space on inserted 1 character lines  (2004-10-26)
       http://python.org/sf/1054821  closed by  tim_one

calendar day/month name lookup too slow  (2004-11-12)
       http://python.org/sf/1065388  closed by  tim_one

sre_parse group limit check missing with 'python -O'  (2004-11-12)
       http://python.org/sf/1065427  closed by  effbot

pprint.pformat requires __repr__  (2004-11-12)
       http://python.org/sf/1065456  closed by  doerwalter

readline()/readlines()  (2003-06-27)
       http://python.org/sf/761787  closed by  fullsail

datetime.replace method description error  (2004-11-14)
       http://python.org/sf/1066438  closed by  tim_one

special methods become static  (2004-11-15)
       http://python.org/sf/1066490  closed by  mwh

pickle won't dump instances after reload  (2002-10-25)
       http://python.org/sf/628925  closed by  facundobatista

Incorrect length of unicode strings using .encode('utf-8')  (2004-11-16)
       http://python.org/sf/1067294  closed by  lemburg

IDLE DOES NOT START ON WinXP Pro  (2004-08-27)
       http://python.org/sf/1017978  closed by  kbk

Overflow error seek()ing with float values > (2 ** 31) - 1  (2004-11-16)
       http://python.org/sf/1067756  closed by  nnorwitz

RFE Closed
__________

os.run function for convinience and robustness  (2004-10-05)
       http://python.org/sf/1040267  closed by  jlgijsbers

From sxanth at ceid.upatras.gr  Wed Nov 17 11:00:58 2004
From: sxanth at ceid.upatras.gr (Stelios Xanthakis)
Date: Wed Nov 17 11:01:01 2004
Subject: [Python-Dev] Re: __pycode__ extension
Message-ID: <Pine.GSO.4.61.0411171133160.27454@diogenis.ceid.upatras.gr>


Fernando Perez:
> Well, not necessarily pushback, but I'd like a clarification at least. 
> What kind of memory overhead does this introduce?  If every function 
> running  around is holding a full copy of all its source, is this 
> overhead potentially significant?

The overhead I'm worried about is the performance cost in
the parser. The memory doesn't worry me because computer's
memory gets more every year while python source remains compact.
It's the debate of maintanability vs optimization.

>  What happens with decorators, which modify functions but are 
> not explicit source-level transformations?

I don't know about that. I guess decorator declarations are
included in the source of a function.

> Since this is already fairly straightforward to implement via inspect, 
> I'd  like to see a pretty strong justification for its real need before 
> seeing it go in.

'inspect' can't keep the source of functions defined with 'exec'
or interactively (or can it?). Moreover, you can edit a function
at runtime and see it with inspect.

As I stated in the OP at c.l.p, this all can be done in python
as it is, by using some framework (Zope, whatever). In fact I started
from such a framework but then it occured to me that it's
the job of the parser to do this.
In other words, with __pycode__ python becomes a framework.

But I'm not here to sell amway products:)
When/if pythoneers decide they like it, let me know.


Regards,

Stelios

From ncoghlan at iinet.net.au  Wed Nov 17 11:02:43 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Wed Nov 17 11:02:48 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <20041116213506.GB22178@panix.com>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
	<4195F883.8030309@iinet.net.au>
	<20041114131708.GA9442@vicky.ecs.soton.ac.uk>
	<4197B42A.8020401@v.loewis.de> <4198B227.8080309@iinet.net.au>
	<419920DF.9070002@v.loewis.de> <4199590B.1040908@iinet.net.au>
	<20041116213506.GB22178@panix.com>
Message-ID: <419B21C3.3020700@iinet.net.au>

Aahz wrote:
> Please fix your Thunderbird to stop adding spurious ^M characters:

Sorry about that. It looks like either Google's SMTP server is doing something 
odd or Thunderbird's TLS implementation has issues - I switched back to my ISP's 
SMTP server and the problem went away.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Wed Nov 17 11:19:08 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Wed Nov 17 11:19:12 2004
Subject: [Python-Dev] Looking for authoritative documentation on	packages, 
	import & ihooks
In-Reply-To: <2mu0rpza39.fsf@starship.python.net>
References: <864d3709041115075629d15045@mail.gmail.com>	<864d37090411160413756ff4c5@mail.gmail.com>	<419A52F4.2020408@v.loewis.de>	<1f7befae0411161145182e417c@mail.gmail.com>
	<2mu0rpza39.fsf@starship.python.net>
Message-ID: <419B259C.3030007@iinet.net.au>

Michael Hudson wrote:
>>>Python is open source, so what is wrong with reading import.c?
>>
>>I've tried.  I usually stop when the pool of blood from my eyeballs
>>rises above my ankles <0.9 wink>.
> 
> 
> I'm glad to hear this isn't just me :-)

Far from it. . . I was looking around in there when I was implementing the '-m' 
switch. I *think* I kinda sorta figured out how normal imports usually work 
(well enough to determine there was not yet a C API equivalent to 
imp.find_module, anyway). Figuring out what sys.metapath does, on the other 
hand. . .

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Wed Nov 17 11:48:22 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Wed Nov 17 11:48:28 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <419A4C1A.5060502@ceid.upatras.gr>
References: <419A4C1A.5060502@ceid.upatras.gr>
Message-ID: <419B2C76.1000709@iinet.net.au>

Stelios Xanthakis wrote:
> 
> Hi.
> 
> 
> I posted a message on c.l.p a couple of days ago about a
> python patch which adds a member __pycde__ to functions and
> classes. This member is a string which holds the python code of
> the function/class. (It works in interactively defined code
> and exec'd definitions)

I think this is a potentially interesting idea, but one worth taking through the 
PEP process for 2.5. This would have the benefit of clearly identifying the 
gains provided by the feature, and also identifying potential downsides to be 
addressed.

The number one downside I can see (beyond the extra memory consumed) is the 
bloating effect on pyc files - for this to work, the compiled bytecode is going 
to have to contain the original source code as strings. Instead of pyc files 
being smaller than the originals, they will suddenly be larger (roughly the 
current size of the py file added to the current size of the pyc file). This may 
also impact the time required to parse the pyc files at import time - 
potentially slowing the startup time of the Python interpreter itself, as well 
as Python applications (a trend in exactly the *wrong* direction).

Also, Python does get used in embedded environments, so memory and filesystem 
space considerations do need to be addressed.

One option would be an addition to coflags that is off by default - then the 
feature could be selectively enabled by an application without impacting normal 
operation.

Cheers,
Nick.
-0, but willing to be persuaded otherwise (hence, PEP)

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From sxanth at ceid.upatras.gr  Wed Nov 17 13:24:18 2004
From: sxanth at ceid.upatras.gr (Stelios Xanthakis)
Date: Wed Nov 17 13:24:22 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <419B2C76.1000709@iinet.net.au>
References: <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au>
Message-ID: <Pine.GSO.4.61.0411171333280.27928@diogenis.ceid.upatras.gr>


Hi Nick.

> One option would be an addition to coflags that is off by default - then 
> the feature could be selectively enabled by an application without 
> impacting normal operation.

100% agree. I used the '-P' command line option to activate this.
It is very important to understand that python will run in
an entirely different mode. People who are not interested
using this should have zero overhead.
Only those who are willing to pay the price will get __pycode__
-- and it's not that big.

So the real question is: will there be people who will use this?



Trying to answer that..
Insight: If this gets in, pythoneers will be tempted to
create a file MYSYSTEM.py with one function:

       #!/usr/bin/python -P
       def save_world():
           F = open ('MYSYSTEM.py')
           F.write ('#!/usr/bin/python -P\n')
           for i in globals():
               if hasattr (eval(a), '__pycode__'):
                   F.write (eval(a).__pycode__)
           F.write ('main()')
       def main():
          import code
          code.interact ('', raw_input, globals())

And that will give birth to their own personal incremental
python shell/library.


> The number one downside I can see (beyond the extra memory consumed) is 
> the bloating effect on pyc files - for this to work, the compiled 
> bytecode is going to have to contain the original source code as 
> strings.

The __pycode__ attribute is not marshaled.
The logic behind this is that if we edit a function/class
and save it in a .pyc file, the next time the .py module
is executed it will overwrite the .pyc file and our changes
are lost.
So for 'import'ed code, __pycode__=None

Hm?

> I think this is a potentially interesting idea, but one worth taking 
> through the PEP process for 2.5. This would have the benefit of clearly 
> identifying the gains provided by the feature, and also identifying 
> potential downsides to be addressed.

Definitelly. But before starting a PEP I'd like to gather
feedback and ideas, have a good patch vs 2.4, and, well,
understand python source a little better.


Stelios



From anthony at interlink.com.au  Wed Nov 17 15:59:36 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed Nov 17 15:59:47 2004
Subject: [Python-Dev] TRUNK FROZEN for 2.4rc1 from around 1200 UTC on Thurs
	18
Message-ID: <200411171459.iAHExag25031@burswood.off.ekorp.com>

Please hold off on any checkins after 1200 UTC on Thursday 18th 
(about 22 hours after this message was sent). We're cutting 2.4rc1
then. Assuming all goes well, we'll be looking at a 2.4 final for
Nov 30. 

Could people please be _very_ conservative with checkins between
now and 2.4 final? A brown-paper-bag 2.4.1 would suck :-(

ta
Anthony
From pje at telecommunity.com  Wed Nov 17 17:01:56 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed Nov 17 17:00:58 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <Pine.GSO.4.61.0411171333280.27928@diogenis.ceid.upatras.gr
 >
References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<419B2C76.1000709@iinet.net.au>
Message-ID: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>

At 02:24 PM 11/17/04 +0200, Stelios Xanthakis wrote:
>       #!/usr/bin/python -P
>       def save_world():
>           F = open ('MYSYSTEM.py')
>           F.write ('#!/usr/bin/python -P\n')
>           for i in globals():
>               if hasattr (eval(a), '__pycode__'):
>                   F.write (eval(a).__pycode__)

There are bugs in this code.  First, the wrong variable is used for eval(), 
and second, it shouldn't use eval.  If there's a global named 'F', for 
example, the code above will not save it, because it will see the local 'F' 
instead.  Loop over globals().values(), and do not use eval.


>The __pycode__ attribute is not marshaled.
>The logic behind this is that if we edit a function/class
>and save it in a .pyc file, the next time the .py module
>is executed it will overwrite the .pyc file and our changes
>are lost.
>So for 'import'ed code, __pycode__=None
>
>Hm?

What good is that?  Likewise, I don't see the point of having this only 
enabled when a command-line option is given.

If the idea is just to allow saving code from interactive mode, why not 
just modify the interactive mode to do this?

From FBatista at uniFON.com.ar  Wed Nov 17 18:52:32 2004
From: FBatista at uniFON.com.ar (Batista, Facundo)
Date: Wed Nov 17 18:54:47 2004
Subject: [Python-Dev] Py 2.4 b2 announcement
Message-ID: <A128D751272CD411BC9200508BC2194D053C7C09@escpl.tcp.com.ar>


Is some weird cache issue from my side, or Py2.4b2 is not announced at
http://www.python.org/?

Regards,

Facundo Batista
Desarrollo de Red
fbatista@unifon.com.ar
(54 11) 5130-4643
Cel: 15 5097 5024


From amk at amk.ca  Wed Nov 17 19:00:29 2004
From: amk at amk.ca (A.M. Kuchling)
Date: Wed Nov 17 19:04:20 2004
Subject: [Python-Dev] Py 2.4 b2 announcement
In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7C09@escpl.tcp.com.ar>
References: <A128D751272CD411BC9200508BC2194D053C7C09@escpl.tcp.com.ar>
Message-ID: <20041117180029.GA4501@rogue.amk.ca>

On Wed, Nov 17, 2004 at 02:52:32PM -0300, Batista, Facundo wrote:
> Is some weird cache issue from my side, or Py2.4b2 is not announced at
> http://www.python.org/?

You're correct; looking at the CVS log for index.ht, there's an entry
for 2.4b1 but not b2.  I've updated the page.

--amk

From pedronis at bluewin.ch  Wed Nov 17 20:37:18 2004
From: pedronis at bluewin.ch (Samuele Pedroni)
Date: Wed Nov 17 20:34:14 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
References: <419B2C76.1000709@iinet.net.au>
	<419A4C1A.5060502@ceid.upatras.gr>	<419B2C76.1000709@iinet.net.au>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
Message-ID: <419BA86E.70309@bluewin.ch>

Phillip J. Eby wrote:

> At 02:24 PM 11/17/04 +0200, Stelios Xanthakis wrote:
> 
>>       #!/usr/bin/python -P
>>       def save_world():
>>           F = open ('MYSYSTEM.py')
>>           F.write ('#!/usr/bin/python -P\n')
>>           for i in globals():
>>               if hasattr (eval(a), '__pycode__'):
>>                   F.write (eval(a).__pycode__)
> 
> 
> There are bugs in this code.  First, the wrong variable is used for 
> eval(), and second, it shouldn't use eval.  If there's a global named 
> 'F', for example, the code above will not save it, because it will see 
> the local 'F' instead.  Loop over globals().values(), and do not use eval.
> 
> 
>> The __pycode__ attribute is not marshaled.
>> The logic behind this is that if we edit a function/class
>> and save it in a .pyc file, the next time the .py module
>> is executed it will overwrite the .pyc file and our changes
>> are lost.
>> So for 'import'ed code, __pycode__=None
>>
>> Hm?
> 
> 
> What good is that?


I agree with the spirit of the observation, but I think that aspect is 
immaterial. Personally I think some form of this feature is valuable as 
long as it means getting inspect.getsource (and companions) or some 
variation working no matter what. So as long as what goes in the 
attribute covers the cases not working today, that should be fine.

>  Likewise, I don't see the point of having this only 
> enabled when a command-line option is given.

well, I think some people may bring memory consumption considerations to 
the table depending of what the final implementation is. For embedding 
scenarios I suspect that a build time define could be ok.

But I think we should rember that some people today rely on the fact 
that shipping only pycs files means not shipping directly the source.


From mal at egenix.com  Wed Nov 17 20:39:22 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed Nov 17 20:39:26 2004
Subject: [Python-Dev] sys.exit(47) ?
Message-ID: <419BA8EA.7030002@egenix.com>

Hi there,

something in the regression suite is generating a temporary
file with content:

#!/bin/sh
exec /path/to/python -c 'import sys; sys.exit(47)'

and not cleaning up after itself.

This started around Oct. 13. Looking at the regression suite
test_subprocess.py seems to be the cause.

BTW, wouldn't sys.exit(42) be more appropriate :-?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 17 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From pedronis at bluewin.ch  Wed Nov 17 20:47:42 2004
From: pedronis at bluewin.ch (Samuele Pedroni)
Date: Wed Nov 17 20:44:59 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <419BA86E.70309@bluewin.ch>
References: <419B2C76.1000709@iinet.net.au>	<419A4C1A.5060502@ceid.upatras.gr>	<419B2C76.1000709@iinet.net.au>	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<419BA86E.70309@bluewin.ch>
Message-ID: <419BAADE.4090800@bluewin.ch>

Samuele Pedroni wrote:
>> What good is that?
> 
> 
> 
> I agree with the spirit of the observation, but I think that aspect is 
> immaterial. Personally I think some form of this feature is valuable as 
> long as it means getting inspect.getsource (and companions) or some 
> variation working no matter what. So as long as what goes in the 
> attribute covers the cases not working today, that should be fine.
> 

of course depending on the scenarios one has in mind, working may mean 
produce always an output but with the same limitations from code in py 
files, or produce a correct output even in the case of file editings.
From astrand at lysator.liu.se  Wed Nov 17 20:53:55 2004
From: astrand at lysator.liu.se (Peter Astrand)
Date: Wed Nov 17 20:54:06 2004
Subject: [Python-Dev] Re: subprocess and EINTR errnos
In-Reply-To: <20041111034803.GA16175@isc.upenn.edu>
References: <20041111034803.GA16175@isc.upenn.edu>
Message-ID: <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se>

On Wed, 10 Nov 2004, John P Speno wrote:

Hi, sorry for the delayed response.

> While using subprocess (aka popen5), I came across one potential gotcha. I've had
> exceptions ending like this:
>
>   File "test.py", line 5, in test
>    cmd = popen5.Popen(args, stdout=PIPE)
>   File "popen5.py", line 577, in __init__
>     data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
> OSError: [Errno 4] Interrupted system call
>
> (on Solaris 9)
>
> Would it make sense for subprocess to use a more robust read() function
> which can handle these cases, i.e. when the parent's read on the pipe
> to the child's stderr is interrupted by a system call, and returns EINTR?
> I imagine it could catch EINTR and EAGAIN and retry the failed read().

I assume you are using signals in your application? The os.read above is
not the only system call that can fail with EINTR. subprocess.py is full
of other system calls that can fail, and I suspect that many other Python
modules are as well.

I've made a patch (attached) to subprocess.py (and test_subprocess.py)
that should guard against EINTR, but I haven't committed it yet. It's
quite large.

Are Python modules supposed to handle EINTR? Why not let the C code handle
this? Or, perhaps the signal module should provide a sigaction function,
so that users can use SA_RESTART.


Index: subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/subprocess.py,v
retrieving revision 1.8
diff -u -r1.8 subprocess.py
--- subprocess.py	7 Nov 2004 14:30:34 -0000	1.8
+++ subprocess.py	17 Nov 2004 19:42:30 -0000
@@ -888,6 +888,50 @@
                     pass


+        def _read_no_intr(self, fd, buffersize):
+            """Like os.read, but retries on EINTR"""
+            while True:
+                try:
+                    return os.read(fd, buffersize)
+                except OSError, e:
+                    if e.errno == errno.EINTR:
+                        continue
+                    else:
+                        raise
+
+
+        def _read_all(self, fd, buffersize):
+            """Like os.read, but retries on EINTR, and reads until EOF"""
+            all = ""
+            while True:
+                data = self._read_no_intr(fd, buffersize)
+                all += data
+                if data == "":
+                    return all
+
+
+        def _write_no_intr(self, fd, s):
+            """Like os.write, but retries on EINTR"""
+            while True:
+                try:
+                    return os.write(fd, s)
+                except OSError, e:
+                    if e.errno == errno.EINTR:
+                        continue
+                    else:
+                        raise
+
+        def _waitpid_no_intr(self, pid, options):
+            """Like os.waitpid, but retries on EINTR"""
+            while True:
+                try:
+                    return os.waitpid(pid, options)
+                except OSError, e:
+                    if e.errno == errno.EINTR:
+                        continue
+                    else:
+                        raise
+
         def _execute_child(self, args, executable, preexec_fn, close_fds,
                            cwd, env, universal_newlines,
                            startupinfo, creationflags, shell,
@@ -963,7 +1007,7 @@
                                                            exc_value,
                                                            tb)
                     exc_value.child_traceback = ''.join(exc_lines)
-                    os.write(errpipe_write, pickle.dumps(exc_value))
+                    self._write_no_intr(errpipe_write, pickle.dumps(exc_value))

                 # This exitcode won't be reported to applications, so it
                 # really doesn't matter what we return.
@@ -979,7 +1023,7 @@
                 os.close(errwrite)

             # Wait for exec to fail or succeed; possibly raising exception
-            data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
+            data = self._read_all(errpipe_read, 1048576) # Exceptions limited to 1 MB
             os.close(errpipe_read)
             if data != "":
                 child_exception = pickle.loads(data)
@@ -1003,7 +1047,7 @@
             attribute."""
             if self.returncode == None:
                 try:
-                    pid, sts = os.waitpid(self.pid, os.WNOHANG)
+                    pid, sts = self._waitpid_no_intr(self.pid, os.WNOHANG)
                     if pid == self.pid:
                         self._handle_exitstatus(sts)
                 except os.error:
@@ -1015,7 +1059,7 @@
             """Wait for child process to terminate.  Returns returncode
             attribute."""
             if self.returncode == None:
-                pid, sts = os.waitpid(self.pid, 0)
+                pid, sts = self._waitpid_no_intr(self.pid, 0)
                 self._handle_exitstatus(sts)
             return self.returncode

@@ -1049,27 +1093,33 @@
                 stderr = []

             while read_set or write_set:
-                rlist, wlist, xlist = select.select(read_set, write_set, [])
+                try:
+                    rlist, wlist, xlist = select.select(read_set, write_set, [])
+                except select.error, e:
+                    if e[0] == errno.EINTR:
+                        continue
+                    else:
+                        raise

                 if self.stdin in wlist:
                     # When select has indicated that the file is writable,
                     # we can write up to PIPE_BUF bytes without risk
                     # blocking.  POSIX defines PIPE_BUF >= 512
-                    bytes_written = os.write(self.stdin.fileno(), input[:512])
+                    bytes_written = self._write_no_intr(self.stdin.fileno(), input[:512])
                     input = input[bytes_written:]
                     if not input:
                         self.stdin.close()
                         write_set.remove(self.stdin)

                 if self.stdout in rlist:
-                    data = os.read(self.stdout.fileno(), 1024)
+                    data = self._read_no_intr(self.stdout.fileno(), 1024)
                     if data == "":
                         self.stdout.close()
                         read_set.remove(self.stdout)
                     stdout.append(data)

                 if self.stderr in rlist:
-                    data = os.read(self.stderr.fileno(), 1024)
+                    data = self._read_no_intr(self.stderr.fileno(), 1024)
                     if data == "":
                         self.stderr.close()
                         read_set.remove(self.stderr)
Index: test/test_subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v
retrieving revision 1.14
diff -u -r1.14 test_subprocess.py
--- test/test_subprocess.py	12 Nov 2004 15:51:48 -0000	1.14
+++ test/test_subprocess.py	17 Nov 2004 19:42:30 -0000
@@ -7,6 +7,7 @@
 import tempfile
 import time
 import re
+import errno

 mswindows = (sys.platform == "win32")

@@ -35,6 +36,16 @@
             fname = tempfile.mktemp()
             return os.open(fname, os.O_RDWR|os.O_CREAT), fname

+    def read_no_intr(self, obj):
+        while True:
+            try:
+                return obj.read()
+            except IOError, e:
+                if e.errno == errno.EINTR:
+                    continue
+                else:
+                    raise
+
     #
     # Generic tests
     #
@@ -123,7 +134,7 @@
         p = subprocess.Popen([sys.executable, "-c",
                           'import sys; sys.stdout.write("orange")'],
                          stdout=subprocess.PIPE)
-        self.assertEqual(p.stdout.read(), "orange")
+        self.assertEqual(self.read_no_intr(p.stdout), "orange")

     def test_stdout_filedes(self):
         # stdout is set to open file descriptor
@@ -151,7 +162,7 @@
         p = subprocess.Popen([sys.executable, "-c",
                           'import sys; sys.stderr.write("strawberry")'],
                          stderr=subprocess.PIPE)
-        self.assertEqual(remove_stderr_debug_decorations(p.stderr.read()),
+        self.assertEqual(remove_stderr_debug_decorations(self.read_no_intr(p.stderr)),
                          "strawberry")

     def test_stderr_filedes(self):
@@ -186,7 +197,7 @@
                           'sys.stderr.write("orange")'],
                          stdout=subprocess.PIPE,
                          stderr=subprocess.STDOUT)
-        output = p.stdout.read()
+        output = self.read_no_intr(p.stdout)
         stripped = remove_stderr_debug_decorations(output)
         self.assertEqual(stripped, "appleorange")

@@ -220,7 +231,7 @@
                          stdout=subprocess.PIPE,
                          cwd=tmpdir)
         normcase = os.path.normcase
-        self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir))
+        self.assertEqual(normcase(self.read_no_intr(p.stdout)), normcase(tmpdir))

     def test_env(self):
         newenv = os.environ.copy()
@@ -230,7 +241,7 @@
                           'sys.stdout.write(os.getenv("FRUIT"))'],
                          stdout=subprocess.PIPE,
                          env=newenv)
-        self.assertEqual(p.stdout.read(), "orange")
+        self.assertEqual(self.read_no_intr(p.stdout), "orange")

     def test_communicate(self):
         p = subprocess.Popen([sys.executable, "-c",
@@ -305,7 +316,8 @@
                           'sys.stdout.write("\\nline6");'],
                          stdout=subprocess.PIPE,
                          universal_newlines=1)
-        stdout = p.stdout.read()
+
+        stdout = self.read_no_intr(p.stdout)
         if hasattr(open, 'newlines'):
             # Interpreter with universal newline support
             self.assertEqual(stdout,
@@ -343,7 +355,7 @@

     def test_no_leaking(self):
         # Make sure we leak no resources
-        max_handles = 1026 # too much for most UNIX systems
+        max_handles = 10 # too much for most UNIX systems
         if mswindows:
             max_handles = 65 # a full test is too slow on Windows
         for i in range(max_handles):
@@ -424,7 +436,7 @@
                               'sys.stdout.write(os.getenv("FRUIT"))'],
                              stdout=subprocess.PIPE,
                              preexec_fn=lambda: os.putenv("FRUIT", "apple"))
-            self.assertEqual(p.stdout.read(), "apple")
+            self.assertEqual(self.read_no_intr(p.stdout), "apple")

         def test_args_string(self):
             # args is a string
@@ -457,7 +469,7 @@
             p = subprocess.Popen(["echo $FRUIT"], shell=1,
                                  stdout=subprocess.PIPE,
                                  env=newenv)
-            self.assertEqual(p.stdout.read().strip(), "apple")
+            self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")

         def test_shell_string(self):
             # Run command through the shell (string)
@@ -466,7 +478,7 @@
             p = subprocess.Popen("echo $FRUIT", shell=1,
                                  stdout=subprocess.PIPE,
                                  env=newenv)
-            self.assertEqual(p.stdout.read().strip(), "apple")
+            self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")

         def test_call_string(self):
             # call() function with string argument on UNIX
@@ -525,7 +537,7 @@
             p = subprocess.Popen(["set"], shell=1,
                                  stdout=subprocess.PIPE,
                                  env=newenv)
-            self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+            self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)

         def test_shell_string(self):
             # Run command through the shell (string)
@@ -534,7 +546,7 @@
             p = subprocess.Popen("set", shell=1,
                                  stdout=subprocess.PIPE,
                                  env=newenv)
-            self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+            self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)

         def test_call_string(self):
             # call() function with string argument on Windows



/Peter ?strand <astrand@lysator.liu.se>

From fperez528 at yahoo.com  Wed Nov 17 20:56:56 2004
From: fperez528 at yahoo.com (Fernando Perez)
Date: Wed Nov 17 20:57:01 2004
Subject: [Python-Dev] Re: __pycode__ extension
References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
Message-ID: <cngae8$s80$1@sea.gmane.org>

Phillip J. Eby wrote:

> If the idea is just to allow saving code from interactive mode, why not
> just modify the interactive mode to do this?

IPython already has most of this:

In [3]: os.path.walk??
Type:           function
Base Class:     <type 'function'>
String Form:    <function walk at 0x550529cc>
Namespace:      Interactive
File:           /usr/src/build/394694-i386/install/usr/lib/python2.3/posixpath.py
Definition:     os.path.walk(top, func, arg)
Source:
def walk(top, func, arg):
    """Directory tree walk with callback function.

    For each directory in the directory tree rooted at top (including top
    itself, but excluding '.' and '..'), call func(arg, dirname, fnames).
    dirname is the name of the directory, and fnames a list of the names of
    the files and subdirectories in dirname (excluding '.' and '..').  func
    may modify the fnames list in-place (e.g. via del or slice assignment),
    and walk will only recurse into the subdirectories whose names remain in
    fnames; this can be used to implement a filter, or to impose a specific
    order of visiting.  No semantics are defined for, or required of, arg,
    beyond that arg is always passed to func.  It can be used, e.g., to pass
    a filename pattern, or a mutable object designed to accumulate
    statistics.  Passing None for arg is common."""

    try:
        names = os.listdir(top)
    except os.error:
        return
    func(arg, top, names)
    for name in names:
        name = join(top, name)
        try:
            st = os.lstat(name)
        except os.error:
            continue
        if stat.S_ISDIR(st.st_mode):
            walk(name, func, arg)

Since it's already available (anyone can copy the code if they want, it's BSD),
I am very weary of the bloat this would introduce if it were a default
feature.  So I'd like to see a good argument about the footprint increase of
this proposal.

Best,

f

From bac at OCF.Berkeley.EDU  Wed Nov 17 20:57:36 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Wed Nov 17 20:57:50 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <41995FBF.3080207@iinet.net.au>
References: <200411152028.iAFKSJgq040962@boa.lbl.gov>
	<41995FBF.3080207@iinet.net.au>
Message-ID: <419BAD30.1010200@ocf.berkeley.edu>

Nick Coghlan wrote:
> Ralf W. Grosse-Kunstleve wrote:
> 
>>As we work on new algorithms we can easily monitor the time/tick
>>without a runtime penalty. As long as the ratio stays reasonably high
>>(e.g. because we use numarray-like array operations already coded in
>>C++) we don't have to do anything. If the ratio goes down significantly
>>we can run the profiler to do detailed analysis of the new code.
> 
> 
> That seems like a reasonable approach to me - one could easily make use 
> of the ticker information in a test framework. So I'll withdraw my 
> objection and give the new function a +0.
> 

I have been mulling over this proposal and I think I am finally settling on +0 
as long as it can be shown that it does not hurt performance at all (yes, it 
shouldn't but this is the eval loop we are talking about and that thing is a 
touchy beast).

This feels like a move towards providing the same level of infor that Java's 
JMX provides (which I am not saying is good or bad; just been writing a Python 
vs. Java research paper and so I have partial Java-on-the-brain).

I think the trick is going to be making it clear in the docs that this is meant 
to give people a rough idea of where there is still a lot of Python code being 
used compared to C and not and should not be used to measure performance.

-Brett
From astrand at lysator.liu.se  Wed Nov 17 21:13:26 2004
From: astrand at lysator.liu.se (Peter Astrand)
Date: Wed Nov 17 21:14:11 2004
Subject: [Python-Dev] sys.exit(47) ?
In-Reply-To: <419BA8EA.7030002@egenix.com>
References: <419BA8EA.7030002@egenix.com>
Message-ID: <Pine.GSO.4.51L2.0411172107120.2244@koeberg.lysator.liu.se>

On Wed, 17 Nov 2004, M.-A. Lemburg wrote:

> something in the regression suite is generating a temporary
> file with content:
>
> #!/bin/sh
> exec /path/to/python -c 'import sys; sys.exit(47)'
>
> and not cleaning up after itself.
>
> This started around Oct. 13. Looking at the regression suite
> test_subprocess.py seems to be the cause.

Thanks for pointing this out. Now fixed, in revision 1.15 of
test_subprocess.py.


> BTW, wouldn't sys.exit(42) be more appropriate :-?

42 is not as random :) See http://en.wikipedia.org/wiki/47_%28number%29

/Peter ?strand <astrand@lysator.liu.se>

From foom at fuhm.net  Wed Nov 17 21:29:03 2004
From: foom at fuhm.net (James Y Knight)
Date: Wed Nov 17 21:29:06 2004
Subject: [Python-Dev] Re: subprocess and EINTR errnos
In-Reply-To: <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se>
References: <20041111034803.GA16175@isc.upenn.edu>
	<Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se>
Message-ID: <52765627-38D7-11D9-AA57-000A95A50FB2@fuhm.net>


On Nov 17, 2004, at 2:53 PM, Peter Astrand wrote:
> I assume you are using signals in your application? The os.read above 
> is
> not the only system call that can fail with EINTR. subprocess.py is 
> full
> of other system calls that can fail, and I suspect that many other 
> Python
> modules are as well.
>
> I've made a patch (attached) to subprocess.py (and test_subprocess.py)
> that should guard against EINTR, but I haven't committed it yet. It's
> quite large.
>
> Are Python modules supposed to handle EINTR? Why not let the C code 
> handle
> this? Or, perhaps the signal module should provide a sigaction 
> function,
> so that users can use SA_RESTART.

Here's a simple way to demonstrate the same problem (without 
subprocess). I think any solution should not be at the subprocess.py 
level.

 >>> import os,sys,signal
 >>> signal.signal(signal.SIGCHLD, lambda x,y: 
sys.stderr.write("SIGCHLD!\n"))
0
 >>> f=os.popen("sleep 5; echo 'Foo'"); print f.read()
SIGCHLD!
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
IOError: [Errno 4] Interrupted system call

This is an issue that Twisted has run into. Because Twisted installs 
its own SIGCHLD handler, other code which uses popen and then 
reads/writes the pipe *will* fail, because the SIGCHLD interrupts the 
final read. The proposed solution for twisted (which no one has 
implemented yet) is to make a small C module which simply adds the 
SA_RESTART flag to an installed signal handler. This has a 
disadvantage: because python queues up signals to run during normal 
interpreter execution, NOT at interrupt time, the handler won't be 
called until after the read finally returns. In our particular case, 
that's good enough, as the SIGCHLD handler is not particularly timing 
critical. As long as it gets called at some point, that's fine.

However, a more general solution would be to emulate the SA_RESTART 
functionality at Python level:
  - Add an optional argument to signal.signal to specify whether you 
want it to restart interrupted calls after handling the signal. This 
would *not* set SA_RESTART, but would instead set a flag for the python 
read/write/etc wrappers to look at.
  - Make the python wrapper for read/write/etc handle EINTR internally: 
first dispatch to the waiting signal handler, then call the syscall 
again, if the sighandler was installed with the restart argument.

 From my documentation for SA_RESTART:
"""The affected system calls include open(2), read(2), write(2),
      sendto(2), recvfrom(2), sendmsg(2) and recvmsg(2) on a 
communications
      channel or a slow device (such as a terminal, but not a regular 
file) and
      during a wait(2) or ioctl(2)."""
So, to emulate SA_RESTART, all those should be wrapped to restart in 
appropriate conditions.

James

From martin at v.loewis.de  Wed Nov 17 21:30:43 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed Nov 17 21:30:48 2004
Subject: [Python-Dev] os.access versus os.exist
In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245214@exchange.hqamor.amorhq.net>
References: <3A81C87DC164034AA4E2DDFE11D258E3245214@exchange.hqamor.amorhq.net>
Message-ID: <419BB4F3.4040402@v.loewis.de>

Robert Brewer wrote:
> Completely understood about LocalSystem. But IMO, os.stat should
> raise a more specific and correct error. As you pointed out, calling
> open() raises "IOError: [Errno 13] Permission denied"; it would be
> nice if stat did the same (in my example). The platform-specific API
> call (for example, Windows _stati64) doesn't provide this--it would
> require another test inside posix_do_stat to return a more
> appropriate error. But it would be a nice touch for us end-users.

No. Python faking up errors that the platform doesn't provide is evil.
You are entitled to precisely the error that the platform reported.

Now, MSVC stat() is severely constraint, in several ways (e.g. it
doesn't support sub-second time-stamps, either), so rewriting stat
with plain Win32 API might be reasonable, at which point giving
more precise error would be an option.

> In my opinion, "no such file" is the wrong error message and is
> therefore a bug. But I can accept others disagreeing, since os.stat()
> simply wraps Windows' _stat* calls (at least, in my example).

It sure looks like a bug - but one of msvcrt.dll.

Regards,
Martin
From pje at telecommunity.com  Wed Nov 17 21:34:08 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed Nov 17 21:33:09 2004
Subject: [Python-Dev] Re: __pycode__ extension
In-Reply-To: <cngae8$s80$1@sea.gmane.org>
References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20041117153303.02c412f0@mail.telecommunity.com>

At 12:56 PM 11/17/04 -0700, Fernando Perez wrote:
>Phillip J. Eby wrote:
>
> > If the idea is just to allow saving code from interactive mode, why not
> > just modify the interactive mode to do this?
>
>IPython already has most of this:
>
>In [3]: os.path.walk??
>Type:           function
>Base Class:     <type 'function'>
>String Form:    <function walk at 0x550529cc>
>Namespace:      Interactive
>File: 
>/usr/src/build/394694-i386/install/usr/lib/python2.3/posixpath.py

If I understand correctly, this is unrelated.  The OP was about retaining 
source of functions defined in interactive mode, not the source of 
functions from files.  'os.path.walk' was obviously not typed in using 
interactive mode.  :)


From mal at egenix.com  Wed Nov 17 21:41:16 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed Nov 17 21:41:18 2004
Subject: [Python-Dev] sys.exit(47) ?
In-Reply-To: <Pine.GSO.4.51L2.0411172107120.2244@koeberg.lysator.liu.se>
References: <419BA8EA.7030002@egenix.com>
	<Pine.GSO.4.51L2.0411172107120.2244@koeberg.lysator.liu.se>
Message-ID: <419BB76C.1020102@egenix.com>

Peter Astrand wrote:
> On Wed, 17 Nov 2004, M.-A. Lemburg wrote:
> 
> 
>>something in the regression suite is generating a temporary
>>file with content:
>>
>>#!/bin/sh
>>exec /path/to/python -c 'import sys; sys.exit(47)'
>>
>>and not cleaning up after itself.
>>
>>This started around Oct. 13. Looking at the regression suite
>>test_subprocess.py seems to be the cause.
> 
> 
> Thanks for pointing this out. Now fixed, in revision 1.15 of
> test_subprocess.py.

Thanks, I'll see whether my nightly build script will pick up the
change.

>>BTW, wouldn't sys.exit(42) be more appropriate :-?
> 
> 
> 42 is not as random :) See http://en.wikipedia.org/wiki/47_%28number%29

Interesting reference :-)

http://en.wikipedia.org/wiki/42_%28number%29

AFAIR, Tim Peters introduced that number to the Python universe.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 17 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From fumanchu at amor.org  Wed Nov 17 22:08:59 2004
From: fumanchu at amor.org (Robert Brewer)
Date: Wed Nov 17 22:10:35 2004
Subject: [Python-Dev] os.access versus os.exist
Message-ID: <3A81C87DC164034AA4E2DDFE11D258E324522B@exchange.hqamor.amorhq.net>

"Martin v. L?wis" wrote:
> Sent: Wednesday, November 17, 2004 12:31 PM
> To: Robert Brewer
> Cc: python-dev@python.org
> Subject: Re: [Python-Dev] os.access versus os.exist
> 
> 
> Robert Brewer wrote:
> > Completely understood about LocalSystem. But IMO, os.stat should
> > raise a more specific and correct error. As you pointed out, calling
> > open() raises "IOError: [Errno 13] Permission denied"; it would be
> > nice if stat did the same (in my example). The platform-specific API
> > call (for example, Windows _stati64) doesn't provide this--it would
> > require another test inside posix_do_stat to return a more
> > appropriate error. But it would be a nice touch for us end-users.
> 
> No. Python faking up errors that the platform doesn't provide is evil.
> You are entitled to precisely the error that the platform reported.
> 
> Now, MSVC stat() is severely constraint, in several ways (e.g. it
> doesn't support sub-second time-stamps, either), so rewriting stat
> with plain Win32 API might be reasonable, at which point giving
> more precise error would be an option.
> 
> > In my opinion, "no such file" is the wrong error message and is
> > therefore a bug. But I can accept others disagreeing, since 
> os.stat()
> > simply wraps Windows' _stat* calls (at least, in my example).
> 
> It sure looks like a bug - but one of msvcrt.dll.

OK. I'll file a small doc patch for os.path.exists, then, I think. Thanks!


Robert Brewer
MIS
Amor Ministries
fumanchu@amor.org
From fperez528 at yahoo.com  Wed Nov 17 22:35:31 2004
From: fperez528 at yahoo.com (Fernando Perez)
Date: Wed Nov 17 22:35:39 2004
Subject: [Python-Dev] Re: Re: __pycode__ extension
References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<cngae8$s80$1@sea.gmane.org>
	<5.1.1.6.0.20041117153303.02c412f0@mail.telecommunity.com>
Message-ID: <cngg73$g40$1@sea.gmane.org>

Phillip J. Eby wrote:

> If I understand correctly, this is unrelated.  The OP was about retaining
> source of functions defined in interactive mode, not the source of
> functions from files.  'os.path.walk' was obviously not typed in using
> interactive mode.  :)

OK, my misunderstanding.  You are right, and inspect does NOT seem to provide a
way to access the source of interactively defined functions (at least not that
I've seen):

In [6]: def foo(x):
   ...:     print x
   ...:

In [7]: psource foo
No source found for foo

Sorry for the confusion.

best,

f

From barry at python.org  Wed Nov 17 23:19:25 2004
From: barry at python.org (Barry Warsaw)
Date: Wed Nov 17 23:19:31 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <419BA86E.70309@bluewin.ch>
References: <419B2C76.1000709@iinet.net.au>
	<419A4C1A.5060502@ceid.upatras.gr>	<419B2C76.1000709@iinet.net.au>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<419BA86E.70309@bluewin.ch>
Message-ID: <1100729964.20177.29.camel@geddy.wooz.org>

On Wed, 2004-11-17 at 14:37, Samuele Pedroni wrote:

> But I think we should rember that some people today rely on the fact 
> that shipping only pycs files means not shipping directly the source.

Absolutely.  If this is added (and in general it sounds like a decent,
though PEP-worthy idea), there must be a way to turn it off.  I'm not
sure a build-time configuration option is enough, but a command line
option might be.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: This is a digitally signed message part
Url : http://mail.python.org/pipermail/python-dev/attachments/20041117/e423104b/attachment.pgp
From pedronis at bluewin.ch  Thu Nov 18 00:44:56 2004
From: pedronis at bluewin.ch (Samuele Pedroni)
Date: Thu Nov 18 00:41:26 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <1100729964.20177.29.camel@geddy.wooz.org>
References: <419B2C76.1000709@iinet.net.au>	
	<419A4C1A.5060502@ceid.upatras.gr>	<419B2C76.1000709@iinet.net.au>	
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>	
	<419BA86E.70309@bluewin.ch>
	<1100729964.20177.29.camel@geddy.wooz.org>
Message-ID: <419BE278.7030402@bluewin.ch>

Barry Warsaw wrote:

> On Wed, 2004-11-17 at 14:37, Samuele Pedroni wrote:
> 
> 
>>But I think we should rember that some people today rely on the fact 
>>that shipping only pycs files means not shipping directly the source.
> 
> 
> Absolutely. 

yes but further thinking I can imagine the following tension

- people will still want a way to ship without shipping the source, or 
such that reconstructing the source is not too easy, with varying 
degrees of what is considered enough "secure"

- although this can already be done partly today, this feature will make 
more tempting (especially in combination with features that can be 
expected from/constructed on top of the ast-branch or with the compiler 
package) to write e.g decorators that operate on functions by playing 
with their source. But then these tools will only work if the source or 
asts are there at runtime in some form [bytecode manipulation can be 
used similary but is less portable across python implementation, and 
seemingly harder than ast manipulations]

> If this is added (and in general it sounds like a decent,
> though PEP-worthy idea), there must be a way to turn it off.  I'm not
> sure a build-time configuration option is enough, but a command line
> option might be.
> 

and/or some setting changeable through a sys method

As I implicitly said I can see (at least these) three levels for the feature

1) disabled
2) store only the sources that cannot be recovered accessing .py files,
i.e. make inspect.getsource always work up to editing of the .py files
3) make things robust wrt .py files editing and other corner cases, i.e. 
likely store everything

to some extent 3 can be implemented without storing things in pyc files, 
as long as there are .py files around by an approach reading both in 
parallel

OTOH to have the tools described above work in a pyc-only scenario, 
source or asts would need to be stored in the pycs themself, opening the 
question about the resulting sufficient obfuscation/expressivity tradeoff
From carribeiro at gmail.com  Thu Nov 18 03:43:39 2004
From: carribeiro at gmail.com (Carlos Ribeiro)
Date: Thu Nov 18 03:44:24 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
Message-ID: <864d3709041117184370f63c1b@mail.gmail.com>

On Thu, 11 Nov 2004 22:50:21 +0100, Johan Hahn <johahn@home.se> wrote:
> Hi
> 
> As far as I can tell from the archive, this has not been discussed before.
> This is the second time in less than a week that I have stumbled over the rather
> clumsy syntax of extracting some elements of a sequence and at the same time
> remove those from the sequence:
> >>> L = 'a b 1 2 3'.split(' ')
> >>> a,b,L = L[0], L[1], L[2:]

I am really late on this thread, but anyway, I've come up with another
approach to solve the problem using iterators. It uses iterator that
is guaranteed to always return a fixed number of elements, regardless
of the size of the sequence; when it finishes, it returns the tail of
the sequence as the last argument. This is a simple-minded proof of
concept, and it's surely highly optimizable in at least a hundred
different ways :-)

def iunpack(seq, times, defaultitem=None):
    for i in range(times):
       if i < len(seq):
           yield seq[i]
       else:
           yield defaultitem
    if i < len(seq):
        yield seq[i+1:]
    else:
        yield ()

Usage is as follows:

>>> tuple(iunpack((1,2,3,4), 1))
(1, (2, 3, 4))
>>> tuple(iunpack((1,2,3,4), 2))
(1, 2, (3, 4))
>>> tuple(iunpack((1,2,3,4), 4))
(1, 2, 3, 4, ())
>>> tuple(iunpack((1,2,3,4), 6))
(1, 2, 3, 4, None, None, ())

As it is, it fails if the requested number of elements is zero, but
this is not a real use case for it anyway. But the best part is yet to
come. Because of the way Python implicitly packs & unpacks tuples, you
can use it *without* calling tuple():

>>> a,b = iunpack((1,2,3,4), 1)
>>> a,b
(1, (2, 3, 4))
>>> a,b,c = iunpack((1,2,3,4), 2)
>>> a,b,c
(1, 2, (3, 4))
>>> a,b,c,d,e = iunpack((1,2,3,4), 4)
>>> a,b,c,d,e
(1, 2, 3, 4, ())
>>> a,b,c,d,e,f,g = iunpack((1,2,3,4), 6)
>>> a,b,c,d,e,f,g
(1, 2, 3, 4, None, None, ())

The only catch is that, if you have only one parameter, then all you
will get is the generator itself.

>>> a = iunpack((1,2,3,4), 1)
>>> a
<generator object at 0x4071bfac>

But that's a corner case, and not the intended use anyway. Besides
that, there is an issue regarding the 'times' parameter; whether it
should return 'times' items plus the tail part, or 'times-1' items and
the tail part. I think that it's fine the way it is.

-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro@gmail.com
mail: carribeiro@yahoo.com
From sxanth at ceid.upatras.gr  Thu Nov 18 05:49:31 2004
From: sxanth at ceid.upatras.gr (Stelios Xanthakis)
Date: Thu Nov 18 05:49:39 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<419B2C76.1000709@iinet.net.au>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
Message-ID: <Pine.GSO.4.61.0411180639180.28978@diogenis.ceid.upatras.gr>



On Wed, 17 Nov 2004, Phillip J. Eby wrote:
>
>> The __pycode__ attribute is not marshaled.
>> So for 'import'ed code, __pycode__=None
>> 
>
> What good is that?  Likewise, I don't see the point of having this only 
> enabled when a command-line option is given.

We can marshal it if we want.
But I'm not convinced we should. Having used this system,
'import' is a good barrier to say "I'm not interested for
the __pycode__ of these". For example in the personal shell/library
were I collect nice recipies from usenet (premuttating letters
and fairy tales), I want to edit the recipe but when I import
httplib, I'm not interested in editing standard modules.
Afterall, we can execfile() in locals() to bypass import
and load a module with pycode.

>
> If the idea is just to allow saving code from interactive mode, why not just 
> modify the interactive mode to do this?
>

I think that is easilly doable by the way tok_nextc() is
implemented (in 2.3.4 at least). We can avoid this if
we have sourcefile/line.

btw, don't forget the exec case:

  exec """def spam():
     x = 1""" in globals(), globals()
  print spam.__pycode__

So having __pycode__==None can mean that
  - either there is source available, and use inspect
  - no source at all because we have pyc only

Another issue though is that if we implement this we
could modify inspect and drop file/lineno/linetab
and go read from __pycode__.

From python at rcn.com  Thu Nov 18 06:59:12 2004
From: python at rcn.com (Raymond Hettinger)
Date: Thu Nov 18 07:01:38 2004
Subject: [Python-Dev] Deprecate PyRange_New()
In-Reply-To: <1f7befae04111120514932c693@mail.gmail.com>
Message-ID: <000801c4cd33$bafbb220$e841fea9@oemcomputer>

> [Raymond Hettinger]
> > A while back, Tim added a comment that PyRange_New() should be
> > deprecated in-part because it is filled with problems and is not
used
> > anywhere in the core.

[Tim]
> And because it's undocumented.  The only place it's even mentioned is
> in the NEWS for 2.3a1:
> 
> - PyRange_New() now raises ValueError if the fourth argument is not 1.
>   This is part of the removal of deprecated features of the xrange
>   object.

[Raymond]
> > Doesn't anyone mind if I mark it in the C-API docs as deprecated so
it
> > can be phased out later?

[Tim]
> +1
> 
> I don't object to *documenting* a PyRange_New() API function, BTW, but
> the current implementation is too broken to fix with less than heroic
> effort.  If PyRange_New() needs to exist in the C API, then it should
> have the same (lo, hi, step) signature as the builtin range(), for
> which we have correct but non-obvious error-checking C code.  The
> funky (start, len, step, reps) signature of the current PyRange_New()
> is much more difficult to check for errors (e.g., detecting C long
> multiplication overflow in portable C is a sub-problem), and the
> current signature is too surprising regardless (because it's so
> different from the builtin range()).


Given that it is undocumented, broken, and recent, would it be over the
top to just remove it from Py2.4?


Raymond






From pje at telecommunity.com  Thu Nov 18 07:15:42 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu Nov 18 07:14:48 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <Pine.GSO.4.61.0411180639180.28978@diogenis.ceid.upatras.gr
 >
References: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<419B2C76.1000709@iinet.net.au>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>

At 06:49 AM 11/18/04 +0200, Stelios Xanthakis wrote:
>On Wed, 17 Nov 2004, Phillip J. Eby wrote:
>>What good is that?  Likewise, I don't see the point of having this only 
>>enabled when a command-line option is given.
>
>We can marshal it if we want.
>But I'm not convinced we should.

Apparently, we have different use cases for the source code.  In my use 
cases, I have little use for the interactive mode or exec.  For example, in 
the exec case, I would already have the source code, so why would I need it 
again?

I think it's important to clarify what this is *for*, and then address 
whatever that use case is directly.  So far, lots of people are projecting 
their own use cases onto it, and therefore we all disagree on what the best 
way to implement it is.


>  Having used this system,
>'import' is a good barrier to say "I'm not interested for
>the __pycode__ of these".

Having used it for whatever purposes *you're* using it for.  Evidently you 
want to use it for something very different from what I would want it for 
(source code transforms, such as AOP "advice" injection).

From ncoghlan at iinet.net.au  Thu Nov 18 11:44:07 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Thu Nov 18 11:44:12 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>
References: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>	<419B2C76.1000709@iinet.net.au>
	<419A4C1A.5060502@ceid.upatras.gr>	<419B2C76.1000709@iinet.net.au>	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>
Message-ID: <419C7CF7.1040808@iinet.net.au>

Phillip J. Eby wrote:
> At 06:49 AM 11/18/04 +0200, Stelios Xanthakis wrote:
 >
> I think it's important to clarify what this is *for*, and then address 
> whatever that use case is directly.  So far, lots of people are 
> projecting their own use cases onto it, and therefore we all disagree on 
> what the best way to implement it is.

True - hopefully a PEP can flush out most of the use cases.

>>  Having used this system,
>> 'import' is a good barrier to say "I'm not interested for
>> the __pycode__ of these".
> 
> 
> Having used it for whatever purposes *you're* using it for.  Evidently 
> you want to use it for something very different from what I would want 
> it for (source code transforms, such as AOP "advice" injection).

The interface I envisioned with my last message was something like:

   exec compile(source_str, source_file, 'exec', compile.STORE_SOURCE)

However, I've now discovered that the flags argument to compile is currently 
used only for controlling which __future__ statements are in effect. So this 
would be a significant change, as it would using a portion of the __future__ 
machinery to control something that is never intended to become default behaviour.

With this approach (or something similar, like a separate 'store=compile.SOURCE' 
argument to compile instead of piggybacking the future machinery), normal code 
remains unaffected, but an application can enable the functionality when it 
wants it by compiling the source code directly with the option enabled.

It may be reasonable to alter the interactive shell to enable the option for 
code entered at the interactive prompt.

This way, code compiled normally does not get its source stored in memory or the 
pyc files. Code which needs the source to operate correctly can be compiled with 
the option, and the relevant parts of the source would then be stored

With the ast-branch in, similar functionality could be provided for 
'compile.STORE_AST' or 'store=compile.AST' (OR the flags to store both source 
and AST).

At-least-we-don't-have-to-worry-about-getting-enough-material-for-a-PEP'ly,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Thu Nov 18 12:07:34 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Thu Nov 18 12:07:40 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <864d3709041117184370f63c1b@mail.gmail.com>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
	<864d3709041117184370f63c1b@mail.gmail.com>
Message-ID: <419C8276.5030809@iinet.net.au>

Carlos Ribeiro wrote:
> On Thu, 11 Nov 2004 22:50:21 +0100, Johan Hahn <johahn@home.se> wrote:
> 
>>Hi
>>
>>As far as I can tell from the archive, this has not been discussed before.
>>This is the second time in less than a week that I have stumbled over the rather
>>clumsy syntax of extracting some elements of a sequence and at the same time
>>remove those from the sequence:
>>
>>>>>L = 'a b 1 2 3'.split(' ')
>>>>>a,b,L = L[0], L[1], L[2:]
> 
> 
> I am really late on this thread, but anyway, I've come up with another
> approach to solve the problem using iterators. It uses iterator that
> is guaranteed to always return a fixed number of elements, regardless
> of the size of the sequence; when it finishes, it returns the tail of
> the sequence as the last argument. This is a simple-minded proof of
> concept, and it's surely highly optimizable in at least a hundred
> different ways :-)

So the original example becomes:
   a, b, L = itertools.iunpack(L, 2)

(and the 2 is an element count, not an index, so, as you say, there's no need to 
subtract 1)

That's certainly quite tidy, but it has the downside that it still copies the 
entire list as happens in the OP's code (the copying is hidden, but it still 
happens - the original list isn't destroyed until the assignment of the third 
value returned by the iterator).

It's also likely to require a trip to the docs to find out how iunpack works, 
and it doesn't fare well in the 'discovery' category (that is, the solution to 
what should be a fairly basic list operation is tucked away in itertools)

If list.pop gets updated to handle slice objects, then it can modify the list in 
place, and avoid any copying of list elements. "a, b = L.pop(slice(2)" should be 
able to give even better performance than "a = L.pop(0); b = L.pop(0)" (which 
is, I believe, the only current approach that avoids copying the entire list). 
And the list operation stays where it belongs - in a method of list.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Thu Nov 18 12:21:11 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Thu Nov 18 12:21:16 2004
Subject: [Python-Dev] Re: subprocess and EINTR errnos
In-Reply-To: <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se>
References: <20041111034803.GA16175@isc.upenn.edu>
	<Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se>
Message-ID: <419C85A7.70304@iinet.net.au>

Peter Astrand wrote:
> I've made a patch (attached) to subprocess.py (and test_subprocess.py)
> that should guard against EINTR, but I haven't committed it yet. It's
> quite large.

This is probably outside the bounds of the "conservative" checkins Anthony has 
requested prior to the release of 2.4.

Given also that it may affect more than just subprocess.py, this looks like a 
2.5 problem.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From carribeiro at gmail.com  Thu Nov 18 12:56:49 2004
From: carribeiro at gmail.com (Carlos Ribeiro)
Date: Thu Nov 18 12:57:03 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <419C8276.5030809@iinet.net.au>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
	<864d3709041117184370f63c1b@mail.gmail.com>
	<419C8276.5030809@iinet.net.au>
Message-ID: <864d370904111803561c650864@mail.gmail.com>

On Thu, 18 Nov 2004 21:07:34 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote:
> Carlos Ribeiro wrote:
> 
> 
> > On Thu, 11 Nov 2004 22:50:21 +0100, Johan Hahn <johahn@home.se> wrote:
> >
> >>Hi
> >>
> >>As far as I can tell from the archive, this has not been discussed before.
> >>This is the second time in less than a week that I have stumbled over the rather
> >>clumsy syntax of extracting some elements of a sequence and at the same time
> >>remove those from the sequence:
> >>
> >>>>>L = 'a b 1 2 3'.split(' ')
> >>>>>a,b,L = L[0], L[1], L[2:]
> >
> >
> > I am really late on this thread, but anyway, I've come up with another
> > approach to solve the problem using iterators. It uses iterator that
> > is guaranteed to always return a fixed number of elements, regardless
> > of the size of the sequence; when it finishes, it returns the tail of
> > the sequence as the last argument. This is a simple-minded proof of
> > concept, and it's surely highly optimizable in at least a hundred
> > different ways :-)
> 
> So the original example becomes:
>    a, b, L = itertools.iunpack(L, 2)
> 
> (and the 2 is an element count, not an index, so, as you say, there's no need to
> subtract 1)
> 
> That's certainly quite tidy, but it has the downside that it still copies the
> entire list as happens in the OP's code (the copying is hidden, but it still
> happens - the original list isn't destroyed until the assignment of the third
> value returned by the iterator).
> 
> It's also likely to require a trip to the docs to find out how iunpack works,
> and it doesn't fare well in the 'discovery' category (that is, the solution to
> what should be a fairly basic list operation is tucked away in itertools)
> 
> If list.pop gets updated to handle slice objects, then it can modify the list in
> place, and avoid any copying of list elements. "a, b = L.pop(slice(2)" should be
> able to give even better performance than "a = L.pop(0); b = L.pop(0)" (which
> is, I believe, the only current approach that avoids copying the entire list).
> And the list operation stays where it belongs - in a method of list.

list.pop doesn't solve the case for  when the data is stored in a
tuple (which is immutable). Also, a C implementation (of the type that
would be done for itertools inclusion) would certainly solve some of
the performance issues. As far as the documentation being tucked away
into itertools, I don't see that as a problem; in my opinion,
itertools already holds a good deal of utility functions which deserve
better study by novices (on a near par with builtins). And finally, I
believe that the name iunpack is already quite obvious on what it
means.

BTW, just because map, zip, enumerate, filter & etc are in builtins
doesn't mean that they are inherently more useful, or pythonic, than
the functions in itertools.  It's just a matter that itertools is a
relatively late addition to the pack; I hope it becomes more
proeminently adopted as people get used to them.

-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro@gmail.com
mail: carribeiro@yahoo.com
From barry at python.org  Thu Nov 18 15:06:28 2004
From: barry at python.org (Barry Warsaw)
Date: Thu Nov 18 15:06:32 2004
Subject: [Python-Dev] Deprecate PyRange_New()
In-Reply-To: <000801c4cd33$bafbb220$e841fea9@oemcomputer>
References: <000801c4cd33$bafbb220$e841fea9@oemcomputer>
Message-ID: <1100786788.9273.194.camel@presto.wooz.org>

On Thu, 2004-11-18 at 00:59, Raymond Hettinger wrote:

> Given that it is undocumented, broken, and recent, would it be over the
> top to just remove it from Py2.4?

Given that 2.4 is (or nearly is) a release candidate, do you even need
to ask that? :)

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: This is a digitally signed message part
Url : http://mail.python.org/pipermail/python-dev/attachments/20041118/76603f9e/attachment.pgp
From sxanth at ceid.upatras.gr  Thu Nov 18 15:14:04 2004
From: sxanth at ceid.upatras.gr (Stelios Xanthakis)
Date: Thu Nov 18 15:18:24 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>
References: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<419B2C76.1000709@iinet.net.au>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>
Message-ID: <Pine.GSO.4.61.0411180901380.3835@zenon.ceid.upatras.gr>



On Thu, 18 Nov 2004, Phillip J. Eby wrote:

>
> I think it's important to clarify what this is *for*, and then address 
> whatever that use case is directly.
>

Ok. Let's see it from the "programming languages" prespective.
In my opinion with the addition of __pycode__ (or __source__ from
now on), we have a good emulation of "data is code and code is data",
just like machine code.

In python we already have "data is code".

  X='''def f():
   pass'''
  exec X

But because of the conversion to bytecode we do not have
the "code is data" part (unless we consider code object data).

Why is this useful? I think it's one of the benefits of
interpreted vs compiled languages..

The real thing would be to attach __source__ to code objects, but
I think __source__ in functions and classes only is more realistic.

Anyway this is getting too theoretical and we can discuss it forever
on 'what is and what isn't code and data'..


As I said before: This *can* already be done in python as it is by
writing some kind of framework/IDE (Zope, Plone, IDLE, ipython,whatever). 
With __source__ though, python becomes a framework, we achieve this
with the minimum overhead and we are sure it will work with all cases
(interactive/exec) and all future extensions.


On the issue of saving __source__ to .pyc files, I'm 50-50.
I'd vote for -0.

Also I'd vote +1 for a command line option which if specified
__source__ will be generated for *all* functions/classes
(not only interactively defined).

Now I will go see about a PEP so that we can stop referring
to it as "it" and say "PEP xxx":)


Stelios



From pje at telecommunity.com  Thu Nov 18 16:33:25 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu Nov 18 16:32:33 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <Pine.GSO.4.61.0411180901380.3835@zenon.ceid.upatras.gr>
References: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<419B2C76.1000709@iinet.net.au>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com>

At 04:14 PM 11/18/04 +0200, Stelios Xanthakis wrote:

>On Thu, 18 Nov 2004, Phillip J. Eby wrote:
>>
>>I think it's important to clarify what this is *for*, and then address 
>>whatever that use case is directly.
>
>Ok. Let's see it from the "programming languages" prespective.
>In my opinion with the addition of __pycode__ (or __source__ from
>now on), we have a good emulation of "data is code and code is data",
>just like machine code.
[snip lots of explanation]

Your explanation still hasn't answered the question of what you plan to 
*do* with __source__.  Absent that, there are no criteria for making design 
decisions regarding the implementation.


>As I said before: This *can* already be done in python as it is by
>writing some kind of framework/IDE (Zope, Plone, IDLE, ipython,whatever).

In the absence of use cases, this is an argument for *not* making a change 
to the Python core.


>On the issue of saving __source__ to .pyc files, I'm 50-50.
>I'd vote for -0.
>
>Also I'd vote +1 for a command line option which if specified
>__source__ will be generated for *all* functions/classes
>(not only interactively defined).

And I'd vote differently on both these matters.  Please state what your use 
cases are, so that solutions can be evaluated on the basis of what use 
cases they satisfy, not merely votes without any context.


>Now I will go see about a PEP so that we can stop referring
>to it as "it" and say "PEP xxx":)

Please take especial care to include in the Motivation section what 
*specific* use cases this is intended to address.

So far the *only* use case you have presented is saving functions that were 
entered interactively, but this can be accomplished in far less 
controversial ways.

From pedronis at bluewin.ch  Thu Nov 18 16:52:35 2004
From: pedronis at bluewin.ch (Samuele Pedroni)
Date: Thu Nov 18 16:49:31 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com>
References: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>	<419B2C76.1000709@iinet.net.au>
	<419A4C1A.5060502@ceid.upatras.gr>	<419B2C76.1000709@iinet.net.au>	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>	<5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>
	<5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com>
Message-ID: <419CC543.1020405@bluewin.ch>

Phillip J. Eby wrote:

> 
> And I'd vote differently on both these matters.  Please state what your 
> use cases are, so that solutions can be evaluated on the basis of what 
> use cases they satisfy, not merely votes without any context.
> 

I don't get why your are particularly opposed to attaching source for 
exec-ed/eval-ed code,

I think that expanding the cases where inspect.getsource just work 
directly is valuable. And doing that seem a natural way to achieve this.

OTOH I personally fully appreciate why you may want to have it work
even if just pycs are around.

Stelios Xanthakis wrote:
 > Having used this system,
 > 'import' is a good barrier to say "I'm not interested for
 > the __pycode__ of these".

but for example having access to co_filename and or equilavent
(something would have to be for done for classes about this)
is probably also good enough to make that distinction:

 >>> def f(): pass
...
 >>> f.func_code.co_filename
'<stdin>'
 >>>

From python at rcn.com  Thu Nov 18 18:36:27 2004
From: python at rcn.com (Raymond Hettinger)
Date: Thu Nov 18 18:39:03 2004
Subject: [Python-Dev] Deprecate PyRange_New()
In-Reply-To: <1100786788.9273.194.camel@presto.wooz.org>
Message-ID: <001001c4cd95$2249ce60$e841fea9@oemcomputer>

> > Given that it is undocumented, broken, and recent, would it be over
the
> > top to just remove it from Py2.4?
> 
> Given that 2.4 is (or nearly is) a release candidate, do you even need
> to ask that? :)

Yes ;-)  The code is buggy and undocumented.  Fixing bugs and adding
documentation are appropriate after the beta and before the release
candidate.  

However, in this case, a complete bugfix would also entail a change in
the signature.  And since the function is unnecessary, removing it
entirely is a safer fix.  Leaving it in only increases the risk that
someone will find it and link to it.

OTOH, I don't care enough about this one to push it.


Raymond 

From rwgk at cci.lbl.gov  Thu Nov 18 19:02:55 2004
From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve)
Date: Thu Nov 18 19:03:04 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
Message-ID: <200411181802.iAII2tnl079279@boa.lbl.gov>

Brett C. wrote:
> I have been mulling over this proposal and I think I am finally
> settling on +0 as long as it can be shown that it does not hurt
> performance at all (yes, it shouldn't but this is the eval loop we are
> talking about and that thing is a touchy beast).

Is there a standard benchmark I could use to demonstrate the
impact of the two new additions on the runtime?
Thanks!
        Ralf
From pje at telecommunity.com  Thu Nov 18 19:06:34 2004
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu Nov 18 19:05:45 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <419CC543.1020405@bluewin.ch>
References: <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com>
	<5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<419B2C76.1000709@iinet.net.au>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>
	<5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com>
Message-ID: <5.1.1.6.0.20041118125507.031c8c60@mail.telecommunity.com>

At 04:52 PM 11/18/04 +0100, Samuele Pedroni wrote:
>Phillip J. Eby wrote:
>>And I'd vote differently on both these matters.  Please state what your 
>>use cases are, so that solutions can be evaluated on the basis of what 
>>use cases they satisfy, not merely votes without any context.
>
>I don't get why your are particularly opposed to attaching source for 
>exec-ed/eval-ed code,

I'm not against it per se.  I'm against adding a feature that crudely 
satisfies certain use cases, but has to be worked around by people who have 
other use cases.

In some cases, a feature can become an "attractive nuisance" that people 
will use because it's there most of the time, but it then becomes 
problematic when it's not available.  This is especially problematic when 
the presence or absence of the feature is on a global basis, potentially 
breaking the composability of Python modules.  (I.e., module A relies on 
the feature's presence, but module B requires its absence.)


>I think that expanding the cases where inspect.getsource just work 
>directly is valuable.

Agreed.


>  And doing that seem a natural way to achieve this.

Also agreed - but my concerns are with the larger consequences of having 
'getsource' *seem* to work everywhere.


>OTOH I personally fully appreciate why you may want to have it work
>even if just pycs are around.

For me, a half-working __source__ feature is almost worse than none at all, 
since the former could lead to the illusion of making progress towards code 
reflection.  This is why I'd like it made clearer what problems this is and 
is not meant to solve, so that programmers who are tempted to use the 
feature will be clearer as to what is and isn't a good idea to do with it.


From pedronis at bluewin.ch  Thu Nov 18 19:34:51 2004
From: pedronis at bluewin.ch (Samuele Pedroni)
Date: Thu Nov 18 19:31:24 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <5.1.1.6.0.20041118125507.031c8c60@mail.telecommunity.com>
References: <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com>	<5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>	<419B2C76.1000709@iinet.net.au>
	<419A4C1A.5060502@ceid.upatras.gr>	<419B2C76.1000709@iinet.net.au>	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>	<5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com>	<5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com>
	<5.1.1.6.0.20041118125507.031c8c60@mail.telecommunity.com>
Message-ID: <419CEB4B.4040306@bluewin.ch>

Phillip J. Eby wrote:

> 
>> OTOH I personally fully appreciate why you may want to have it work
>> even if just pycs are around.
> 
> 
> For me, a half-working __source__ feature is almost worse than none at 
> all, since the former could lead to the illusion of making progress 
> towards code reflection.  This is why I'd like it made clearer what 
> problems this is and is not meant to solve, so that programmers who are 
> tempted to use the feature will be clearer as to what is and isn't a 
> good idea to do with it.
> 

well, personally I think an inspect.getsource that is working and 
correct as much as possible is a worthy goal. Or a new getast.

At this point I see these possible levels of implementation/functionality:

0) disabled
1) attach source only to exec-ed/eval-ed code
    (with this level inspect.getsource would still do the wrong thing
     in the case for example of editing)
2) try to always attach source code, don't store it in pyc files,
for correct behavior a likely approach is to read pyc and py files
in parallel
3) always attach source code, store source in some form in the pyc files

I repeat that other means can be devised than flavor 1 of 
implementation, to distinguish exec-ed/eval-ed/interactive code from 
code from .py files.

Because of people with memory/disk usage considerations and people 
preferring no-source shipping we may have to implement to some extent 
all of 0,2,3.

Unless e.g. storing ASTs instead of source will be deemed enough safe,
or the fact that in 3 the source is not really just open-text-editor 
away (I doubt that).



From rsenra at acm.org  Thu Nov 18 23:16:06 2004
From: rsenra at acm.org (Rodrigo Dias Arruda Senra)
Date: Thu Nov 18 23:16:25 2004
Subject: [Python-Dev] __pycode__ extension
In-Reply-To: <419BE278.7030402@bluewin.ch>
References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr>
	<419B2C76.1000709@iinet.net.au>
	<5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com>
	<419BA86E.70309@bluewin.ch>
	<1100729964.20177.29.camel@geddy.wooz.org>
	<419BE278.7030402@bluewin.ch>
Message-ID: <20041118201606.69277207@Goku>

 [ Samuele Pedroni <pedronis@bluewin.ch> ]
 -----------------------------------------------
 | - people will still want a way to ship without shipping the source, or 
 | such that reconstructing the source is not too easy, with varying 
 | degrees of what is considered enough "secure"

 Instead of adding source code to .pyc, why not just add references to the
 original .py file inside the runtime PyObjects (iff cmdline directive is present).

 If the .py module is not present, then the absence of
 source *should* be respected.

 cheers,
 Rod Senra

-- 
   ,_           
   | )          Rodrigo Senra       <rsenra |at| acm.org>                      
   |(______     -----------------------------------------------
 _(    (|__|]   GPr Sistemas http://www.gpr.com.br                              
_ |    (|___|]  IC - Unicamp http://www.ic.unicamp.br/~921234  
___    (|__|]                       
   L___(|_|]    -----------------------------------------------
From doko at cs.tu-berlin.de  Fri Nov 19 01:21:42 2004
From: doko at cs.tu-berlin.de (Matthias Klose)
Date: Fri Nov 19 01:21:52 2004
Subject: [Python-Dev] python 2.3.5 release?
Message-ID: <16797.15510.883083.658735@gargle.gargle.HOWL>

Python 2.3.4 was released on 2004-05-27. According to the bi-annual
release schedule that would assume another release from the 2.3 branch
this year. Are people still interested in a formal 2.3.5 release? At
least the 2.3 branch does have a couple of patches worth making a
release. Is there anything that needs to be added before a 2.3.5
release is scheduled, or should we wait for the 2.4 release to
identify patches that need to be backported to a potential 2.3.5
release?

Thanks for any comment, Matthias
From barry at python.org  Fri Nov 19 01:27:59 2004
From: barry at python.org (Barry Warsaw)
Date: Fri Nov 19 01:28:03 2004
Subject: [Python-Dev] python 2.3.5 release?
In-Reply-To: <16797.15510.883083.658735@gargle.gargle.HOWL>
References: <16797.15510.883083.658735@gargle.gargle.HOWL>
Message-ID: <1100824079.8787.89.camel@geddy.wooz.org>

On Thu, 2004-11-18 at 19:21, Matthias Klose wrote:
> Python 2.3.4 was released on 2004-05-27. According to the bi-annual
> release schedule that would assume another release from the 2.3 branch
> this year. Are people still interested in a formal 2.3.5 release?

Yes, and I think Anthony is planning to do one a few weeks after 2.4
final.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 307 bytes
Desc: This is a digitally signed message part
Url : http://mail.python.org/pipermail/python-dev/attachments/20041118/e7a31ef2/attachment.pgp
From tim.peters at gmail.com  Fri Nov 19 01:30:35 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov 19 01:30:37 2004
Subject: [Python-Dev] python 2.3.5 release?
In-Reply-To: <16797.15510.883083.658735@gargle.gargle.HOWL>
References: <16797.15510.883083.658735@gargle.gargle.HOWL>
Message-ID: <1f7befae0411181630376522a9@mail.gmail.com>

Anthony announced his intent to produce a 2.3.5 release, after 2.4
final.  Brett announced his intent to send a "one-month warning" about
2.3.5 so people could lobby for patches.  Since I haven't seen such a
warning yet, 2.3.5 must be at least a month away <wink>.
From tim.peters at gmail.com  Fri Nov 19 03:08:20 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov 19 03:08:23 2004
Subject: [Python-Dev] New critical thread bug
Message-ID: <1f7befae041118180859aabf4a@mail.gmail.com>

This one is a puzzler.  See

    http://www.python.org/sf/1069160

for details.  The short course is that the PyThreadState_SetAsyncExc()
implementation fell into a common trap, and can cause segfaults under
rare conditions (like every other Python thread segfault bug we've
ever had).

This is easily repaired (although I've got no interest in doing the
coding, or even in contriving a test case -- this was an obvious "by
eyeball" bug).

The puzzle is how to treat this wrt 2.4.  Since it's a critical bug, I
suppose it "should" force another release candidate.  OTOH, this is a
C-only API (there's no exposure from Python) that's never used in the
core.  We could add code to make it segfault every time <wink>, and
nothing in the distribution would notice.

OTOH, if we broke its intended behavior while fixing the bug, we'd
never know that either.  "Never used in the core" means never -- the
function isn't tested.

On the third hand, it's a simple function with an obvious segfault
mode that has an obvious fix.

I'll leave it to the release manager <wink>.
From anthony at interlink.com.au  Fri Nov 19 04:57:00 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri Nov 19 05:00:23 2004
Subject: [Python-Dev] python 2.3.5 release?
In-Reply-To: <1f7befae0411181630376522a9@mail.gmail.com>
References: <16797.15510.883083.658735@gargle.gargle.HOWL>
	<1f7befae0411181630376522a9@mail.gmail.com>
Message-ID: <419D6F0C.3020102@interlink.com.au>

Tim Peters wrote:
> Anthony announced his intent to produce a 2.3.5 release, after 2.4
> final.  Brett announced his intent to send a "one-month warning" about
> 2.3.5 so people could lobby for patches.  Since I haven't seen such a
> warning yet, 2.3.5 must be at least a month away <wink>.

At the current time, it looks likely that this will be in early January.

Anthony


-- 
Anthony Baxter     <anthony@interlink.com.au>
It's never too late to have a happy childhood.
From bac at OCF.Berkeley.EDU  Fri Nov 19 05:01:42 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Fri Nov 19 05:01:47 2004
Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation()
In-Reply-To: <200411181802.iAII2tnl079279@boa.lbl.gov>
References: <200411181802.iAII2tnl079279@boa.lbl.gov>
Message-ID: <419D7026.6010007@ocf.berkeley.edu>

Ralf W. Grosse-Kunstleve wrote:
> Brett C. wrote:
> 
>>I have been mulling over this proposal and I think I am finally
>>settling on +0 as long as it can be shown that it does not hurt
>>performance at all (yes, it shouldn't but this is the eval loop we are
>>talking about and that thing is a touchy beast).
> 
> 
> Is there a standard benchmark I could use to demonstrate the
> impact of the two new additions on the runtime?
> Thanks!

=)  Parrotbench and PyBench would be enough for me to sign off on any 
performance hit.  There is also pystone.

-Brett
From bac at OCF.Berkeley.EDU  Fri Nov 19 05:09:31 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Fri Nov 19 05:09:46 2004
Subject: [Python-Dev] python 2.3.5 release?
In-Reply-To: <419D6F0C.3020102@interlink.com.au>
References: <16797.15510.883083.658735@gargle.gargle.HOWL>	<1f7befae0411181630376522a9@mail.gmail.com>
	<419D6F0C.3020102@interlink.com.au>
Message-ID: <419D71FB.3040605@ocf.berkeley.edu>

Anthony Baxter wrote:
> Tim Peters wrote:
> 
>> Anthony announced his intent to produce a 2.3.5 release, after 2.4
>> final.  Brett announced his intent to send a "one-month warning" about
>> 2.3.5 so people could lobby for patches.  Since I haven't seen such a
>> warning yet, 2.3.5 must be at least a month away <wink>.
> 
> 
> At the current time, it looks likely that this will be in early January.
> 

I will probably make some announcement in early December to remind people to 
get stuff in, especially with people having Winter vacation and thus either 
having less time because they are off galavanting around or more because they 
get to relax in front of the fire coding.

Anthony, can you keep me posted if you think it will slip much past early January?

-Brett
From bac at OCF.Berkeley.EDU  Fri Nov 19 05:10:53 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Fri Nov 19 05:11:16 2004
Subject: [Python-Dev] python 2.3.5 release?
In-Reply-To: <419D6F0C.3020102@interlink.com.au>
References: <16797.15510.883083.658735@gargle.gargle.HOWL>	<1f7befae0411181630376522a9@mail.gmail.com>
	<419D6F0C.3020102@interlink.com.au>
Message-ID: <419D724D.2080600@ocf.berkeley.edu>

Anthony Baxter wrote:
> Tim Peters wrote:
> 
>> Anthony announced his intent to produce a 2.3.5 release, after 2.4
>> final.  Brett announced his intent to send a "one-month warning" about
>> 2.3.5 so people could lobby for patches.  Since I haven't seen such a
>> warning yet, 2.3.5 must be at least a month away <wink>.
> 
> 
> At the current time, it looks likely that this will be in early January.
> 

I will probably make some announcement in early December to remind people to 
get stuff in, especially with people having Winter vacation and thus either 
having less time because they are gallivanting around or more because they get 
to relax in front of the fire coding.

Anthony, can you keep me posted if you think it will slip much past early January?

-Brett
From anthony at interlink.com.au  Fri Nov 19 08:16:45 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri Nov 19 08:17:05 2004
Subject: [Python-Dev] RELEASED Python 2.4, release candidate 1
Message-ID: <200411191816.52309.anthony@interlink.com.au>

On behalf of the Python development team and the Python community, I'm
happy to announce the first release candidate of Python 2.4.

Python 2.4c1 is a release candidate - we'd greatly appreciate it if
you could download it, kick the tires and let us know of any problems
you find, but it is probably not suitable for production usage. At
this point, we're particularly interested in build bugs. If you could
grab the source archive, build it and then run the test suite, we'd
appreciate it.

    http://www.python.org/2.4

Notable changes in rc1 include a handful of bug fixes, including a
thread shutdown race bug. See either the highlights, the What's New in
Python 2.4, or the detailed NEWS file -- all available from the Python
2.4 webpage.

This is the first release candidate of Python 2.4. Python 2.4 is now
close to release - there should be few or no new features, merely bug
fixes. Assuming no problems, we're looking at a final release around the
end of November (if there are problems, there will be another release
candidate).

Please log any problems you have with this release in the SourceForge
bug tracker (noting that you're using 2.4c1):

    http://sourceforge.net/bugs/?group_id=5470

Enjoy the new release,
Anthony

Anthony Baxter
anthony@python.org
Python Release Manager
(on behalf of the entire python-dev team)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20041119/9d4db1e6/attachment.pgp
From ncoghlan at iinet.net.au  Fri Nov 19 11:29:45 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Fri Nov 19 11:29:49 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <864d370904111803561c650864@mail.gmail.com>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>	
	<864d3709041117184370f63c1b@mail.gmail.com>	
	<419C8276.5030809@iinet.net.au>
	<864d370904111803561c650864@mail.gmail.com>
Message-ID: <419DCB19.5000609@iinet.net.au>

Carlos Ribeiro wrote:
> list.pop doesn't solve the case for  when the data is stored in a
> tuple (which is immutable).

For immutable objects, you *have* to make a copy, so I don't see any real 
downside to just using:

   a, b, T = T[0],  T[1], T[2:]

While I think iunpack is kind of neat, I don't think it solves anything which is 
currently a major problem, as it is really just a different way of spelling the 
above slicing. The major portion (sans some index checking) of iunpack(T, 2) can 
be written on one line:

   a, b, T = (T[i] for i in (range(2) + [slice(2, None)]))

When the number of elements to be unpacked is known at compile time (as it has 
to be to use tuple unpacking assignment), there seems little benefit in moving 
things inside a generator instead of spelling them out as a tuple of slices.

However, the OP's original question related to a list, where the goal was to 
both read the first couple of elements *and* remove them from the list (i.e. 
'pop'ing them). This is similar to the motivating use cases which came up the 
last time the "a, b, *c = L" idea was raised.

At the moment, the main choices are to use the syntax above (and make an 
unnecessary copy), make multiple calls to list.pop (with associated function 
overhead), or call del directly after reading the slice you want (i.e. doing a 
list.pop of a slice in Python code).

According to the current docs, L.pop(i) is defined as being equivalent to:

   x = L[i]; del L[i]; return x

And Armin pointed out that setting "i = slice(<whatever>)" actually causes that 
stated equivalence to break:

 >>> L = [1, 2, 3, 4]
 >>> i = slice(2)
 >>> L.pop(i)
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
TypeError: an integer is required
 >>> L[i]
[1, 2]
 >>> del L[i]
 >>> L
[3, 4]
 >>>

So, we could either fix the docs to explicitly exclude slices from list.pop, or 
just fix the method so slices work as the argument. The latter is a little more 
work, but it does provides a nice way of spelling "read and remove" for a range 
of elements. Which is why I'm hoping one of the proponents of "a, b, *c = L" 
will be willing to provide an implementation.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From anthony at interlink.com.au  Fri Nov 19 12:03:52 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri Nov 19 12:22:26 2004
Subject: [Python-Dev] TRUNK UNFROZEN
Message-ID: <200411192203.52490.anthony@interlink.com.au>

Ok, it's been long enough without any screamingly obvious problems, so I'm
declaring the trunk unfrozen.

Thanks to Tim's latest discovery, it looks like we might need an rc2 - 
I might try to aim for this in a week's time (assuming that's ok with
Fred and Martin) and try to keep things moving along to 2.4 final. 
After which, I'm going to go out and get very drunk <wink>

Anthony
From ncoghlan at iinet.net.au  Fri Nov 19 16:03:42 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Fri Nov 19 16:03:49 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
Message-ID: <419E0B4E.7040109@iinet.net.au>

This was meant to be a "Yup, she's all good" exercise before 2.4 final goes out 
the door.

Not quite so simple, unfortunately. . . (FWIW, I doubt this is a Python problem, 
but I'd like some of the other folks here who use Cygwin to have a look at the 
failure symptoms)

My Python build (built & running under Cygwin) seems to be having trouble with 
any tests involving threads (test_asynchat, test_thread, test_threaded_import, 
test_socket_ssl for starters).

When run using "./python -E Lib/test/regrtest.py -unetwork,decimal -r", several 
tests execute normally, then the test execution hangs. From what I can tell, the 
hang generally occurs on the first test that makes use of threading.

When run on their own, the affected tests run for a short while, then terminate 
abruptly (i.e. regrtest.py doesn't print its result summary, and the output from 
the tests themselves shows they haven't run to completion).

I suspect (hope?) that the problem is actually in my Cygwin setup rather than in 
Python, but I'm trying to figure out how to go about proving that. I did a 'make 
clean' and rebuild to see if stale files were the problem, then a 'make install' 
to see if the problem was accidentally picking the old python2.3 DLL's instead 
of the python2.4 DLL's, but neither of those appeared to make any difference.

I only got the regression tests working *at all* today, after doing 'find . 
-name "\.dll" | rebaseall -T -' from my python/dist/src directory to get rid of 
the Cygwin remap errors when loading DLL's.

I figured I'd ask the folks here for ideas (is Jason still around?), before 
trying to ask the question from the Cygwin end.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From carribeiro at gmail.com  Fri Nov 19 16:24:48 2004
From: carribeiro at gmail.com (Carlos Ribeiro)
Date: Fri Nov 19 16:24:51 2004
Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list
In-Reply-To: <419DCB19.5000609@iinet.net.au>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
	<864d3709041117184370f63c1b@mail.gmail.com>
	<419C8276.5030809@iinet.net.au>
	<864d370904111803561c650864@mail.gmail.com>
	<419DCB19.5000609@iinet.net.au>
Message-ID: <864d37090411190724440dfe38@mail.gmail.com>

On Fri, 19 Nov 2004 20:29:45 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote:
> Carlos Ribeiro wrote:
> > list.pop doesn't solve the case for  when the data is stored in a
> > tuple (which is immutable).
> 
> For immutable objects, you *have* to make a copy, so I don't see any real
> downside to just using:
> 
>    a, b, T = T[0],  T[1], T[2:]

(First of all, let me say that I am not at all against making list.pop
accept slices. It's nice that it seems to be agreed that this is a
worthy addition. And I don't intend to keep pushing for iunpack(); but
I feel that is important to make a few clarifications.)

If I want to do:

a,b,X = T[0],  T[1], T[2:]   (?)

The lesson here is that it can't be assumed for unpacking purposes
that the programmer wanted to modify the original list. Of course, if
he does want it, then list.pop just comes nice.

(also, list.pop(slice) can be used to remove elements from anywhere on
the list, which is a big plus)

> While I think iunpack is kind of neat, I don't think it solves anything which is
> currently a major problem, as it is really just a different way of spelling the
> above slicing. The major portion (sans some index checking) of iunpack(T, 2) can
> be written on one line:
> 
>    a, b, T = (T[i] for i in (range(2) + [slice(2, None)]))

I would rather prefer not to have to use this idiom -- it's neither
obvious nor convenient. Either list.pop or unpack would be cleaner for
this purpose.

> When the number of elements to be unpacked is known at compile time (as it has
> to be to use tuple unpacking assignment), there seems little benefit in moving
> things inside a generator instead of spelling them out as a tuple of slices.

For more than a few arguments, it seems to be silly to require the
user to write it as:

a,b,c,d,e = t[0],t[1],t[2],t[3],t[4:]

[lots of remarks about list.pop snipped]

I'm not against list.pop; it's just that iunpack provides a different
approach to the problem. It's fairly generic: it works for both
mutable and immutable lists. The implementation provided is a proof of
concept. The fact that it does not modify the original arguments *is*
a design feature (althought not what the OP wanted for his particular
case). After all, as shown in the example above, conventional tuple
unpacking on assignment doesn't change the right-side arguments
either.

One posible improvement for iunpack() is to accept any iterable as an
argument. On return, instead of a tuple to represent the remaining
elements, it would return the generator itself. This would allow for
code that consume the generator in "chunks" to be written rather
simply:

a,b,c,g = iunpack(3, g)

In this case, to solve the OP problem, instead to have all the values
to be processed stored into a list, he could rely on a generator to
produce values on the fly. Depending on the scenario this would come
handy (less memory consumption, more responsiveness).

-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro@gmail.com
mail: carribeiro@yahoo.com
From tim.peters at gmail.com  Fri Nov 19 18:37:13 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Fri Nov 19 18:37:15 2004
Subject: [Python-Dev] TRUNK UNFROZEN
In-Reply-To: <200411192203.52490.anthony@interlink.com.au>
References: <200411192203.52490.anthony@interlink.com.au>
Message-ID: <1f7befae04111909376b170b56@mail.gmail.com>

[Anthony Baxter]
...
> Thanks to Tim's latest discovery, it looks like we might need an rc2 -
> I might try to aim for this in a week's time (assuming that's ok with
> Fred and Martin) and try to keep things moving along to 2.4 final.
> After which, I'm going to go out and get very drunk <wink>

FWIW, I wouldn't bother with an rc2 just for the
PyThreadState_SetAsyncExc segfault bug.  It's never been reported in
real life, it can't occur unless a 3rd-party C extension is using this
API, and some googling suggests that the API is barely used by any
extensions.

To get in trouble, C code has to use this API to target a thread that
happens to be shutting down, and then suffer a sequence of unlikely
timing accidents.  It seems unlikely that code using this API *would*
target a thread that's shutting down (e.g., it's more likely to target
a thread that's *too* active <wink>), which may be why it hasn't been
reported yet in real life.

So it's OK by my if everyone pretends I didn't file the bug report
until a month after 2.4 final was released; Guido may even be able to
arrange to make that true.

Of course it's fine to get very drunk regardless.
From mulk at gmx.net  Fri Nov 19 18:55:23 2004
From: mulk at gmx.net (Matthias Andreas Benkard)
Date: Fri Nov 19 18:48:38 2004
Subject: [Python-Dev] Confusion about os.urandom()
Message-ID: <1100886923.10228.7.camel@mulkomp.mst-plus>

Hello there.

I was just reading through http://www.python.org/2.4/highlights.html and
noticed the following:

    os.urandom() has been added for systems that support a source of
    random data (entropy)

At first, I was a bit stunned about the choice of name here. Why would
anyone call this method urandom()? That confused me a bit, for, AFAICS,
under Linux at least, /dev/random is the entropy pool and /dev/urandom
is a PRNG (or rather, a source of random numbers which falls back to a
PRNG when the entropy pool runs out of numbers). So I would not expect a
method that is supposed to yield cryptographically secure random numbers
to be called `urandom()'.

Anyway, that might be non-intuitive (which is unusual in Python), but
certainly not a severe bug. But then I noticed that if I
rename /dev/urandom to something else (say, /dev/notrandomenough),
os.urandom() (in Python 2.4b2) raises a NotImplementedError:

Python 2.4b2 (#2, Nov 11 2004, 23:44:54)
[GCC 3.3.5 (Debian 1:3.3.5-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.urandom(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.4/os.py", line 693, in urandom
    raise NotImplementedError("/dev/urandom (or equivalent) not found")
NotImplementedError: /dev/urandom (or equivalent) not found
>>>

Now I'm really confused. Does os.urandom() use /dev/urandom under Linux?
Either my brain has got a serious bug that makes me misunderstand the
random(4) manpage, which states that "[w]hen read, /dev/urandom device
will return as many bytes as are requested.  As a result, if there is
not sufficient entropy in the entropy pool, the returned values are
theoretically vulnerable to a cryptographic attack on the algorithms
used by the driver", or the implementation of os.urandom() is severely
flawed.

(BTW, I just checked Python 2.4c1. Judging from the source code, it
seems to have the same behaviour.)

That's what help(os.urandom) says:

    urandom(n) -> str
    Return a string of n random bytes suitable for cryptographic use.

So it should be using /dev/random rather than /dev/urandom, shouldn't
it?

- Matthias A. Benkard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.python.org/pipermail/python-dev/attachments/20041119/1a293b22/attachment.pgp
From FBatista at uniFON.com.ar  Fri Nov 19 20:02:55 2004
From: FBatista at uniFON.com.ar (Batista, Facundo)
Date: Fri Nov 19 20:04:48 2004
Subject: [Python-Dev] Misinformation in highlights webpage
Message-ID: <A128D751272CD411BC9200508BC2194D053C7C25@escpl.tcp.com.ar>

In http://www.python.org/2.4/highlights.html it presents decimal between the
"New or upgraded built-ins", while it should talk about it in "New or
upgraded modules and packages".

Should I post a bug or this is enough? (never know what to do regarding
webpages).

Thanks!

Facundo Batista
Desarrollo de Red
fbatista@unifon.com.ar
(54 11) 5130-4643
Cel: 15 5097 5024


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20041119/0f3d7c2a/attachment.html
From irmen at xs4all.nl  Fri Nov 19 20:16:55 2004
From: irmen at xs4all.nl (Irmen de Jong)
Date: Fri Nov 19 20:16:56 2004
Subject: [Python-Dev] RELEASED Python 2.4, release candidate 1
In-Reply-To: <200411191816.52309.anthony@interlink.com.au>
References: <200411191816.52309.anthony@interlink.com.au>
Message-ID: <419E46A7.1080501@xs4all.nl>

Anthony Baxter wrote:
> Notable changes in rc1 include a handful of bug fixes, including a
> thread shutdown race bug. See either the highlights, the What's New in
> Python 2.4, or the detailed NEWS file -- all available from the Python
> 2.4 webpage.


I'm looking in the NEWS file, but can't seem to locate the
fixed bugs/patches mentioned in the status page of the last
bugday: http://www.python.org/moin/PythonBugDayStatus
Are they left out? Or simply not mentioned?

--Irmen de Jong
From anthony at interlink.com.au  Fri Nov 19 20:41:41 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri Nov 19 20:41:52 2004
Subject: [Python-Dev] RELEASED Python 2.4, release candidate 1
In-Reply-To: <419E46A7.1080501@xs4all.nl>
References: <200411191816.52309.anthony@interlink.com.au>
	<419E46A7.1080501@xs4all.nl>
Message-ID: <200411200641.41478.anthony@interlink.com.au>

On Saturday 20 November 2004 06:16, Irmen de Jong wrote:
> I'm looking in the NEWS file, but can't seem to locate the
> fixed bugs/patches mentioned in the status page of the last
> bugday: http://www.python.org/moin/PythonBugDayStatus
> Are they left out? Or simply not mentioned?

Beats me. Did whoever do the checkin add entries in the NEWS
file? If not, it's unlikely that they will have been added. I long ago
gave up on trying to trawl back through CVS changelogs and match
them to NEWS entries -  life's too short.

Anthony
From tjreedy at udel.edu  Fri Nov 19 21:06:27 2004
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri Nov 19 21:06:22 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized list
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME><864d3709041117184370f63c1b@mail.gmail.com><419C8276.5030809@iinet.net.au><864d370904111803561c650864@mail.gmail.com><419DCB19.5000609@iinet.net.au>
	<864d37090411190724440dfe38@mail.gmail.com>
Message-ID: <cnljnp$8r8$1@sea.gmane.org>


"Carlos Ribeiro" <carribeiro@gmail.com> wrote in message 
news:864d37090411190724440dfe38@mail.gmail.com...
> For more than a few arguments, it seems to be silly to require the
> user to write it as:
> a,b,c,d,e = t[0],t[1],t[2],t[3],t[4:]

and also impossible.
>>> t=range(10)
>>> a,b,c,d=t[:4]; e=t[4:]
>>> a,b,c,d,e
(0, 1, 2, 3, [4, 5, 6, 7, 8, 9])

but this does require that return values be captured in a temporary first. 
This issue has come up enough that a PEP would be useful (if there isn't 
one already).

Terry J. Reedy



From martin at v.loewis.de  Fri Nov 19 21:46:47 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri Nov 19 21:46:47 2004
Subject: [Python-Dev] TRUNK UNFROZEN
In-Reply-To: <200411192203.52490.anthony@interlink.com.au>
References: <200411192203.52490.anthony@interlink.com.au>
Message-ID: <419E5BB7.1040902@v.loewis.de>

Anthony Baxter wrote:
> Thanks to Tim's latest discovery, it looks like we might need an rc2 - 
> I might try to aim for this in a week's time (assuming that's ok with
> Fred and Martin) and try to keep things moving along to 2.4 final. 

That's fine with me.

Regards,
Martin
From foom at fuhm.net  Fri Nov 19 21:48:19 2004
From: foom at fuhm.net (James Y Knight)
Date: Fri Nov 19 21:48:22 2004
Subject: [Python-Dev] print "%X" % id(object()) not so nice
Message-ID: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net>

I think id() should never be returning a negative number. Both these 
behaviors are poor:

In 2.3:
 >>> print "%X" % id(o)
__main__:1: FutureWarning: %u/%o/%x/%X of negative int will return a 
signed string in Python 2.4 and up
A5F48198

In 2.4:
 >>> print "%X" %id(o)
-5FC84D08

Pointers are conventionally never treated or printed as signed. In 2.3 
and before, it usually ended up okay, besides the warning, because "%X" 
had broken behavior. In 2.4, now it's ending up doing the wrong thing 
and printing a confusing value. I propose that id() always return a 
positive value. This means that it will sometimes have to return a long 
instead of an int, but, it already does that under some circumstances 
on some architectures.

Comments?

James

From martin at v.loewis.de  Fri Nov 19 21:59:06 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri Nov 19 21:59:06 2004
Subject: [Python-Dev] Confusion about os.urandom()
In-Reply-To: <1100886923.10228.7.camel@mulkomp.mst-plus>
References: <1100886923.10228.7.camel@mulkomp.mst-plus>
Message-ID: <419E5E9A.7080502@v.loewis.de>

Matthias Andreas Benkard wrote:
> At first, I was a bit stunned about the choice of name here. Why would
> anyone call this method urandom()? That confused me a bit, for, AFAICS,
> under Linux at least, /dev/random is the entropy pool and /dev/urandom
> is a PRNG (or rather, a source of random numbers which falls back to a
> PRNG when the entropy pool runs out of numbers).

That is not true. It doesn't first exhaust the pool and then falls back
to PRNG. Instead, it gradually moves to a PRNG, depending on the amount
of entropy in the pool. The values returned are still cryptographically
secure, except in purely theoretical cases (where a lot of entropy is
drawn from random or urandom, and nothing is filled in).

> Now I'm really confused. Does os.urandom() use /dev/urandom under Linux?

Yes, it does.

> That's what help(os.urandom) says:
> 
>     urandom(n) -> str
>     Return a string of n random bytes suitable for cryptographic use.
> 
> So it should be using /dev/random rather than /dev/urandom, shouldn't
> it?

No, it shouldn't. /dev/random may block, which os.urandom() will not.
The name urandom deliberately tells users that there is a theoretical
flaw (which is practically irrelevant). If users cannot stand the
theoretical flaw, they need to use /dev/random (which also has
theoretical flaws that just happen to be even less practically
relevant). In that case
a) they have to accept that reading /dev/random might block
    indefinitely, and
b) their code will become more system-dependent.

Regards,
Martin
From martin at v.loewis.de  Fri Nov 19 22:02:15 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri Nov 19 22:02:16 2004
Subject: [Python-Dev] print "%X" % id(object()) not so nice
In-Reply-To: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net>
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net>
Message-ID: <419E5F57.10509@v.loewis.de>

James Y Knight wrote:
> Comments?

This cannot be implemented for 2.4. For 2.5, it is reasonable;
feel free to submit a patch.

Regards,
Martin
From jason at tishler.net  Sat Nov 20 00:27:33 2004
From: jason at tishler.net (Jason Tishler)
Date: Sat Nov 20 00:25:04 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
In-Reply-To: <419E0B4E.7040109@iinet.net.au>
References: <419E0B4E.7040109@iinet.net.au>
Message-ID: <20041119232733.GB1680@tishler.net>

Nick,

On Sat, Nov 20, 2004 at 01:03:42AM +1000, Nick Coghlan wrote:
> I only got the regression tests working *at all* today, after doing
> 'find .  -name "\.dll" | rebaseall -T -' from my python/dist/src
> directory to get rid of the Cygwin remap errors when loading DLL's.

The above is expected behavior, but see below...

> I figured I'd ask the folks here for ideas (is Jason still around?),
> before trying to ask the question from the Cygwin end.

Yes, I'm still around. :,)

What Cygwin version are you using?  1.5.12?

FWIW, I got the following results when running Python-2.4b1's normal
(i.e., no extras) regression test:

    $ ./python.exe -E -tt ./Lib/test/regrtest.py -l
    ...
    252 tests OK.
    2 tests failed:
        test_subprocess test_tcl
    ...

The above results was under a pre-1.5.12 Cygwin CVS on XP Pro SP1:

    $ uname -a
    CYGWIN_NT-5.1 gelpdevjt022 1.5.12(0.116/4/2) 2004-10-28 07:28 ...

Amazingly, I did *not* need to run rebase like I normal do!

Please let me know if you need help tracking down Cygwin related
problems.

Thanks,
Jason

-- 
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D  8784 1AFD E4CC ECF4 8EF6
From tjreedy at udel.edu  Sat Nov 20 01:38:21 2004
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat Nov 20 01:38:18 2004
Subject: [Python-Dev] Re: print "%X" % id(object()) not so nice
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net>
Message-ID: <cnm3lj$9nj$1@sea.gmane.org>


"James Y Knight" <foom@fuhm.net> wrote in message 
news:582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net...
>[snip]
>I propose that id() always return a positive value ...
> Comments?

1. CPython intentionally searches builtins afters globals and pre-imports 
the former as __builtins__ just so one can wrap a builtin to modify its 
apparent behavior for personal needs.

>>> def id(o, max = 2**32):
...   i = __builtins__.id(o)
...   return (i < 0) and (max - i) or i
...
>>> id(a)
9158224
# only semi-tested because I don't have negatives to test and can't force

2. Given that, why bother changing the language for what must be an 
esoteric need (to formattedly print and view ids)?

The id of an object is constant and unique with respect to contemporaneous 
objects but, for CPython, definitely not with respect to objects with 
non-overlapping lifetimes.  (Newbies often get tripped by the last fact.). 
>From the viewpoint of Python, ids have no meaning and are only useful for 
identity comparision.  For this purpose, arbitrary strings would have 
worked as well as integers.

For convenience, CPython uses the apparent address stored as an int.  But 
this is strictly an implementation detail.  On modern systems, that 
'address' is, I believe, a process-specific virtual address which the 
hardware memory management system maps the hidden real address -- which is 
the only reason why systems with less than 2**31 memory can have addresses 
at or above 2**31 to become negative ints.

Terry J. Reedy



From Scott.Daniels at Acm.Org  Sat Nov 20 01:52:55 2004
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Sat Nov 20 01:51:41 2004
Subject: [Python-Dev] Re: print "%X" % id(object()) not so nice
In-Reply-To: <cnm3lj$9nj$1@sea.gmane.org>
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net>
	<cnm3lj$9nj$1@sea.gmane.org>
Message-ID: <cnm4ek$b4v$1@sea.gmane.org>

Terry Reedy wrote:
>>>>def id(o, max = 2**32):
> 
> ...   i = __builtins__.id(o)
> ...   return (i < 0) and (max - i) or i
> ...
> 
This code doesn't need an extra parameter:
     def id(o):
         return (sys.maxint * 2 + 1) & __builtins__.id(o)

-- 
-- Scott David Daniels
Scott.Daniels@Acm.Org

From bob at redivi.com  Sat Nov 20 01:55:08 2004
From: bob at redivi.com (Bob Ippolito)
Date: Sat Nov 20 01:55:53 2004
Subject: [Python-Dev] Re: print "%X" % id(object()) not so nice
In-Reply-To: <cnm3lj$9nj$1@sea.gmane.org>
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net>
	<cnm3lj$9nj$1@sea.gmane.org>
Message-ID: <D3128502-3A8E-11D9-925A-000A9567635C@redivi.com>


On Nov 20, 2004, at 2:38 AM, Terry Reedy wrote:

>
> "James Y Knight" <foom@fuhm.net> wrote in message
> news:582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net...
>> [snip]
>> I propose that id() always return a positive value ...
>> Comments?
>
> 1. CPython intentionally searches builtins afters globals and 
> pre-imports
> the former as __builtins__ just so one can wrap a builtin to modify its
> apparent behavior for personal needs.
>
>>>> def id(o, max = 2**32):
> ...   i = __builtins__.id(o)
> ...   return (i < 0) and (max - i) or i
> ...
>>>> id(a)
> 9158224
> # only semi-tested because I don't have negatives to test and can't 
> force
>
> 2. Given that, why bother changing the language for what must be an
> esoteric need (to formattedly print and view ids)?
>
> The id of an object is constant and unique with respect to 
> contemporaneous
> objects but, for CPython, definitely not with respect to objects with
> non-overlapping lifetimes.  (Newbies often get tripped by the last 
> fact.).
>> From the viewpoint of Python, ids have no meaning and are only useful 
>> for
> identity comparision.  For this purpose, arbitrary strings would have
> worked as well as integers.
>
> For convenience, CPython uses the apparent address stored as an int.  
> But
> this is strictly an implementation detail.  On modern systems, that
> 'address' is, I believe, a process-specific virtual address which the
> hardware memory management system maps the hidden real address -- 
> which is
> the only reason why systems with less than 2**31 memory can have 
> addresses
> at or above 2**31 to become negative ints.

The problem, more than anything else, is the following behavior that 
can happen during a random __repr__ or repr-like-function if the object 
happens to have a certain address range:

- (Python 2.3) You get an unexpected and unwanted warning but expected 
output anyway
- (Python 2.4) You get a repr with a strange looking negative hex 
number (0x-FF0102)

Neither of these are fatal, of course, it's just annoying.. I find the 
Python 2.3 behavior more obnoxious than Python 2.4's, personally.

FYI, I have also encountered this "problem" this week on a Powerbook G4 
w/ only 1GB physical memory on both Python 2.3 and 2.4.  I'm at the 
PyPy sprint, and a lot of the tools we are using make use of repr.  
Fortunately we have control over all of this code, so I checked in a 
workaround that makes sure a sane value was passed to the hex 
formatter:

import sys
HUGEINT = (sys.maxint + 1L) * 2L

def uid(obj):
     """
     Return the id of an object as an unsigned number so that its hex
     representation makes sense
     """
     rval = id(obj)
     if rval < 0:
         rval += HUGEINT
     return rval

-bob

From ncoghlan at iinet.net.au  Sat Nov 20 03:28:50 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 20 03:28:56 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
In-Reply-To: <20041119232733.GB1680@tishler.net>
References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net>
Message-ID: <419EABE2.9040804@iinet.net.au>

Jason Tishler wrote:
> Nick,
> 
> On Sat, Nov 20, 2004 at 01:03:42AM +1000, Nick Coghlan wrote:
> 
>>I only got the regression tests working *at all* today, after doing
>>'find .  -name "\.dll" | rebaseall -T -' from my python/dist/src
>>directory to get rid of the Cygwin remap errors when loading DLL's.
 >
> The above is expected behavior, but see below...

I thought so - it was your advice that Google turned up to tell me how to fix it :)

>>I figured I'd ask the folks here for ideas (is Jason still around?),
>>before trying to ask the question from the Cygwin end.
> 
> Yes, I'm still around. :,)
> 
> What Cygwin version are you using?  1.5.12?

1.5.11 (also on XP Pro SP1)

$ uname -a
CYGWIN_NT-5.1 pseudopolis 1.5.11(0.116/4/2) 2004-09-04 23:17 ...

I'll try updating all the Cygwin packages and see where that gets me.

<that didn't take long>

Now it's hanging in configure (and the m4 interpreter was one of the things that 
got updated). *sigh* I need to wake up more before I dig into this.

If you (or anyone else) could check that the unit tests work on Cygwin for you, 
that would make me more certain that this is just a problem with my own Cygwin 
setup, and not related to anything in 2.4c1 that wasn't in 2.4b1.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Sat Nov 20 03:29:50 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 20 03:29:55 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized
	list
In-Reply-To: <cnljnp$8r8$1@sea.gmane.org>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME><864d3709041117184370f63c1b@mail.gmail.com><419C8276.5030809@iinet.net.au><864d370904111803561c650864@mail.gmail.com><419DCB19.5000609@iinet.net.au>	<864d37090411190724440dfe38@mail.gmail.com>
	<cnljnp$8r8$1@sea.gmane.org>
Message-ID: <419EAC1E.6020902@iinet.net.au>

Terry Reedy wrote:
> "Carlos Ribeiro" <carribeiro@gmail.com> wrote in message 
> news:864d37090411190724440dfe38@mail.gmail.com...
> 
>>For more than a few arguments, it seems to be silly to require the
>>user to write it as:
>>a,b,c,d,e = t[0],t[1],t[2],t[3],t[4:]
> 
> 
> and also impossible.
> 
>>>>t=range(10)
>>>>a,b,c,d=t[:4]; e=t[4:]
>>>>a,b,c,d,e
> 
> (0, 1, 2, 3, [4, 5, 6, 7, 8, 9])

Hmm. . .

_>>>t = range(10)
_>>>a, b, c, d, e = t[:4] + [t[4:]]
_>>>a, b, c, d, e
_(0, 1, 2, 3, [4, 5, 6, 7, 8, 9])

So this actually can scale (sort of) to longer tuples.

Anyway, I don't have any real objection to iunpack. If it was designed to work 
on any iterator (return the first few elements, then return the partially 
consumed iterator), I'd be +1 (since, as Carlos pointed out, slicing doesn't 
work for arbitrary iterators).

Something like:

_>>>def iunpack(itr, numitems, defaultitem=None):
_...    for i in range(numitems):
_...        try:
_...            yield itr.next()
_...        except StopIteration:
_...            yield defaultitem
_...    yield itr

_>>> g = (x for x in range(10))
_>>> a, b, c, d, e = iunpack(g, 4)
_>>> a, b, c, d, e
_(0, 1, 2, 3, <generator object at 0xa0cd02c>)

_>>> a, b, c, d, e = iunpack(g, 4)
_>>> a, b, c, d, e
_(4, 5, 6, 7, <generator object at 0xa0cd02c>)

_>>> a, b, c, d, e = iunpack(g, 4)
_>>> a, b, c, d, e
_(8, 9, None, None, <generator object at 0xa0cd02c>)

Cheers,
Nick.
I think we're now to the stage of violently agreeing with each other. . .

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From tjreedy at udel.edu  Sat Nov 20 03:53:08 2004
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat Nov 20 03:53:07 2004
Subject: [Python-Dev] Re: print "%X" % id(object()) not so nice
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net>
	<cnm3lj$9nj$1@sea.gmane.org>
Message-ID: <cnmbia$msh$1@sea.gmane.org>


"Terry Reedy" <tjreedy@udel.edu> wrote in message 
news:cnm3lj$9nj$1@sea.gmane.org...
> ...   return (i < 0) and (max - i) or i
> # only semi-tested because I don't have negatives to test and can't force

whoops, untested negative case should be max + i
(or max-abs(i) as I was thinking)



From tjreedy at udel.edu  Sat Nov 20 04:03:15 2004
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat Nov 20 04:03:14 2004
Subject: [Python-Dev] Re: Re: print "%X" % id(object()) not so nice
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net><cnm3lj$9nj$1@sea.gmane.org>
	<D3128502-3A8E-11D9-925A-000A9567635C@redivi.com>
Message-ID: <cnmc59$npt$1@sea.gmane.org>


"Bob Ippolito" <bob@redivi.com> wrote in message 
news:D3128502-3A8E-11D9-925A-000A9567635C@redivi.com...
> The problem, more than anything else, is the following behavior that can 
> happen during a random __repr__ or repr-like-function if the object 
> happens to have a certain address range:
>
> - (Python 2.3) You get an unexpected and unwanted warning but expected 
> output anyway
> - (Python 2.4) You get a repr with a strange looking negative hex number 
> (0x-FF0102)
>
> Neither of these are fatal, of course, it's just annoying.. I find the 
> Python 2.3 behavior more obnoxious than Python 2.4's, personally.

Non-CS users probably find *all* hex numbers a little strange looking.  If 
CPython were to simply print ids as decimal integers, instead of being 
fancy with hex 'addresses' there would have been no warnings and no change 
;-).  Is the absolute hex value ever of any use?  If so, how often?

Terry J. Reedy




From ncoghlan at iinet.net.au  Sat Nov 20 04:41:19 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 20 04:41:28 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized
	list
In-Reply-To: <cnljnp$8r8$1@sea.gmane.org>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME><864d3709041117184370f63c1b@mail.gmail.com><419C8276.5030809@iinet.net.au><864d370904111803561c650864@mail.gmail.com><419DCB19.5000609@iinet.net.au>	<864d37090411190724440dfe38@mail.gmail.com>
	<cnljnp$8r8$1@sea.gmane.org>
Message-ID: <419EBCDF.1070400@iinet.net.au>

Terry Reedy wrote:
> This issue has come up enough that a PEP would be useful (if there isn't 
> one already).

Nope, I had a look. I've got a draft one attached, though - it combines 
rejecting the syntax change with adding the other changes Carlos and I have 
suggested. If people think it's worth pursuing, I'll fire it off to the PEP editors.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
-------------- next part --------------
PEP: XXX
Title: Unpacking sequence elements using tuple assignment
Version: $Revision: 1.4 $
Last-Modified: $Date: 2003/09/22 04:51:50 $
Author: Nick Coghlan <ncoghlan@email.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Nov-2004
Python-Version: 2.5
Post-History: 20-Nov-2004


Abstract
========

This PEP addresses the repeated proposal that "a, b, *c = some_sequence"
be permissible Python syntax.

The PEP does NOT propose that this syntax be allowed. In fact, acceptance
of this PEP will indicate agreement that the above syntax will NOT
become part of Python.

However, this PEP does suggest two additions that will hopefully reduce
the demand for the above syntax:

- modify ``list.pop`` and ``array.pop`` to also accept slice objects as
the index argument

- add an ``itertools.iunpack`` function that accepts an arbitrary iterator,
yields a specified number of elements from it, and then yields the
partially consumed iterator.


Rationale
=========

The proposal to extend the tuple unpacking assignment syntax has now been
posted to python-dev on 3 separate occasions, and rejected on all 3
occasions.[1]_

The subsequent discussion on the most recent occasion yielded the two
suggestions documented in this PEP (modifying ``list.pop`` was suggested by the
PEP author, the ``itertools.iunpack`` function was suggested by Carlos Ribeiro).

Modifying ``list.pop`` to accept slice objects as well as integers brings it
in line with the standard sequence subscripting methods (``__getitem__``,
``__setitem__``, ``__delitem__``). It also makes ``list.pop`` consistent with
its current documentation (the Python code given as equivalent to ``list.pop``
accepts slice objects, but ``list.pop`` does not).

The modification of ``array.pop`` is mainly for consistency with the new
behaviour of ``list.pop``.

The ``itertools.iunpack`` function has the advantage of working for arbitrary
iterators, not just lists or arrays. However, it always yields copies of 
the elements, while the ``pop`` methods are able to avoid unnecessary copying.


Proposed Semantics
==================

``list.pop`` would be updated to conform to its documentation as being
equivalent to::

    x = L[i]
    del L[i]
    return x

In Python 2.4, the above equivalence does not hold if ``i`` is a slice object.
``array.pop`` would be updated similarly.

``itertools.iunpack`` would be equivalent to the following::

    def iunpack(itr, numitems, defaultitem=None):
        for i in range(numitems):
            try:
                yield itr.next()
            except StopIteration:
                yield defaultitem
        yield itr 


Reference Implementation
========================

As yet, no reference implementation is available for either part of the proposal.


Open Issues
===========

- Should ``itertools.iunpack`` call ``iter()`` on its first argument?


References
==========

.. [1] python-dev archives of tuple unpacking discussions
   (http://mail.python.org/pipermail/python-dev/2002-November/030349.html)
   (http://mail.python.org/pipermail/python-dev/2004-August/046684.html)
   (http://mail.python.org/pipermail/python-dev/2004-November/049895.html)


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:
From bob at redivi.com  Sat Nov 20 08:48:15 2004
From: bob at redivi.com (Bob Ippolito)
Date: Sat Nov 20 08:49:01 2004
Subject: [Python-Dev] Re: Re: print "%X" % id(object()) not so nice
In-Reply-To: <cnmc59$npt$1@sea.gmane.org>
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net><cnm3lj$9nj$1@sea.gmane.org>
	<D3128502-3A8E-11D9-925A-000A9567635C@redivi.com>
	<cnmc59$npt$1@sea.gmane.org>
Message-ID: <894CD7B9-3AC8-11D9-925A-000A9567635C@redivi.com>


On Nov 20, 2004, at 5:03 AM, Terry Reedy wrote:

>
> "Bob Ippolito" <bob@redivi.com> wrote in message
> news:D3128502-3A8E-11D9-925A-000A9567635C@redivi.com...
>> The problem, more than anything else, is the following behavior that 
>> can
>> happen during a random __repr__ or repr-like-function if the object
>> happens to have a certain address range:
>>
>> - (Python 2.3) You get an unexpected and unwanted warning but expected
>> output anyway
>> - (Python 2.4) You get a repr with a strange looking negative hex 
>> number
>> (0x-FF0102)
>>
>> Neither of these are fatal, of course, it's just annoying.. I find the
>> Python 2.3 behavior more obnoxious than Python 2.4's, personally.
>
> Non-CS users probably find *all* hex numbers a little strange looking. 
>  If
> CPython were to simply print ids as decimal integers, instead of being
> fancy with hex 'addresses' there would have been no warnings and no 
> change
> ;-).  Is the absolute hex value ever of any use?  If so, how often?

It makes it quite easy to match pdb output with gdb output! :)

-bob

From jason at tishler.net  Sat Nov 20 16:06:37 2004
From: jason at tishler.net (Jason Tishler)
Date: Sat Nov 20 16:05:00 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
In-Reply-To: <419EABE2.9040804@iinet.net.au>
References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net>
	<419EABE2.9040804@iinet.net.au>
Message-ID: <20041120150637.GA3024@tishler.net>

Nick,

On Sat, Nov 20, 2004 at 12:28:50PM +1000, Nick Coghlan wrote:
> Jason Tishler wrote:
> >What Cygwin version are you using?  1.5.12?
> 
> 1.5.11 (also on XP Pro SP1)

Cygwin 1.5.11 has some issues (e.g., pthreads), which is why I skipped
that version... :,)

> I'll try updating all the Cygwin packages and see where that gets me.
> 
> <that didn't take long>

Did you upgrade Cygwin to 1.5.12?

> Now it's hanging in configure (and the m4 interpreter was one of the
> things that got updated). *sigh* I need to wake up more before I dig
> into this.

The above is strange, but the m4 package was just taken over by another
maintainer.

> If you (or anyone else) could check that the unit tests work on Cygwin
> for you, that would make me more certain that this is just a problem
> with my own Cygwin setup, and not related to anything in 2.4c1 that
> wasn't in 2.4b1.

I just upgraded one of my machines to Cygwin 1.5.12, but I did not
upgrade the other packages.  I get the following results:

    failures:

        test_shutil (passed under older Cygwin and/or Python versions)
        test_subprocess (new test)
        test_tcl (new test)

    hangs:

        test_threaded_import (passed under older Cygwin and/or Python versions)

I will investigate the above and report back.

Thanks,
Jason

-- 
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D  8784 1AFD E4CC ECF4 8EF6
From tjreedy at udel.edu  Sat Nov 20 16:12:37 2004
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat Nov 20 16:12:32 2004
Subject: [Python-Dev] Re: Re: Re: print "%X" % id(object()) not so nice
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net><cnm3lj$9nj$1@sea.gmane.org><D3128502-3A8E-11D9-925A-000A9567635C@redivi.com><cnmc59$npt$1@sea.gmane.org>
	<894CD7B9-3AC8-11D9-925A-000A9567635C@redivi.com>
Message-ID: <cnnmsr$vc2$1@sea.gmane.org>


"Bob Ippolito" <bob@redivi.com> wrote in message news:894CD7B9-

> On Nov 20, 2004, at 5:03 AM, Terry Reedy wrote:
>> Is the absolute hex value ever of any use?  If so, how often?

> It makes it quite easy to match pdb output with gdb output! :)

Ah, the missing use case, which you and the OP probably took for granted. 
I, on the other hand, having never used either, find the difference in 
printed ids in

>>> def f(): pass
...
>>> f, id(f)
(<function f at 0x00868158>, 8814936)

at least mildly disturbing.  Do you only need to do such matching for 
complex objects that get the <type name at 0x########> representation?

Terry J. Reedy



From mwh at python.net  Sat Nov 20 16:41:24 2004
From: mwh at python.net (Michael Hudson)
Date: Sat Nov 20 16:41:27 2004
Subject: [Python-Dev] Re: Re: Re: print "%X" % id(object()) not so nice
In-Reply-To: <cnnmsr$vc2$1@sea.gmane.org> (Terry Reedy's message of "Sat, 20
	Nov 2004 10:12:37 -0500")
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net>
	<cnm3lj$9nj$1@sea.gmane.org>
	<D3128502-3A8E-11D9-925A-000A9567635C@redivi.com>
	<cnmc59$npt$1@sea.gmane.org>
	<894CD7B9-3AC8-11D9-925A-000A9567635C@redivi.com>
	<cnnmsr$vc2$1@sea.gmane.org>
Message-ID: <2mekiowmsb.fsf@starship.python.net>

"Terry Reedy" <tjreedy@udel.edu> writes:

> I, on the other hand, having never used either, find the difference in 
> printed ids in
>
>>>> def f(): pass
> ...
>>>> f, id(f)
> (<function f at 0x00868158>, 8814936)
>
> at least mildly disturbing.  Do you only need to do such matching for 
> complex objects that get the <type name at 0x########> representation?

This hardly seems worth discussing :)

It's a pointer.  Pointers are printed in hex.  It's Just The Way It
Is.  I don't know why.  

Actually, the "0x00868158" above is produced by C's %p format
operator.  So, in fact, ANSI C is probably why it is The Way It Is.

Cheers,
mwh

-- 
  Remember - if all you have is an axe, every problem looks 
  like hours of fun.                                        -- Frossie
               -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html
From carribeiro at gmail.com  Sat Nov 20 16:55:38 2004
From: carribeiro at gmail.com (Carlos Ribeiro)
Date: Sat Nov 20 16:55:41 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized
	list
In-Reply-To: <419EBCDF.1070400@iinet.net.au>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>
	<864d3709041117184370f63c1b@mail.gmail.com>
	<419C8276.5030809@iinet.net.au>
	<864d370904111803561c650864@mail.gmail.com>
	<419DCB19.5000609@iinet.net.au>
	<864d37090411190724440dfe38@mail.gmail.com>
	<cnljnp$8r8$1@sea.gmane.org> <419EBCDF.1070400@iinet.net.au>
Message-ID: <864d370904112007554e3720a2@mail.gmail.com>

On Sat, 20 Nov 2004 13:41:19 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote:
> Terry Reedy wrote:
> > This issue has come up enough that a PEP would be useful (if there isn't
> > one already).
> 
> Nope, I had a look. I've got a draft one attached, though - it combines
> rejecting the syntax change with adding the other changes Carlos and I have
> suggested. If people think it's worth pursuing, I'll fire it off to the PEP editors.
 
I may be wrong (and if that's the case, I would like to be politely
educated on why), but isn't it already on a PEP-worthy point? IOW, if
people are not interested, then the PEP will simply be rejected, which
is a good thing, because it will at least document the case. I also
believe that the pre-PEP should be posted to the main Python list,
where it may be beaten to death & flamed by a larger audience. I am
willing to do it myself, but I assume that is more polite on my part
if I ask you to do it :-)

<PEP boilerplate & rationale snipped> 
> Reference Implementation
> ========================
> 
> As yet, no reference implementation is available for either part of the proposal.

I though the iunpack() code in the previous section would be
acceptable as a 'reference implementation'.

> Open Issues
> ===========
> 
> - Should ``itertools.iunpack`` call ``iter()`` on its first argument?

+1
 
> References
> ==========
> 
> .. [1] python-dev archives of tuple unpacking discussions
>    (http://mail.python.org/pipermail/python-dev/2002-November/030349.html)
>    (http://mail.python.org/pipermail/python-dev/2004-August/046684.html)
>    (http://mail.python.org/pipermail/python-dev/2004-November/049895.html)
> 
> Copyright
> =========
> 
> This document has been placed in the public domain.
> 
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    End:
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/carribeiro%40gmail.com
> 
> 
> 


-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro@gmail.com
mail: carribeiro@yahoo.com
From tim.peters at gmail.com  Sat Nov 20 17:21:28 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Sat Nov 20 17:21:31 2004
Subject: [Python-Dev] Re: Re: Re: print "%X" % id(object()) not so nice
In-Reply-To: <2mekiowmsb.fsf@starship.python.net>
References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net>
	<cnm3lj$9nj$1@sea.gmane.org>
	<D3128502-3A8E-11D9-925A-000A9567635C@redivi.com>
	<cnmc59$npt$1@sea.gmane.org>
	<894CD7B9-3AC8-11D9-925A-000A9567635C@redivi.com>
	<cnnmsr$vc2$1@sea.gmane.org> <2mekiowmsb.fsf@starship.python.net>
Message-ID: <1f7befae04112008214f9d810c@mail.gmail.com>

[Terry Reedy]
>> I, on the other hand, having never used either, find the difference in
>> printed ids in
>>
>> >>> def f(): pass
>> >>> f, id(f)
>> (<function f at 0x00868158>, 8814936)
>>
>> at least mildly disturbing.  Do you only need to do such matching for
>> complex objects that get the <type name at 0x########>
>> representation?

[Michael Hudson]
> This hardly seems worth discussing :)

Then it's a topic for me <wink>!

> It's a pointer.  Pointers are printed in hex.  It's Just The Way It
> Is.  I don't know why.
>
> Actually, the "0x00868158" above is produced by C's %p format
> operator.  So, in fact, ANSI C is probably why it is The Way It Is.

repr starts with %p, but %p is ill-defined, so Python goes on to
ensure the result starts with "0x".  C doesn't even say that %p
produces hex digits, but all C systems we know of do(*), so Python
doesn't try to force that part.

As to "why hex?", it's for low-level debugging.  For example, stack,
register and memory dumps for binary machines almost always come in
some power-of-2 base, usually hex, and searching for a stored address
is much easier if it's shown in the same base.

OTOH, id(Q) promises to return an integer that won't be the same as
the id() of any other object over Q's lifetime.  CPython returns Q's
memory address, but CPython never moves objects in memory, so CPython
can get away with returning the address.  Jython does something very
different for id(), because it must -- the Java VM may move an object
in memory.

Python doesn't promise to return a postive integer for id(), although
it may have been nicer if it did.  It's dangerous to change that now,
because some code does depend on the "32 bit-ness as a signed integer"
accident of CPython's id() implementation on 32-bit machines.  For
example, code using struct.pack(), or code using one of ZODB's
specialized int-key BTree types with id's as keys.

Speaking of which, current ZODB has a positive_id() function, used to
format id()'s in strings where a sign bit would get in the way.

(*) The %p in some C's for early x86 systems, using "segment + offset"
mode, stuck a colon "in the middle" of the pointer output, to visually
separate the segment from the offset.  The two parts were still shown
in hex, though.
From ncoghlan at iinet.net.au  Sat Nov 20 17:47:49 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 20 17:48:07 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
In-Reply-To: <20041120150637.GA3024@tishler.net>
References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net>
	<419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net>
Message-ID: <419F7535.9050802@iinet.net.au>

Jason Tishler wrote:
> Did you upgrade Cygwin to 1.5.12?

I did. However, the bash shell was just behaving strangely after that, as was 
the Cygwin installer itself. I tried to do a clean install, and the install 
program is now hanging. So I don't think we can take any error reports from my 
system seriously at this point. Something isn't happy :(

Anyway, I won't be posting any more about this on py-dev, since I've established 
that *my* problems probably aren't Python related.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Sat Nov 20 18:34:34 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 20 18:34:41 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist
In-Reply-To: <001501c4cf25$50aef780$e841fea9@oemcomputer>
References: <001501c4cf25$50aef780$e841fea9@oemcomputer>
Message-ID: <419F802A.5060209@iinet.net.au>

Raymond Hettinger wrote:
> Also, the pep would likely kill the remote, but non-zero chance of ever
> getting the a,b,*c syntax.  IOW, the pep may be a step backwards.

Officially killing that syntax was actually one of the stated goals of the PEP.

However, the PEP editors would be well within their rights to reject the draft 
as 'lacking focus'. I wouldn't be offended given the whole 20 minutes it took to 
write it :)

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Sat Nov 20 18:37:40 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 20 18:37:46 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized
	list
In-Reply-To: <864d370904112007554e3720a2@mail.gmail.com>
References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME>	
	<864d3709041117184370f63c1b@mail.gmail.com>	
	<419C8276.5030809@iinet.net.au>	
	<864d370904111803561c650864@mail.gmail.com>	
	<419DCB19.5000609@iinet.net.au>	
	<864d37090411190724440dfe38@mail.gmail.com>	
	<cnljnp$8r8$1@sea.gmane.org> <419EBCDF.1070400@iinet.net.au>
	<864d370904112007554e3720a2@mail.gmail.com>
Message-ID: <419F80E4.1050009@iinet.net.au>

Carlos Ribeiro wrote:
> I may be wrong (and if that's the case, I would like to be politely
> educated on why), but isn't it already on a PEP-worthy point? IOW, if
> people are not interested, then the PEP will simply be rejected, which
> is a good thing, because it will at least document the case.

Good point - I've sent it to the PEP editors to request a number.

> I also
> believe that the pre-PEP should be posted to the main Python list,
> where it may be beaten to death & flamed by a larger audience. I am
> willing to do it myself, but I assume that is more polite on my part
> if I ask you to do it :-)

I'd wait to see what the PEP editors think, myself. I'm not entirely sure the 
idea is focused well enough to make a good PEP (I could argue either way, and I 
wrote the thing!).

However, if you want to post it over there for discussion, feel free.

>>Reference Implementation
>>========================
>>
>>As yet, no reference implementation is available for either part of the proposal.
> 
> 
> I though the iunpack() code in the previous section would be
> acceptable as a 'reference implementation'.

Unfortunately, itertools is a C module :P

>>Open Issues
>>===========
>>
>>- Should ``itertools.iunpack`` call ``iter()`` on its first argument?
> 
> +1

We'll go with that then. . . y'know, I could have made that change before 
sending the PEP draft in. Ah well, we can change it later - I doubt the PEP will 
surive c.l.p. (or even py-dev) unscathed, anyway.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From carribeiro at gmail.com  Sat Nov 20 19:14:59 2004
From: carribeiro at gmail.com (Carlos Ribeiro)
Date: Sat Nov 20 19:15:03 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist
In-Reply-To: <419F802A.5060209@iinet.net.au>
References: <001501c4cf25$50aef780$e841fea9@oemcomputer>
	<419F802A.5060209@iinet.net.au>
Message-ID: <864d37090411201014cbb0007@mail.gmail.com>

On Sun, 21 Nov 2004 03:34:34 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote:
> Raymond Hettinger wrote:
> > Also, the pep would likely kill the remote, but non-zero chance of ever
> > getting the a,b,*c syntax.  IOW, the pep may be a step backwards.
> 
> Officially killing that syntax was actually one of the stated goals of the PEP.
> 
> However, the PEP editors would be well within their rights to reject the draft
> as 'lacking focus'. I wouldn't be offended given the whole 20 minutes it took to
> write it :)

By itself, both alternatives presented in the implementation do not
"kill" the a,b,*c syntax. I prefer to argue that list.pop(slice) and
iunpack(...) do facilitate future support for it, while at the same
time removing immediate pressure by providing an easy alternative to
support one version of the idiom that still uses current Python
syntax.

Other potential argument against is that the PEP provides two ways to
make it, contrary to the "one way to do it" motto. But both cases,
while related, are really focused on solving two different problems;
one is a generic solution for iterables (which happend to include a
whole bunch of objects, including tuples and lists), and the other is
an optimization for mutable lists. There's no reason not to support
both (of course, if the whole point of the PEP is accepted).

-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro@gmail.com
mail: carribeiro@yahoo.com
From ncoghlan at iinet.net.au  Sat Nov 20 19:20:24 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sat Nov 20 19:20:33 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist
In-Reply-To: <001601c4cf28$7e328ca0$e841fea9@oemcomputer>
References: <001601c4cf28$7e328ca0$e841fea9@oemcomputer>
Message-ID: <419F8AE8.3010807@iinet.net.au>

Raymond Hettinger wrote:
>>Officially killing that syntax was actually one of the stated goals of
> 
> the
> 
>>PEP.
> 
> I don't think that is a worthy goal.  The is some merit to the a,b,*c
> syntax.  When the time is ripe, it should have its day in court.

There's currently 3 things in the PEP draft:
1. Allow slice objects as arguments list.pop and array.pop
2. Add itertools.iunpack
3. Record the rejection of a, b, *c = L in a PEP (since it's been rejected three 
times now - once in 2001, and twice this year)

The first of those could probably be done without a PEP (given that a strict 
reading of the documentation suggests that those methods *should* handle slices)

The 2nd can certainly be separated out. Since I now know that, for actual 
iterators (rather than iterables), islice(n) will usually consume the first n 
elements, it loses some of its appeal (i.e. islice already has most of the 
features I like about iunpack).

The 3rd - well, I thought Guido's rejection in August suggested his position was 
unlikely to change (the AST branch going in for 2.5 doesn't seem to be 
significant enough to prompt a change of heart, and I can't think of anything 
else likely to affect his stated reasons for rejection).

I have no great problem with leaving the a, b, *c = L idea unPEPed if there is 
interest in leaving the idea 'open', though.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ad at papyrus-gmbh.de  Sat Nov 20 21:42:53 2004
From: ad at papyrus-gmbh.de (Andreas Degert)
Date: Sat Nov 20 21:42:55 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
Message-ID: <87pt289rqq.fsf@pluto.noname>

Hello,

I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used
with 2 unicode strings, it converts them to wchar strings and uses
wcscoll. The bug is that the wchar strings are not 0-terminated.
Checking with

assert(ws1[len1-1] == 0 && ws2[len2-1] == 0);

right before the line

result = PyInt_FromLong(wcscoll(ws1, ws2));

confirms the bug. I'm not quite sure what the best fix is.

PyUnicode_AsWideChar() copies the unicode chars, but not the
terminating 0-char of the unicode string (which is not used in python,
but its there anyhow, if I understand the implementation
correctly). So one fix would be to change PyUnicode_AsWideChar to copy
the terminating 0-char if there's enough space in the output
buffer. Another fix would be to terminate the strings in
PyLocale_strcoll() before using them:

----------------------------------------------------------
--- _localemodule.c~	Sat Nov 20 21:33:17 2004
+++ _localemodule.c	Sat Nov 20 21:35:04 2004
@@ -353,15 +353,19 @@
         PyErr_NoMemory();
         goto done;
     }
-    if (PyUnicode_AsWideChar((PyUnicodeObject*)os1, ws1, len1) == -1)
+    len1 = PyUnicode_AsWideChar((PyUnicodeObject*)os1, ws1, len1);
+    if (len1 == -1)
         goto done;
+    ws1[len1-1] = 0;
     ws2 = PyMem_MALLOC(len2 * sizeof(wchar_t));
     if (!ws2) {
         PyErr_NoMemory();
         goto done;
     }
-    if (PyUnicode_AsWideChar((PyUnicodeObject*)os2, ws2, len2) == -1)
+    len2 = PyUnicode_AsWideChar((PyUnicodeObject*)os2, ws2, len2);
+    if (len2 == -1)
         goto done;
+    ws2[len2-1] = 0;
     /* Collate the strings. */
     result = PyInt_FromLong(wcscoll(ws1, ws2));
   done:
----------------------------------------------------------

cheers
Andreas
From ad at papyrus-gmbh.de  Sat Nov 20 22:05:59 2004
From: ad at papyrus-gmbh.de (Andreas Degert)
Date: Sat Nov 20 22:06:01 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
In-Reply-To: <87pt289rqq.fsf@pluto.noname> (Andreas Degert's message of
	"Sat, 20 Nov 2004 21:42:53 +0100")
References: <87pt289rqq.fsf@pluto.noname>
Message-ID: <87hdnk9qo8.fsf@pluto.noname>

sorry to reply to my own post.. corrected patch following

--- _localemodule.c-2.49	Sat Nov 20 22:02:11 2004
+++ _localemodule.c	Sat Nov 20 22:01:48 2004
@@ -313,6 +313,7 @@
     }
     if (PyUnicode_AsWideChar((PyUnicodeObject*)os1, ws1, len1) == -1)
         goto done;
+    ws1[len1-1] = 0;
     ws2 = PyMem_MALLOC(len2 * sizeof(wchar_t));
     if (!ws2) {
         PyErr_NoMemory();
@@ -320,6 +321,7 @@
     }
     if (PyUnicode_AsWideChar((PyUnicodeObject*)os2, ws2, len2) == -1)
         goto done;
+    ws2[len2-1] = 0;
     /* Collate the strings. */
     result = PyInt_FromLong(wcscoll(ws1, ws2));
   done:
From aahz at pythoncraft.com  Sun Nov 21 00:07:12 2004
From: aahz at pythoncraft.com (Aahz)
Date: Sun Nov 21 00:07:15 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
In-Reply-To: <87pt289rqq.fsf@pluto.noname>
References: <87pt289rqq.fsf@pluto.noname>
Message-ID: <20041120230712.GA801@panix.com>

On Sat, Nov 20, 2004, Andreas Degert wrote:
> 
> I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used
> with 2 unicode strings, it converts them to wchar strings and uses
> wcscoll. The bug is that the wchar strings are not 0-terminated.

If you're sure this is a bug, please file on SF and report back the ID.
(If you're not sure, what until you get confirmation from one of the
Unicode experts and then file the bug. ;-)
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

WiFi is the SCSI of the 21st Century -- there are fundamental technical
reasons for sacrificing a goat.  (with no apologies to John Woods)
From ncoghlan at iinet.net.au  Sun Nov 21 04:43:38 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sun Nov 21 04:43:44 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist
In-Reply-To: <000001c4cf33$b5a83d00$e841fea9@oemcomputer>
References: <000001c4cf33$b5a83d00$e841fea9@oemcomputer>
Message-ID: <41A00EEA.4030704@iinet.net.au>

Raymond Hettinger wrote:
>>I have no great problem with leaving the a, b, *c = L idea unPEPed if
>>there is
>>interest in leaving the idea 'open', though.
> 
> 
> Yes.  I would like to have it left open.

OK, here's what I'll do in the next draft of the proposed PEP (PEPP?):

1. Add a discussion of the advantages of iunpack over islice:
   - guaranteed minimum length (handy for use with tuple unpacking)
   - return the iterator as the final object (this is nicer when the original 
object is an iterable, rather than an iterator - e.g. a list or tuple)

2. Summarise the use cases as given by the previous proponents of the "a, b, *c 
= S" syntax.

3. Soften the PEP's language regarding the "a, b *c = S" syntax.
   Point out that this is a way to get the functionality in a fairly easy to 
spell form for 2.5 that doesn't require changing Python's syntax.
   itertools.iunpack could serve as a reference for defining the behaviour of
that syntax at a later date.

I find the following to be rather readable:

   a, b, itr = iunpack(S, 2)

"iunpack" is also easy to look up in the documentation index.

If Carlos is amenable, I'll add him as a co-author on the PEP - after all, he's 
the one who sold *me* on the idea of iunpack, and most of the points defending 
it are originally his.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From skip at pobox.com  Sun Nov 21 05:49:20 2004
From: skip at pobox.com (Skip Montanaro)
Date: Sun Nov 21 05:47:08 2004
Subject: [Python-Dev] enhanced gdbinit file - looking for Python+Emacs+gdb
	users
Message-ID: <16800.7760.739724.695902@montanaro.dyndns.org>


I got a little carried away with the gdbinit file that lives in the Python
distribution at Misc/gdbinit today.  I decided there's no particular reason
that gdb shouldn't display the current Python file when moving up and down
the C stack any time it encounters PyEval_EvalFrame.  After a little digging
I realized this would be pretty easy.  I made two changes to the gdbinit
file:

    1. I wrote a lineno command for gdb that calculates and prints the
       current Python line number.  (It's PyCode_Addr2Line written in gdb's
       command language.)

    2. Using that and the current file gleaned from PyEval_EvalFrame I emit
       the file:line info in the correct format for Emacs to display the
       Python line number at the right time using overridden up and down
       commands.

There's one slight hitch.  It works perfectly when travelling up the C stack
using the up command, however in my environment at least (MacOSX w/ Apple's
gcc and gdb) it causes gdb to segfault when travelling down the stack.  I
can't see anything that would make the behavior different going up instead
of down.  I need some people to try this thing out to see if it barfs for
them as well.  Maybe it's just something peculiar to my environment.  OTOH,
perhaps it's a bug in gdb.

Since I can't really check this beast in given its current rather dicey
behavior, I dropped a copy at

    http://www.musi-cal.com/~skip/python/gdbinit

If you use Python with gdb via Emacs please give it a whirl and let me know
how it works for you.

Thanks,

Skip

P.S.  I think I know how to make the gdb print command print Python objects
as well...
From martin at v.loewis.de  Sun Nov 21 10:03:51 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun Nov 21 10:03:58 2004
Subject: [Python-Dev] enhanced gdbinit file - looking for Python+Emacs+gdb
	users
In-Reply-To: <16800.7760.739724.695902@montanaro.dyndns.org>
References: <16800.7760.739724.695902@montanaro.dyndns.org>
Message-ID: <41A059F7.4080909@v.loewis.de>

Skip Montanaro wrote:
> There's one slight hitch.  It works perfectly when travelling up the C stack
> using the up command, however in my environment at least (MacOSX w/ Apple's
> gcc and gdb) it causes gdb to segfault when travelling down the stack.  I
> can't see anything that would make the behavior different going up instead
> of down.  I need some people to try this thing out to see if it barfs for
> them as well.  Maybe it's just something peculiar to my environment.  OTOH,
> perhaps it's a bug in gdb.

Without actually looking at your code: a segfault in gdb is always a
bug, even if triggered through gdb macros.

Regards,
Martin
From mal at egenix.com  Sun Nov 21 15:32:30 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Sun Nov 21 15:32:31 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
In-Reply-To: <20041120230712.GA801@panix.com>
References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com>
Message-ID: <41A0A6FE.10203@egenix.com>

Aahz wrote:
> On Sat, Nov 20, 2004, Andreas Degert wrote:
> 
>>I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used
>>with 2 unicode strings, it converts them to wchar strings and uses
>>wcscoll. The bug is that the wchar strings are not 0-terminated.
> 
> 
> If you're sure this is a bug, please file on SF and report back the ID.
> (If you're not sure, what until you get confirmation from one of the
> Unicode experts and then file the bug. ;-)

Please also check that the bug is still present in Python 2.4 and/or
CVS. We've corrected a bug in the PyUnicode_*WideChar*() APIs just
recently for Python 2.4.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 21 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From ad at papyrus-gmbh.de  Sun Nov 21 23:22:02 2004
From: ad at papyrus-gmbh.de (Andreas Degert)
Date: Sun Nov 21 23:22:04 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
In-Reply-To: <41A0A6FE.10203@egenix.com> (M.'s message of "Sun, 21 Nov 2004
	15:32:30 +0100")
References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com>
	<41A0A6FE.10203@egenix.com>
Message-ID: <871xemalmd.fsf@pluto.noname>

"M.-A. Lemburg" <mal@egenix.com> writes:

> Aahz wrote:
>> On Sat, Nov 20, 2004, Andreas Degert wrote:
>>
>>>I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used
>>>with 2 unicode strings, it converts them to wchar strings and uses
>>>wcscoll. The bug is that the wchar strings are not 0-terminated.
>> If you're sure this is a bug, please file on SF and report back the
>> ID.
>> (If you're not sure, what until you get confirmation from one of the
>> Unicode experts and then file the bug. ;-)
>
> Please also check that the bug is still present in Python 2.4 and/or
> CVS. We've corrected a bug in the PyUnicode_*WideChar*() APIs just
> recently for Python 2.4.

The off-by-one error fix in unicodeobject.c (2.228 -> 2.229) is
correcting a buffer overflow, is just in the same piece of code.

I didn't find a clear statement if the unicode string should be
0-terminated or not. In _PyUnicode_New it's 0-terminated, even in the
case when it had to call unicode_resize (though there is a comment in
unicode_resize "Ux0000 terminated -- XXX is this needed ?"). If these
is the only place where unicode objects are created or modified, they
seem to be always 0-terminated.

wchar strings must be 0-terminated if they are to be used with the
wcs* functions. So it's not a good idea to return a non-terminated
string from PyUnicode_AsWideChar. If the unicode strings are always
0-terminated (the unicode buffer size is length+1), then we could just
change

    if (size > PyUnicode_GET_SIZE(unicode))
	size = PyUnicode_GET_SIZE(unicode);

to 

    if (size > PyUnicode_GET_SIZE(unicode)+1)
	size = PyUnicode_GET_SIZE(unicode)+1;

in PyUnicode_AsWideChar to get 0-terminated wchars.

Ok... I'm still not sure if I should file a bug for PyLocale_strcoll
or PyUnicode_AsWideChar and if the patch for the latter should assume
that the unicode string buffer is 0-terminated...

cheers
Andreas
From fdrake at acm.org  Mon Nov 22 05:58:38 2004
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Mon Nov 22 05:58:49 2004
Subject: [Python-Dev] TRUNK UNFROZEN
In-Reply-To: <200411192203.52490.anthony@interlink.com.au>
References: <200411192203.52490.anthony@interlink.com.au>
Message-ID: <200411212358.38022.fdrake@acm.org>

On Friday 19 November 2004 06:03 am, Anthony Baxter wrote:
 > Thanks to Tim's latest discovery, it looks like we might need an rc2 -
 > I might try to aim for this in a week's time (assuming that's ok with
 > Fred and Martin) and try to keep things moving along to 2.4 final.

Works for me.

 > After which, I'm going to go out and get very drunk <wink>

Might wanna join you for that as well.  ;-)


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>

From oliver.walczak at momatec.de  Mon Nov 22 08:29:17 2004
From: oliver.walczak at momatec.de (Oliver Walczak)
Date: Mon Nov 22 08:29:19 2004
Subject: [Python-Dev] Constructor bug
Message-ID: <FB26DAD89E43D411B25F0050DA517F382E4657@MOMATEC01>

Dear List,

recently i got stuck on a strange class initialization bug. Please refer the
sample attached for a demonstration and watch what happens to d2 in the
second class instance.
This only works on dictionaries. Any other types initialized outside of
__init__ seem to work perfectly.

So, is it a bug or a feature? In my eyes this behaviour is not what the
programmer has got to expect. Who knows whats happening there and can
explain me if it makes any sense?

Best regards
Oliver
-------------- next part --------------
#!/usr/bin/python
# -*- coding: iso-8859-1 -*-

class test:
    d1 = {'key':'value'}
    def __init__(self):
        self.d2 = {'key':'value'}
    
    def Show(self):
        print 'd1=',self.d1
        print 'd2=',self.d2

c1 = test()
print 'c1, before:'
c1.Show()
c1.d1['key'] = 'value2'
c1.d2['key'] = 'value2'
print 'c1, after:'
c1.Show()

c2 = test()
print 'c2:'
c2.Show()
From mal at egenix.com  Mon Nov 22 09:17:44 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon Nov 22 09:17:40 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
In-Reply-To: <871xemalmd.fsf@pluto.noname>
References: <87pt289rqq.fsf@pluto.noname>
	<20041120230712.GA801@panix.com>	<41A0A6FE.10203@egenix.com>
	<871xemalmd.fsf@pluto.noname>
Message-ID: <41A1A0A8.8010803@egenix.com>

Andreas Degert wrote:
> "M.-A. Lemburg" <mal@egenix.com> writes:
> 
> 
>>Aahz wrote:
>>
>>>On Sat, Nov 20, 2004, Andreas Degert wrote:
>>>
>>>
>>>>I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used
>>>>with 2 unicode strings, it converts them to wchar strings and uses
>>>>wcscoll. The bug is that the wchar strings are not 0-terminated.
>>>
>>>If you're sure this is a bug, please file on SF and report back the
>>>ID.
>>>(If you're not sure, what until you get confirmation from one of the
>>>Unicode experts and then file the bug. ;-)
>>
>>Please also check that the bug is still present in Python 2.4 and/or
>>CVS. We've corrected a bug in the PyUnicode_*WideChar*() APIs just
>>recently for Python 2.4.
> 
> 
> The off-by-one error fix in unicodeobject.c (2.228 -> 2.229) is
> correcting a buffer overflow, is just in the same piece of code.
> 
> I didn't find a clear statement if the unicode string should be
> 0-terminated or not.

You're right: they are always 0-terminated just like 8-bit strings
and even though it doesn't seem to be necessary since Python
functions will always use the size field when working on
a Unicode object rather than rely on the 0-termination.

> In _PyUnicode_New it's 0-terminated, even in the
> case when it had to call unicode_resize (though there is a comment in
> unicode_resize "Ux0000 terminated -- XXX is this needed ?"). If these
> is the only place where unicode objects are created or modified, they
> seem to be always 0-terminated.

Right.

> wchar strings must be 0-terminated if they are to be used with the
> wcs* functions. So it's not a good idea to return a non-terminated
> string from PyUnicode_AsWideChar. If the unicode strings are always
> 0-terminated (the unicode buffer size is length+1), then we could just
> change
> 
>     if (size > PyUnicode_GET_SIZE(unicode))
> 	size = PyUnicode_GET_SIZE(unicode);
> 
> to 
> 
>     if (size > PyUnicode_GET_SIZE(unicode)+1)
> 	size = PyUnicode_GET_SIZE(unicode)+1;
> 
> in PyUnicode_AsWideChar to get 0-terminated wchars.
> 
> Ok... I'm still not sure if I should file a bug for PyLocale_strcoll
> or PyUnicode_AsWideChar and if the patch for the latter should assume
> that the unicode string buffer is 0-terminated...

I think it's probably wise to fix both:

Looking again, the patch we applied to PyUnicode_AsWideChar()
only fixes the 0-termination problem in the case where
HAVE_USABLE_WCHAR_T is set. This should be extended to
the memcpy() as well.

Still, if the buffer passed to PyUnicode_AsWideChar()
is not big enough, you won't get the 0-termination (due
to truncation), so PyLocale_strcoll() must be either very
careful to allocate a buffer that is always big enough
or apply 0-termination itself.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 22 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From ncoghlan at iinet.net.au  Mon Nov 22 10:18:20 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Mon Nov 22 10:18:31 2004
Subject: [Python-Dev] Constructor bug
In-Reply-To: <FB26DAD89E43D411B25F0050DA517F382E4657@MOMATEC01>
References: <FB26DAD89E43D411B25F0050DA517F382E4657@MOMATEC01>
Message-ID: <41A1AEDC.3070105@iinet.net.au>

Oliver Walczak wrote:
> So, is it a bug or a feature? In my eyes this behaviour is not what the
> programmer has got to expect. Who knows whats happening there and can
> explain me if it makes any sense?

The output is exactly what I expected:

c1, before:
d1= {'key': 'value'}
d2= {'key': 'value'}
c1, after:
d1= {'key': 'value2'}
d2= {'key': 'value2'}
c2:
d1= {'key': 'value2'}
d2= {'key': 'value'}

Exactly what do you believe is mysterious here? c2 sees the modified version of 
the class variable (d1), and sees a fresh copy of the instance variable (d2).

Please make you question more explicit.

Regards,
Nick.



-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Mon Nov 22 10:20:02 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Mon Nov 22 10:20:06 2004
Subject: [Python-Dev] TRUNK UNFROZEN
In-Reply-To: <200411212358.38022.fdrake@acm.org>
References: <200411192203.52490.anthony@interlink.com.au>
	<200411212358.38022.fdrake@acm.org>
Message-ID: <41A1AF42.2090908@iinet.net.au>

Fred L. Drake, Jr. wrote:
>  > After which, I'm going to go out and get very drunk <wink>
> 
> Might wanna join you for that as well.  ;-)

That's a long trip just for a drink!

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ncoghlan at iinet.net.au  Mon Nov 22 11:36:11 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Mon Nov 22 11:36:18 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist
In-Reply-To: <004e01c4cf83$0166db80$e841fea9@oemcomputer>
References: <004e01c4cf83$0166db80$e841fea9@oemcomputer>
Message-ID: <41A1C11B.4020808@iinet.net.au>

Raymond Hettinger wrote:
> Also, if the goal is just to get iunpack(), the good news is that PEP is
> not required.  Only the a,b,*c syntax would need a PEP.  The bad news is
> that I have the final say over what goes into itertools (I've rejected
> eight proposals so far, including one from Guido).

Actually, the goal (or my goal, anyway) was to avoid having the "a, b, *c = seq" 
idea proposed again in a few months time. Google searches of the py-dev archive 
can work sometimes, but a PEP is generally more visible.

However, if you follow through the rambling below, you'll find it doesn't leave 
much for a PEP to say. . . so, unless Carlos or anyone else wants to run with 
it, consider my PEPP from this thread to be dead :)

Cheers,
Nick.

<Rambling summary of the discussion so far>
<I included it, since it is what lead me to the second paragraph above>

The basic idea of the "a, b, *c = seq" syntax is to unpack the first few items 
of a sequence into specific variables, and then drop the rest of the sequence 
into another variable (sometimes the original variable).

To me, the presented use cases are nowhere near significant enough to justify 
special syntax. The other argument in favour (symmetry with extended function 
call syntax) isn't compelling either, since function calls and assignment 
statements are, well, different.

However, while I didn't find the use cases presented enough to justify a syntax 
change, I wondered if they pointed towards some potentially missing features. 
The current PEP draft grew out of the question "Is there a way to provide the 
_functionality_ without altering Python's syntax?".

For objects that support slicing, the following works now:

   a, b, the_rest = seq[:2] + [seq[2:]]

For iterators, roughly equivalent spelling is:

   a, b, the_rest = [islice(itr, 2)] + [itr]

Carlos's suggestion gives a common way to spell it for any iterable:

   a, b, the_rest = iunpack(iterable, 2)

Alternatively, if we stick with different spellings for mutable sequences and 
iterators, we can get to:

   a, b = seq.pop(slice(2))
   a, b = islice(itr, 2)

In both of these cases, the original object (seq, itr) ends up containing "the 
rest". Neither approach works for an immutable sequence, though - for those, it 
is still necessary to use the explicit slicing spelling above (or create an 
iterator and use the islice approach).

Anyway, if you're firmly opposed to itertools.iunpack, there isn't much point in 
pursuing it (since, in the end, your opinion is the one that really counts). 
Carlos may choose to post it as a recipe over at ASPN.

That still leaves adding slice handling to list.pop and array.pop. If nobody 
else steps forward to do it, I expect I'll eventually get around to coming up 
with a patch for that.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From ad at papyrus-gmbh.de  Mon Nov 22 12:46:01 2004
From: ad at papyrus-gmbh.de (Andreas Degert)
Date: Mon Nov 22 12:46:04 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
In-Reply-To: <41A1A0A8.8010803@egenix.com> (M.'s message of "Mon, 22 Nov
	2004 09:17:44 +0100")
References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com>
	<41A0A6FE.10203@egenix.com> <871xemalmd.fsf@pluto.noname>
	<41A1A0A8.8010803@egenix.com>
Message-ID: <87sm7285ty.fsf@pluto.noname>

"M.-A. Lemburg" <mal@egenix.com> writes:

> You're right: they are always 0-terminated just like 8-bit strings
> and even though it doesn't seem to be necessary since Python
> functions will always use the size field when working on
> a Unicode object rather than rely on the 0-termination.

OK, should be documented in the code

>> Ok... I'm still not sure if I should file a bug for PyLocale_strcoll
>> or PyUnicode_AsWideChar and if the patch for the latter should assume
>> that the unicode string buffer is 0-terminated...
>
> I think it's probably wise to fix both:
>
> Looking again, the patch we applied to PyUnicode_AsWideChar()
> only fixes the 0-termination problem in the case where
> HAVE_USABLE_WCHAR_T is set. This should be extended to
> the memcpy() as well.

What I read from the code is that now in both cases the string is
copied without 0 and that is consistent with the size the buffer is
checked for (PyUnicode_GET_SIZE gives the value of the length field
and that doesn't include the 0-termination)

> Still, if the buffer passed to PyUnicode_AsWideChar()
> is not big enough, you won't get the 0-termination (due
> to truncation), so PyLocale_strcoll() must be either very
> careful to allocate a buffer that is always big enough
> or apply 0-termination itself.

PyLocale_strcoll() acts quite careful but even so it didn't get what
it expected ;-). This bug is masked by the bug you referred to when
the copy loop is used (ie. if wchar sizes don't match) and the output
buffer string is big enough (like in the strcoll case because the
buffer size already accounts for the 0-termination).

I appended a (untested) patch for unicodeobject.c.

The documentation should be clarified too. Would a patch against
concrete.tex be accepted where I change

- 'Unicode object' to 'Unicode string' when only the string part of
  the python object is referenced,

- 'size of the object' to 'length of the string'

- mention the 0-termination of the return-value of
  PyUnicode_AS_UNICODE()

- mention the 0-termination of the return-value of
  PyUnicode_AsWideChar

- '... represents a 16-bit...' to something that explains 16 vs. 32
  but depending on internal representation (UCS-2 or UCS-4) selected at
  compile time

--- unicodeobject.c-2.229	Mon Nov 22 10:58:42 2004
+++ unicodeobject.c	Mon Nov 22 11:10:07 2004
@@ -144,8 +144,7 @@
         return -1;
     }
 
-    /* We allocate one more byte to make sure the string is
-       Ux0000 terminated -- XXX is this needed ? */
+    /* We allocate one more Py_UNICODE and make the string Ux0000 terminated */
     oldstr = unicode->str;
     PyMem_RESIZE(unicode->str, Py_UNICODE, length + 1);
     if (!unicode->str) {
@@ -167,8 +166,7 @@
     return 0;
 }
 
-/* We allocate one more byte to make sure the string is
-   Ux0000 terminated -- XXX is this needed ?
+/* We allocate length+1 and make the string Ux0000 terminated
 
    XXX This allocator could further be enhanced by assuring that the
        free list never reduces its size below 1.
@@ -384,8 +382,10 @@
 	PyErr_BadInternalCall();
 	return -1;
     }
-    if (size > PyUnicode_GET_SIZE(unicode))
-	size = PyUnicode_GET_SIZE(unicode);
+    /* copy all chars including 0-termination if the output buffer
+       size is sufficient */
+    if (size > PyUnicode_GET_SIZE(unicode)+1)
+	size = PyUnicode_GET_SIZE(unicode)+1;
 #ifdef HAVE_USABLE_WCHAR_T
     memcpy(w, unicode->str, size * sizeof(wchar_t));
 #else
From ncoghlan at iinet.net.au  Mon Nov 22 12:52:24 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Mon Nov 22 12:52:28 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
In-Reply-To: <419F7535.9050802@iinet.net.au>
References: <419E0B4E.7040109@iinet.net.au>
	<20041119232733.GB1680@tishler.net>	<419EABE2.9040804@iinet.net.au>
	<20041120150637.GA3024@tishler.net> <419F7535.9050802@iinet.net.au>
Message-ID: <41A1D2F8.9040900@iinet.net.au>

Nick Coghlan wrote:
> Anyway, I won't be posting any more about this on py-dev, since I've 
> established that *my* problems probably aren't Python related.

I lied. The *real* final word - some digging with Google shows that there are 
still some issues with Cygwin and hyperthreaded processors. Sure enough, my 
desktop is a hyper-threaded P4.

Just a minor detail the "Known Problems" section in the Cygwin FAQ neglects to 
mention. . .

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From oliver.walczak at momatec.de  Mon Nov 22 13:20:00 2004
From: oliver.walczak at momatec.de (Oliver Walczak)
Date: Mon Nov 22 13:20:01 2004
Subject: AW: [Python-Dev] Constructor bug
In-Reply-To: <FB26DAD89E43D411B25F0050DA517F38365308@MOMATEC01>
Message-ID: <FB26DAD89E43D411B25F0050DA517F382E4659@MOMATEC01>

Dear Nick, dear Andreas,
Imagine you add two ints i1 and i2, one in and one outside of __init__, none
of them gets shared to other instances of test. They are always initialized
to what the class definition says to. Please refer the modified example.
In your sense, Andreas, c2 should also see a modified version of i1. And in
yours, Nick, i1 should be a unique int shared by all instances of test.
That's not true actually.
In my opinion, initialization outside of __init__ without self. and inside
__init__ with self. should work absolutely identical.
Best regards
Oliver



Oliver Walczak wrote:
> So, is it a bug or a feature? In my eyes this behaviour is not what the
> programmer has got to expect. Who knows whats happening there and can
> explain me if it makes any sense?

The output is exactly what I expected:

c1, before:
d1= {'key': 'value'}
d2= {'key': 'value'}
c1, after:
d1= {'key': 'value2'}
d2= {'key': 'value2'}
c2:
d1= {'key': 'value2'}
d2= {'key': 'value'}

Exactly what do you believe is mysterious here? c2 sees the modified version
of 
the class variable (d1), and sees a fresh copy of the instance variable
(d2).

Please make you question more explicit.

Regards,
Nick.
"Oliver Walczak" <oliver.walczak@momatec.de> writes:

> Dear List,
>
> recently i got stuck on a strange class initialization bug. Please refer
the
> sample attached for a demonstration and watch what happens to d2 in the
> second class instance.
> This only works on dictionaries. Any other types initialized outside of
> __init__ seem to work perfectly.

Did you try with type list? You modify the object referenced by d1,
and that object is created when the class is defined. All instances of
test share the dictionary referenced by d2, and have a unique
dictionary referenced by d1. That what your class definition says, so
I'd say it's a feature not a bug.

>
> So, is it a bug or a feature? In my eyes this behaviour is not what the
> programmer has got to expect. Who knows whats happening there and can
> explain me if it makes any sense?
>
> Best regards
> Oliver
>
> #!/usr/bin/python
> # -*- coding: iso-8859-1 -*-
>
> class test:
>     d1 = {'key':'value'}
>     def __init__(self):
>         self.d2 = {'key':'value'}
>     
>     def Show(self):
>         print 'd1=',self.d1
>         print 'd2=',self.d2
>
> c1 = test()
> print 'c1, before:'
> c1.Show()
> c1.d1['key'] = 'value2'
> c1.d2['key'] = 'value2'
> print 'c1, after:'
> c1.Show()
>
> c2 = test()
> print 'c2:'
> c2.Show()
-------------- next part --------------
#!/usr/bin/python
# -*- coding: iso-8859-1 -*-

class test:
    d1 = {'key':'value'}
    i1 = 1
    def __init__(self):
        self.d2 = {'key':'value'}
        self.i2 = 1
    
    def Show(self):
        from pprint import pprint
        print 'd1=',
        pprint(self.d1)
        print 'd2=',
        pprint(self.d2)
        print 'i1=',self.i1
        print 'i2=',self.i2

c1 = test()
print 'c1, before:'
c1.Show()
c1.d1['key'] = 'value2'
c1.d2['key'] = 'value2'
c1.i1        = 2
c1.i2        = 2
print 'c1, after:'
c1.Show()

c2 = test()
print 'c2:'
c2.Show()
From fdrake at acm.org  Mon Nov 22 13:21:42 2004
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Mon Nov 22 13:21:57 2004
Subject: [Python-Dev] TRUNK UNFROZEN
In-Reply-To: <41A1AF42.2090908@iinet.net.au>
References: <200411192203.52490.anthony@interlink.com.au>
	<200411212358.38022.fdrake@acm.org> <41A1AF42.2090908@iinet.net.au>
Message-ID: <200411220721.42132.fdrake@acm.org>

On Monday 22 November 2004 04:20 am, Nick Coghlan wrote:
 > That's a long trip just for a drink!

Yeah, well, it's unusual enough that I have more than one or two beers that it 
seems somehow... fitting.  Of course, it may be a trip made "in spirit."  ;-)


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>

From ncoghlan at iinet.net.au  Mon Nov 22 13:39:47 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Mon Nov 22 13:39:53 2004
Subject: AW: [Python-Dev] Constructor bug
In-Reply-To: <FB26DAD89E43D411B25F0050DA517F382E4659@MOMATEC01>
References: <FB26DAD89E43D411B25F0050DA517F382E4659@MOMATEC01>
Message-ID: <41A1DE13.7090402@iinet.net.au>

Short answer for py-dev:

This is not a bug. It is how the language works.
I'll give some further explanation in private mail - any additional queries 
should go to comp.lang.python.

Cheers,
Nick.

-- 
Nick Coghlan               |     Brisbane, Australia
Email: ncoghlan@email.com  | Mobile: +61 409 573 268
From mal at egenix.com  Mon Nov 22 14:03:23 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon Nov 22 14:03:29 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
In-Reply-To: <87sm7285ty.fsf@pluto.noname>
References: <87pt289rqq.fsf@pluto.noname>
	<20041120230712.GA801@panix.com>	<41A0A6FE.10203@egenix.com>
	<871xemalmd.fsf@pluto.noname>	<41A1A0A8.8010803@egenix.com>
	<87sm7285ty.fsf@pluto.noname>
Message-ID: <41A1E39B.3030308@egenix.com>

Andreas Degert wrote:
> "M.-A. Lemburg" <mal@egenix.com> writes:
> 
> 
>>You're right: they are always 0-terminated just like 8-bit strings
>>and even though it doesn't seem to be necessary since Python
>>functions will always use the size field when working on
>>a Unicode object rather than rely on the 0-termination.
> 
> 
> OK, should be documented in the code

It is, but I wasn't sure whether it is really such a good
idea to waist the extra memory and wanted to keep the option
of removing the 0-termination.

>>>Ok... I'm still not sure if I should file a bug for PyLocale_strcoll
>>>or PyUnicode_AsWideChar and if the patch for the latter should assume
>>>that the unicode string buffer is 0-terminated...
>>
>>I think it's probably wise to fix both:
>>
>>Looking again, the patch we applied to PyUnicode_AsWideChar()
>>only fixes the 0-termination problem in the case where
>>HAVE_USABLE_WCHAR_T is set. This should be extended to
>>the memcpy() as well.
> 
> 
> What I read from the code is that now in both cases the string is
> copied without 0 and that is consistent with the size the buffer is
> checked for (PyUnicode_GET_SIZE gives the value of the length field
> and that doesn't include the 0-termination)
> 
> 
>>Still, if the buffer passed to PyUnicode_AsWideChar()
>>is not big enough, you won't get the 0-termination (due
>>to truncation), so PyLocale_strcoll() must be either very
>>careful to allocate a buffer that is always big enough
>>or apply 0-termination itself.
> 
> 
> PyLocale_strcoll() acts quite careful but even so it didn't get what
> it expected ;-). This bug is masked by the bug you referred to when
> the copy loop is used (ie. if wchar sizes don't match) and the output
> buffer string is big enough (like in the strcoll case because the
> buffer size already accounts for the 0-termination).
> 
> I appended a (untested) patch for unicodeobject.c.

I've just checked in a patch which should correct the
problem.

> The documentation should be clarified too. Would a patch against
> concrete.tex be accepted where I change
> 
> - 'Unicode object' to 'Unicode string' when only the string part of
>   the python object is referenced,

Not sure what you mean here.

> - 'size of the object' to 'length of the string'

Dito.

> - mention the 0-termination of the return-value of
>   PyUnicode_AS_UNICODE()
> 
> - mention the 0-termination of the return-value of
>   PyUnicode_AsWideChar

I don't think we should document this. Programmers should always
use the size of the object rather than rely on the 0-termination.

> - '... represents a 16-bit...' to something that explains 16 vs. 32
>   but depending on internal representation (UCS-2 or UCS-4) selected at
>   compile time

+1

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 22 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From theller at python.net  Mon Nov 22 15:22:31 2004
From: theller at python.net (Thomas Heller)
Date: Mon Nov 22 15:21:42 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
In-Reply-To: <41A1E39B.3030308@egenix.com> (M.'s message of "Mon, 22 Nov
	2004 14:03:23 +0100")
References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com>
	<41A0A6FE.10203@egenix.com> <871xemalmd.fsf@pluto.noname>
	<41A1A0A8.8010803@egenix.com> <87sm7285ty.fsf@pluto.noname>
	<41A1E39B.3030308@egenix.com>
Message-ID: <hdni2cbc.fsf@python.net>

>> - mention the 0-termination of the return-value of
>>   PyUnicode_AS_UNICODE()
>> - mention the 0-termination of the return-value of
>>   PyUnicode_AsWideChar
>
> I don't think we should document this. Programmers should always
> use the size of the object rather than rely on the 0-termination.

Then, *this* should be clearly mentioned somewhere. imo.

Thomas

From mal at egenix.com  Mon Nov 22 15:30:38 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon Nov 22 15:30:42 2004
Subject: [Python-Dev] Bug in PyLocale_strcoll
In-Reply-To: <hdni2cbc.fsf@python.net>
References: <87pt289rqq.fsf@pluto.noname>
	<20041120230712.GA801@panix.com>	<41A0A6FE.10203@egenix.com>
	<871xemalmd.fsf@pluto.noname>	<41A1A0A8.8010803@egenix.com>
	<87sm7285ty.fsf@pluto.noname>	<41A1E39B.3030308@egenix.com>
	<hdni2cbc.fsf@python.net>
Message-ID: <41A1F80E.8000508@egenix.com>

Thomas Heller wrote:
>>>- mention the 0-termination of the return-value of
>>>  PyUnicode_AS_UNICODE()
>>>- mention the 0-termination of the return-value of
>>>  PyUnicode_AsWideChar
>>
>>I don't think we should document this. Programmers should always
>>use the size of the object rather than rely on the 0-termination.
> 
> 
> Then, *this* should be clearly mentioned somewhere. imo.

FYI, I added this to the PyUnicode_AsWideChar() documentation.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 22 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From carribeiro at gmail.com  Mon Nov 22 15:57:46 2004
From: carribeiro at gmail.com (Carlos Ribeiro)
Date: Mon Nov 22 15:58:18 2004
Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist
In-Reply-To: <41A1C11B.4020808@iinet.net.au>
References: <004e01c4cf83$0166db80$e841fea9@oemcomputer>
	<41A1C11B.4020808@iinet.net.au>
Message-ID: <864d37090411220657703083a8@mail.gmail.com>

On Mon, 22 Nov 2004 20:36:11 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote:
> Anyway, if you're firmly opposed to itertools.iunpack, there isn't much point in
> pursuing it (since, in the end, your opinion is the one that really counts).
> Carlos may choose to post it as a recipe over at ASPN.

I also grew some doubts about iunpack(), as currently proposed. I
still feel that it has enough of a use case to deserve discussion, but
I'm not so sure about the details. I did check itertools before
submitting my original implementation, and for some reason, I didn't
perceive that islice coud be used for a similar effect with a slightly
different usage pattern.

One of the advantages of writing a PEP (or pre-PEP, as it is now) is
that, even if not absolutely required for the proposed addition to
itertools, it's still a way to encourage discussion and document the
results. I also still think that the pre-PEP could be published on the
main c.l.py list, for discussion and feedback.

The key point is the comparison between iunpack and islice. If iunpack
is deemed to be just an obvious extension to behavior that can already
be achieved with iunpack, then fine -- forget it. The most that I can
propose is to include an example in the documentation on how to
achieve iunpack behavior with islice. On the other hand, if it becomes
clear that the actual usage pattern for iunpack and islice are in fact
different, then I would stand for the addition of iunpack. Of course,
Raymond has the final word :-)

-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro@gmail.com
mail: carribeiro@yahoo.com
From misa at redhat.com  Mon Nov 22 19:52:11 2004
From: misa at redhat.com (Mihai Ibanescu)
Date: Mon Nov 22 19:52:14 2004
Subject: [Python-Dev] Multilib strikes back
Message-ID: <20041122185211.GL19174@abulafia.devel.redhat.com>

Hello,

This is yet another pain caused by the existance of both 32 and 64-bit
libraries/binaries on the same system

This is the bug that landed on my plate:

http://bugzilla.redhat.com/beta/show_bug.cgi?id=139911

Even though I had no plan to have both 32 and 64-bit python on the same
system, the reporter of this bug is trying to compule a 32-bit version of
libxml on a 64-bit system, and is getting into weird errors, mostly because
the .h files have the wrong definition.

Please see comment #2 from Jakub about possible solutions. If one has strong
preferences one way or another please let me know and I'll write the patch,
otherwise I tend to favor the (int(sizeof()) option.

Thanks,
Mihai
From tim.peters at gmail.com  Mon Nov 22 20:04:17 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Mon Nov 22 20:04:21 2004
Subject: [Python-Dev] Multilib strikes back
In-Reply-To: <20041122185211.GL19174@abulafia.devel.redhat.com>
References: <20041122185211.GL19174@abulafia.devel.redhat.com>
Message-ID: <1f7befae0411221104119c1a08@mail.gmail.com>

[Mihai Ibanescu]
> This is yet another pain caused by the existance of both 32 and 64-bit
> libraries/binaries on the same system
> 
> This is the bug that landed on my plate:
> 
> http://bugzilla.redhat.com/beta/show_bug.cgi?id=139911
> 
> Even though I had no plan to have both 32 and 64-bit python on the
> same system, the reporter of this bug is trying to compule a 32-bit
> version of libxml on a 64-bit system, and is getting into weird errors,
> mostly because the .h files have the wrong definition.
>
> Please see comment #2 from Jakub about possible solutions. If one
> has strong preferences one way or another please let me know and I'll
> write the patch, otherwise I tend to favor the (int(sizeof()) option.

I can't make time to pretend to understand this, but this solution won't fly:

    Particularly for the SIZEOF_* defines, one solution is to change them
    from constants to sizeof (long), sizeof (double) etc.  In C that should
    make absolutely no difference, ...

The SIZEOF_* defines are used in C preprocessor #if expressions, and
sizeof() can't be used in that context.  That's why Python config
defines them as integer literals to begin with.
From misa at redhat.com  Mon Nov 22 20:49:39 2004
From: misa at redhat.com (Mihai Ibanescu)
Date: Mon Nov 22 20:49:48 2004
Subject: [Python-Dev] Multilib strikes back
In-Reply-To: <1f7befae0411221104119c1a08@mail.gmail.com>
References: <20041122185211.GL19174@abulafia.devel.redhat.com>
	<1f7befae0411221104119c1a08@mail.gmail.com>
Message-ID: <20041122194939.GN19174@abulafia.devel.redhat.com>

On Mon, Nov 22, 2004 at 02:04:17PM -0500, Tim Peters wrote:
> [Mihai Ibanescu]
> > This is yet another pain caused by the existance of both 32 and 64-bit
> > libraries/binaries on the same system
> > 
> > This is the bug that landed on my plate:
> > 
> > http://bugzilla.redhat.com/beta/show_bug.cgi?id=139911
> > 
> > Even though I had no plan to have both 32 and 64-bit python on the
> > same system, the reporter of this bug is trying to compule a 32-bit
> > version of libxml on a 64-bit system, and is getting into weird errors,
> > mostly because the .h files have the wrong definition.
> >
> > Please see comment #2 from Jakub about possible solutions. If one
> > has strong preferences one way or another please let me know and I'll
> > write the patch, otherwise I tend to favor the (int(sizeof()) option.
> 
> I can't make time to pretend to understand this, but this solution won't fly:
> 
>     Particularly for the SIZEOF_* defines, one solution is to change them
>     from constants to sizeof (long), sizeof (double) etc.  In C that should
>     make absolutely no difference, ...
> 
> The SIZEOF_* defines are used in C preprocessor #if expressions, and
> sizeof() can't be used in that context.  That's why Python config
> defines them as integer literals to begin with.

Indeed, that would be a problem.
How about a _pyconfig_32.h and a _pyconfig_64.h and have pyconfig.h include
one or the other, depending on the environment?

Thanks again!
Misa
From skip at pobox.com  Mon Nov 22 21:01:24 2004
From: skip at pobox.com (Skip Montanaro)
Date: Mon Nov 22 20:58:32 2004
Subject: [Python-Dev] Multilib strikes back
In-Reply-To: <20041122194939.GN19174@abulafia.devel.redhat.com>
References: <20041122185211.GL19174@abulafia.devel.redhat.com>
	<1f7befae0411221104119c1a08@mail.gmail.com>
	<20041122194939.GN19174@abulafia.devel.redhat.com>
Message-ID: <16802.17812.721194.512642@montanaro.dyndns.org>

    Mihai> How about a _pyconfig_32.h and a _pyconfig_64.h and have
    Mihai> pyconfig.h include one or the other, depending on the
    Mihai> environment?

How would that be detected?  As I understand the original bug report, the
user gave gcc a -m32 flag.  How would that be reflected in the environment?

Let's assume you can coax libxml into compiling in 32-bit mode with Python
support enabled even though you have a 64-bit mode Python installed.  Won't
that just push the problem further down the tool chain?  Instead of a
compilation failure wouldn't you get a runtime error of some sort, or would
the two coexist?

Skip
From martin at v.loewis.de  Mon Nov 22 23:42:18 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon Nov 22 23:42:15 2004
Subject: [Python-Dev] Multilib strikes back
In-Reply-To: <20041122185211.GL19174@abulafia.devel.redhat.com>
References: <20041122185211.GL19174@abulafia.devel.redhat.com>
Message-ID: <41A26B4A.3080401@v.loewis.de>

Mihai Ibanescu wrote:
> Please see comment #2 from Jakub about possible solutions. If one has strong
> preferences one way or another please let me know and I'll write the patch,
> otherwise I tend to favor the (int(sizeof()) option.

I like Jakub's third proposal best: have three different pyconfig.h
files, and install them into different locations. I assume that, in
a multilib installation, there are per-lib directories in the include
path - this is where each different pyconfig.h should go.

Regards,
Martin
From tlb at tlb.org  Mon Nov 22 23:11:23 2004
From: tlb at tlb.org (Trevor Blackwell)
Date: Tue Nov 23 03:02:23 2004
Subject: [Python-Dev] PEP318 and @decorators
Message-ID: <1101161483.55773.352.camel@lab>


I just read the plan for decorators in Python 2.4, and I'm surprised to
see no discussion of what seems to me the cleanest answer, to optionally
write function definitions as assignments:

   foo=def(a,b):
       print a,b

which allows trivial modification by decorators:

   bar1=staticmethod(def(a, b)):
       print a,b

   bar2=accepts(int, int, staticmethod(def(a, b))):
       print a,b

   bar3=adddocstring("this function prints its arguments", 
                     trace(def(a,b))):
       print a,b

It makes it clear in which order the functions are called, unlike with
the @ syntax, and what their arguments are. It has obvious line-break
syntax. You can include arguments to the functions without having to
curry them, making it much easier to write one's own decorator
functions.

Both the "def foo(...):" and "foo=def(...):" syntaxes would be accepted.
I don't think there's any ambiguity. 

Classes would work the same way:

   SimpleClass=class(object):
       ...

   FancyClass=tweakclass(class(object)):
       ...

Among other things, by unifying assignment and definition, it makes the
semantics of function name (re-)binding obvious, so if you read code
like,

   foo=def(a,b):
       print "goodbye"

   foo2=foo

   foo=def(c,b):
       print "hello"

   foo()
   foo2()

it's obvious what will happen, while with the current syntax people
familiar with other languages would expect an error or undefined
behavior. It also works nicely for nested functions, making it obvious
that it binds in the local namespace just like regular assignment.
Making function definition just another sort of assignment emphasizes
the dynamic nature of the language, and will encourage people to treat
functions as first-class data types.

The precise rule is that when there is a def or class in a line (and
there can be only one) there must be a colon and a block following.

You could even use this to provide a more general alternative to lambda,
allowing multi-statement lambdas:

   x=map(def(a), [1,2,3]):
       b=sin(a)
       return b*3
   assert x == [3*sin(1), 3*sin(2), 3*sin(3)]

though this might not be something to encourage. And it makes stashing
functions somewhere easy:

   atexit.register(def()):
       print "exiting..."

   converters.append(def(a)):
       ... convert something ...

Etags doesn't currently grok this syntax, but it should anyway because
it would also help tag global variables. It's easy to implement with a
regex, something like /^\s*(\w+)\s*=.*\bdef\b/

One bummer is that it may be more difficult to set the __name__ property
of a function.

-- 
Trevor Blackwell      tlb@tlb.org          (650) 776-7870


From misa at redhat.com  Tue Nov 23 03:19:14 2004
From: misa at redhat.com (Mihai Ibanescu)
Date: Tue Nov 23 03:19:20 2004
Subject: [Python-Dev] Multilib strikes back
In-Reply-To: <16802.17812.721194.512642@montanaro.dyndns.org>
References: <20041122185211.GL19174@abulafia.devel.redhat.com>
	<1f7befae0411221104119c1a08@mail.gmail.com>
	<20041122194939.GN19174@abulafia.devel.redhat.com>
	<16802.17812.721194.512642@montanaro.dyndns.org>
Message-ID: <20041123021914.GC26050@abulafia.devel.redhat.com>

On Mon, Nov 22, 2004 at 02:01:24PM -0600, Skip Montanaro wrote:
>     Mihai> How about a _pyconfig_32.h and a _pyconfig_64.h and have
>     Mihai> pyconfig.h include one or the other, depending on the
>     Mihai> environment?
> 
> How would that be detected?  As I understand the original bug report, the
> user gave gcc a -m32 flag.  How would that be reflected in the environment?

With the help of Jakub: pyconfig.h would have something close to:

#include <limits.h>

#if CHAR_BIT == 8 && LONG_MAX == 0x7fffffff 
#include "_pyconfig_32.h"
#elif CHAR_BIT == 8 && LONG_MAX == 0x7fffffffffffffff 
#include "_pyconfig_64.h"
#else
#error Unable to detect architecture word length
#endif

This way, if the compiler is in 32-bit mode, the proper file is included.

> Let's assume you can coax libxml into compiling in 32-bit mode with Python
> support enabled even though you have a 64-bit mode Python installed.  Won't
> that just push the problem further down the tool chain?  Instead of a
> compilation failure wouldn't you get a runtime error of some sort, or would
> the two coexist?

Ideally, and that is something I would like to tackle asap, it should be
possible to have both a 32-bit and a 64-bit version of python on the same
machine. This is not exactly possible currently since the binaries are named
the same, but that's the whole purpose of multilib :-)

Misa
From kbk at shore.net  Tue Nov 23 05:02:32 2004
From: kbk at shore.net (Kurt B. Kaiser)
Date: Tue Nov 23 05:02:39 2004
Subject: [Python-Dev] Bugfix to 2.4?
Message-ID: <87r7mlnrfr.fsf@hydra.localdomain>

Hye-Shik Chang worked out a fix for a thread / interrupt bug.  The fix
involves ceval.c.

www.python.org/sf/875692

Skip and I have reviewed it.  IMO it seems to be a minor risk, should
I check it in?

-- 
KBK
From foom at fuhm.net  Tue Nov 23 05:56:46 2004
From: foom at fuhm.net (James Y Knight)
Date: Tue Nov 23 05:57:30 2004
Subject: [Python-Dev] Multilib strikes back
In-Reply-To: <20041123021914.GC26050@abulafia.devel.redhat.com>
References: <20041122185211.GL19174@abulafia.devel.redhat.com>
	<1f7befae0411221104119c1a08@mail.gmail.com>
	<20041122194939.GN19174@abulafia.devel.redhat.com>
	<16802.17812.721194.512642@montanaro.dyndns.org>
	<20041123021914.GC26050@abulafia.devel.redhat.com>
Message-ID: <1400815C-3D0C-11D9-96B0-000A95A50FB2@fuhm.net>


On Nov 22, 2004, at 9:19 PM, Mihai Ibanescu wrote:

> On Mon, Nov 22, 2004 at 02:01:24PM -0600, Skip Montanaro wrote:
>>     Mihai> How about a _pyconfig_32.h and a _pyconfig_64.h and have
>>     Mihai> pyconfig.h include one or the other, depending on the
>>     Mihai> environment?
>>
>> How would that be detected?  As I understand the original bug report, 
>> the
>> user gave gcc a -m32 flag.  How would that be reflected in the 
>> environment?
>
> With the help of Jakub: pyconfig.h would have something close to:
>
> #include <limits.h>
>
> #if CHAR_BIT == 8 && LONG_MAX == 0x7fffffff
> #include "_pyconfig_32.h"
> #elif CHAR_BIT == 8 && LONG_MAX == 0x7fffffffffffffff
> #include "_pyconfig_64.h"
> #else
> #error Unable to detect architecture word length
> #endif
>
> This way, if the compiler is in 32-bit mode, the proper file is 
> included.

I think that is the wrong solution. The right solution is to put 
pyconfig.h in an arch-specific directory. In current standards, that'd 
probably have to be /usr/lib*: e.g. /usr/lib/python2.4/include/ and 
/usr/lib64/python2.4/include/.  In the proposed multiarch standard 
(http://www.linuxbase.org/~taggart/multiarch.html), that would be 
/usr/include/i386-linux/ and /usr/include/x86_64-linux/.

Different, but slightly related: since .py files aren't platform 
dependent, /usr/lib* is really the wrong place for them. They should be 
going in /usr/share/python2.X/ not /usr/lib*/python2.X/. Obviously the 
.so files still need to go in /usr/lib*, though, so that might be a bit 
of a trick.

James

From python at rcn.com  Tue Nov 23 06:23:04 2004
From: python at rcn.com (Raymond Hettinger)
Date: Tue Nov 23 06:25:44 2004
Subject: [Python-Dev] Bugfix to 2.4?
In-Reply-To: <87r7mlnrfr.fsf@hydra.localdomain>
Message-ID: <008901c4d11c$82734b60$e841fea9@oemcomputer>

[Kurt B. Kaiser]
> 
> Hye-Shik Chang worked out a fix for a thread / interrupt bug.  The fix
> involves ceval.c.
> 
> www.python.org/sf/875692
> 
> Skip and I have reviewed it.  IMO it seems to be a minor risk, should
> I check it in?

It's up to Anthony.
I would like to see it go in.


Raymond

From tjreedy at udel.edu  Tue Nov 23 08:32:54 2004
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue Nov 23 08:32:45 2004
Subject: [Python-Dev] Re: PEP318 and @decorators
References: <1101161483.55773.352.camel@lab>
Message-ID: <cnup2p$9nh$1@sea.gmane.org>


"Trevor Blackwell" <tlb@tlb.org> wrote in message 
news:1101161483.55773.352.camel@lab...
>
> I just read the plan for decorators in Python 2.4, and I'm surprised to
> see no discussion

You are a bit late.  Alternatives to @ were discussed last August/September 
in hundreds of posts on pydev and c.l.p/mailing list and summarized on the 
Wiki.  The community selected one which was ably presented to Guido who 
seriously considered it but ultimately rejected it.  End of discussion. 
2.4 is nearly done and ready for release.

> of what seems to me the cleanest answer, to optionally
> write function definitions as assignments:

The issue of multiline anonymous function definitions is mostly 
independent of decorators.  Requests and informal proposals in this regard 
have appeared on c.l.p for years and, I believe, been ignored or informally 
rejected by Guido.  But I don't know that there has been a formal PEP to be 
formally rejected.

I suspect that most developers would prefer that further discussion of this 
issue continue on c.l.p and the corresponding python-list.

Terry J. Reedy



From jason at tishler.net  Tue Nov 23 13:45:53 2004
From: jason at tishler.net (Jason Tishler)
Date: Tue Nov 23 13:44:12 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
In-Reply-To: <20041120150637.GA3024@tishler.net>
References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net>
	<419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net>
Message-ID: <20041123124553.GB2096@tishler.net>

On Sat, Nov 20, 2004 at 10:06:37AM -0500, Jason Tishler wrote:
> > If you (or anyone else) could check that the unit tests work on
> > Cygwin for you, that would make me more certain that this is just a
> > problem with my own Cygwin setup, and not related to anything in
> > 2.4c1 that wasn't in 2.4b1.
> 
> I just upgraded one of my machines to Cygwin 1.5.12, but I did not
> upgrade the other packages.  I get the following results:
> 
>     failures:
> 
>         test_shutil (passed under older Cygwin and/or Python versions)
>         test_subprocess (new test)
>         test_tcl (new test)
> 
>     hangs:
> 
>         test_threaded_import (passed under older Cygwin and/or ...
> 
> I will investigate the above and report back.

AFAICT, the above (with possibly one exception) are due to Cygwin bugs.
I can provide more details if anyone is interested.

On Sat, Nov 20, 2004 at 01:03:42AM +1000, Nick Coghlan wrote:
> This was meant to be a "Yup, she's all good" exercise before 2.4 final
> goes out the door.

IMO, "Yup, she's all good!" :,)

The test_tcl test case is failing under Cygwin because it expects
loadtk() to fail if the DISPLAY environment variable is not set.
However, under Cygwin we have the following:

    >>> import os, Tkinter
    >>> del os.environ['DISPLAY']
    >>> Tkinter.Tcl().loadtk()
    >>>

The above works because Cygwin's Tcl/Tk renders directly on the Windows
GDI instead of going through an X Server.  Can I submit a patch to
test_tcl to skip this test as is done for win and darwin?

Thanks,
Jason

-- 
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D  8784 1AFD E4CC ECF4 8EF6
From jason at tishler.net  Tue Nov 23 13:47:22 2004
From: jason at tishler.net (Jason Tishler)
Date: Tue Nov 23 13:45:39 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
In-Reply-To: <41A1D2F8.9040900@iinet.net.au>
References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net>
	<419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net>
	<419F7535.9050802@iinet.net.au> <41A1D2F8.9040900@iinet.net.au>
Message-ID: <20041123124722.GC2096@tishler.net>

Nick,

On Mon, Nov 22, 2004 at 09:52:24PM +1000, Nick Coghlan wrote:
> Nick Coghlan wrote:
> >Anyway, I won't be posting any more about this on py-dev, since I've
> >established that *my* problems probably aren't Python related.
> 
> I lied. The *real* final word - some digging with Google shows that
> there are still some issues with Cygwin and hyperthreaded processors.
> Sure enough, my desktop is a hyper-threaded P4.

Mine too.  I guess the above may explain why I can't get
test_threaded_import to hang again...

Jason

-- 
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D  8784 1AFD E4CC ECF4 8EF6
From anthony at interlink.com.au  Tue Nov 23 14:19:53 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Tue Nov 23 14:20:11 2004
Subject: [Python-Dev] py2.4 and gcc4
Message-ID: <200411240019.54226.anthony@interlink.com.au>

Fedora rawhide now ships with a gcc4 preview package - I thought I'd build
current-cvs with it. 

Readline fails to build with 
gcc4 -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC 
-fno-strict-aliasing -I. -I/home/anthony/src/py/pyhead/dist/src/./Include 
-I/usr/local/include -I/home/anthony/src/py/pyhead/dist/src/Include 
-I/home/anthony/src/py/pyhead/dist/src 
-c /home/anthony/src/py/pyhead/dist/src/Modules/readline.c -o 
build/temp.linux-i686-2.4/readline.o
/home/anthony/src/py/pyhead/dist/src/Modules/readline.c:97: error: static 
declaration of ?history_length? follows non-static declaration
/usr/include/readline/history.h:224: error: previous declaration of 
?history_length? was here

test_cfgparser causes a segfault, unfortunately. Since this doesn't
occur with any other compilers that I know of (*) I'm inclined to 
blame gcc4, and not bother too much with it further. I'm pretty 
short of time, but if someone wants to spend time figuring out
why, let me know and I'll dump a SF bugreport with the stack
trace &c...

Anthony

(*) Ok - the HP compiler on HP/UX on Itanium causes segfaults and
random breakages _everywhere_ but that's because of the platform.
From nick at craig-wood.com  Tue Nov 23 16:08:46 2004
From: nick at craig-wood.com (Nick Craig-Wood)
Date: Tue Nov 23 16:09:26 2004
Subject: [Python-Dev] Two patches for the new subprocess module
Message-ID: <20041123150846.GA22827@craig-wood.com>

I have submitted two patches for the subprocess module against cvs
HEAD.  The first notifies the user of a common mistake and the second
adds an IHMO very important utility function.

I'd consider the first to be a bug fix and the second a new feature.
However a rather nice new feature ;-)

------------------------------------------------------------
[ 1071755 ] raise error for common mistake with subprocess
http://python.org/sf/1071755
------------------------------------------------------------

subprocess has a very easy to mistake error in it - forgetting to pass
the command as a list.  Eg

>>> import subprocess
>>> subprocess.call("ls")
BaseHTTPServer.py      dummy_threading.py  pickletools.py   socket.py
Bastion.py             email               pipes.py         sre.py
[...]
dummy_thread.py        pickle.pyc          sndhdr.py        zipfile.py
0
>>> subprocess.call("ls", "-l")
BaseHTTPServer.py      dummy_threading.py  pickletools.py   socket.py
Bastion.py             email               pipes.py         sre.py
[...]
dummy_thread.py        pickle.pyc          sndhdr.py        zipfile.py
0
>>> 

# Note no error, warning or anything, but no "ls -l" either

And with the patch...

>>> subprocess.call("ls", "-l")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "subprocess.py", line 428, in call
    return Popen(*args, **kwargs).wait()
  File "subprocess.py", line 508, in __init__
    raise TypeError("bufsize must be an integer - "
TypeError: bufsize must be an integer - did you forget to pass your arguments in a list?
>>> 

------------------------------------------------------------
[ 1071764 ] a new subprocess.call which raises an error on non-zero rc
http://python.org/sf/1071764
------------------------------------------------------------

The attached patch introduces a 3rd utility function - xcall() to the
subprocess module.  This function acts just like call but raises
errors if the command had a non-zero return code.

This saves writing

if call(...):
    raise OSError(...)

It is most useful for shell script replacement programming.  Checking
the return codes of commands called is often forgotten in shell
programming.

When you've moved up to python because shell is too limiting (usually
about 10 lines of shell in my case ;-) you want to make sure that all
your commands work so you write robust code.

I consider raising an exception to be much more pythonic than checking
a return code (ie "Errors should never pass silently" from Zen of
Python)

Eg

# An easy to miss error

>>> subprocess.call(["mkdir", "a/b"])
mkdir: cannot create directory `a/b': No such file or directory
1
>>> # user forgot to check non-zero return code

# becomes an impossible to miss exception

>>> subprocess.xcall(["mkdir", "a/b"])
mkdir: cannot create directory `a/b': No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "subprocess.py", line 462, in xcall
    raise CalledProcessError(rc, "Command %s returned non zero exit status" % args[0])
subprocess.CalledProcessError: [Errno 1] Command ['mkdir', 'a/b'] returned non zero exit status
>>> 

See patch for more!  Its been tested under python 2.3 on windows and
linux.

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
From hyeshik at gmail.com  Tue Nov 23 18:26:12 2004
From: hyeshik at gmail.com (Hye-Shik Chang)
Date: Tue Nov 23 18:26:14 2004
Subject: [Python-Dev] py2.4 and gcc4
In-Reply-To: <200411240019.54226.anthony@interlink.com.au>
References: <200411240019.54226.anthony@interlink.com.au>
Message-ID: <4f0b69dc041123092651196947@mail.gmail.com>

On Wed, 24 Nov 2004 00:19:53 +1100, Anthony Baxter
<anthony@interlink.com.au> wrote:
> Fedora rawhide now ships with a gcc4 preview package - I thought I'd build
> current-cvs with it.
> 
> Readline fails to build with
> gcc4 -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC
> -fno-strict-aliasing -I. -I/home/anthony/src/py/pyhead/dist/src/./Include
> -I/usr/local/include -I/home/anthony/src/py/pyhead/dist/src/Include
> -I/home/anthony/src/py/pyhead/dist/src
> -c /home/anthony/src/py/pyhead/dist/src/Modules/readline.c -o
> build/temp.linux-i686-2.4/readline.o
> /home/anthony/src/py/pyhead/dist/src/Modules/readline.c:97: error: static
> declaration of 'history_length' follows non-static declaration
> /usr/include/readline/history.h:224: error: previous declaration of
> 'history_length' was here

history_length is exported by readline and our readline module has
another history_length for its interal static variable.  I could
build readline module by renaming the variable.

> test_cfgparser causes a segfault, unfortunately. Since this doesn't
> occur with any other compilers that I know of (*) I'm inclined to
> blame gcc4, and not bother too much with it further. I'm pretty
> short of time, but if someone wants to spend time figuring out
> why, let me know and I'll dump a SF bugreport with the stack
> trace &c...

Seemingly, wrong optimization of gcc4.  I was struck by another
segfaults from Objects/stringobject.c:1028 and etc.  But I could
avoid segfaults by disturbing optimizations with some debug outputs.
And even full regression test is passed when I built it with -O0.
No worry. :)

Hye-Shik
From rowen at cesmail.net  Tue Nov 23 21:47:37 2004
From: rowen at cesmail.net (Russell E. Owen)
Date: Tue Nov 23 21:47:43 2004
Subject: [Python-Dev] Re: Two patches for the new subprocess module
References: <20041123150846.GA22827@craig-wood.com>
Message-ID: <rowen-E0468C.12473623112004@sea.gmane.org>

In article <20041123150846.GA22827@craig-wood.com>,
 Nick Craig-Wood <nick@craig-wood.com> wrote:

> I have submitted two patches for the subprocess module against cvs
> HEAD.  The first notifies the user of a common mistake...
>
> subprocess has a very easy to mistake error in it - forgetting to pass
> the command as a list.... with the patch...
> 
> >>> subprocess.call("ls", "-l")
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "subprocess.py", line 428, in call
>     return Popen(*args, **kwargs).wait()
>   File "subprocess.py", line 508, in __init__
>     raise TypeError("bufsize must be an integer - "
> TypeError: bufsize must be an integer - did you forget to pass your arguments 
> in a list?

I hope you didn't totally eliminate the ability for Popen and call to 
accept args as a string? It is a useful feature and one I've been using.

If you have broken this, please consider some alternative to your patch.

Perhaps changing the default for shell to True if args is a string, 
False if a list would suffice? I'm not sure this is really the right 
thing to do on all platforms, but it'd be an easy fix (use shell=None in 
the argument list and then convert it to True or False as needed).

One could imagine other fixes, such as offering one variant each of 
Popen and call that insists on args=string, leaving the main version to 
insist on args=list.

Regards,

-- Russell

From bac at OCF.Berkeley.EDU  Wed Nov 24 04:02:18 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Wed Nov 24 04:02:27 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
In-Reply-To: <20041123124553.GB2096@tishler.net>
References: <419E0B4E.7040109@iinet.net.au>
	<20041119232733.GB1680@tishler.net>	<419EABE2.9040804@iinet.net.au>
	<20041120150637.GA3024@tishler.net>
	<20041123124553.GB2096@tishler.net>
Message-ID: <41A3F9BA.3090702@ocf.berkeley.edu>

Jason Tishler wrote:

> The test_tcl test case is failing under Cygwin because it expects
> loadtk() to fail if the DISPLAY environment variable is not set.
> However, under Cygwin we have the following:
> 
>     >>> import os, Tkinter
>     >>> del os.environ['DISPLAY']
>     >>> Tkinter.Tcl().loadtk()
>     >>>
> 
> The above works because Cygwin's Tcl/Tk renders directly on the Windows
> GDI instead of going through an X Server.  Can I submit a patch to
> test_tcl to skip this test as is done for win and darwin?

You could, but I just fixed it anyway. =)  In revision 1.6 .

-Brett
From kbk at shore.net  Wed Nov 24 07:23:33 2004
From: kbk at shore.net (Kurt B. Kaiser)
Date: Wed Nov 24 07:23:38 2004
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200411240623.iAO6NXE3029528@h006008a7bda6.ne.client2.attbi.com>

Patch / Bug Summary
___________________

Patches :  254 open ( +8) /  2700 closed ( +1) /  2954 total ( +9)
Bugs    :  784 open (+14) /  4629 closed (+15) /  5413 total (+29)
RFE     :  156 open ( +0) /   135 closed ( +0) /   291 total ( +0)

New / Reopened Patches
______________________

os.path.exists returns False if no permission  (2004-11-17)
       http://python.org/sf/1068277  opened by  Robert Brewer

small update for pdb docs  (2004-11-17)
CLOSED http://python.org/sf/1068456  opened by  Ilya Sandler

incomplete support for AF_PACKET in socketmodule.c  (2004-11-19)
       http://python.org/sf/1069624  opened by  Gustavo J. A. M. Carneiro

xmlrpclib - marshalling new-style classes.  (2004-11-20)
       http://python.org/sf/1070046  opened by  Gabriel Pastor

Add BLANK_LINE to token.py  (2004-11-20)
       http://python.org/sf/1070218  opened by  logistix

bug#1021756: more informative error message  (2004-11-23)
       http://python.org/sf/1071739  opened by  Christos Georgiou

raise error for common mistake with subprocess  (2004-11-23)
       http://python.org/sf/1071755  opened by  Nick Craig-Wood

a new subprocess.call which raises an error on non-zero rc  (2004-11-23)
       http://python.org/sf/1071764  opened by  Nick Craig-Wood

Add support for db 4.3  (2004-11-23)
       http://python.org/sf/1071911  opened by  Michal &#268;iha&#345;

Patches Closed
______________

small update for pdb docs  (2004-11-18)
       http://python.org/sf/1068456  closed by  rhettinger

New / Reopened Bugs
___________________

subprocess is not EINTR-safe  (2004-11-17)
       http://python.org/sf/1068268  opened by  Peter ?strand

linecache.py::update_cache strips directory info from files  (2004-11-18)
       http://python.org/sf/1068477  opened by  Tim Leslie

confusing new method names for lists  (2004-11-18)
CLOSED http://python.org/sf/1068590  opened by  McErnie

TclError not a subclass of StandardError  (2004-11-18)
       http://python.org/sf/1068881  opened by  Russell Owen

segfault on printing nested sequences of None/Ellipsis  (2004-11-19)
       http://python.org/sf/1069092  opened by  Jonas K?lker

PyThreadState_SetAsyncExc segfault  (2004-11-18)
       http://python.org/sf/1069160  opened by  Tim Peters

OS X (Panther) Framework Install causes version mismatch  (2004-11-18)
       http://python.org/sf/1069198  opened by  Dusty Harr

"Python/C API Reference Manual" lost one Function  (2004-11-19)
CLOSED http://python.org/sf/1069287  opened by  Raise L. Sail

C:\Python24\Lib\compileall.py returns False  (2004-11-19)
       http://python.org/sf/1069409  opened by  PieterB

import on Windows: please call SetErrorMode first  (2004-11-19)
       http://python.org/sf/1069410  opened by  Dimitri Papadopoulos

Python 2.4c1 does not have universal newline support  (2004-11-19)
       http://python.org/sf/1069433  opened by  PieterB

setup.py fails - cannot find byteyears.py  (2004-11-20)
       http://python.org/sf/1070121  reopened by  bcannon

setup.py fails - cannot find byteyears.py  (2004-11-20)
CLOSED http://python.org/sf/1070121  opened by  crescentd

endianness detection fails on IRIX 5.3  (2004-11-20)
       http://python.org/sf/1070140  opened by  Georg Schwarz

urllib2 authentication redirection error(?)  (2004-11-21)
       http://python.org/sf/1070735  opened by  Allan B. Wilson

urllib2 authentication redirection error(?)  (2004-11-22)
       http://python.org/sf/1071147  opened by  Allan B. Wilson

os.walk example for deleting a full tree is sometime wrong  (2004-11-22)
CLOSED http://python.org/sf/1071087  opened by  Olivier Bornet

some latex reject the pdfinfo  macro while generating html  (2004-11-22)
       http://python.org/sf/1071094  opened by  Marc-Antoine Parent

Documented grammar for List displays incorrect (testlist)  (2004-11-22)
       http://python.org/sf/1071248  opened by  Lenard Lindstrom

email: no way to add additional parameter to Content-Type  (2004-11-22)
CLOSED http://python.org/sf/1071459  opened by  Tessa Lau

test_sutil fails on cygwin  (2004-11-23)
       http://python.org/sf/1071513  reopened by  tebeka

test_sutil fails on cygwin  (2004-11-23)
       http://python.org/sf/1071513  opened by  Miki Tebeka

test_tcl fails on cygwin  (2004-11-23)
CLOSED http://python.org/sf/1071514  opened by  Miki Tebeka

test_subprocess fails on cygwin  (2004-11-23)
       http://python.org/sf/1071516  opened by  Miki Tebeka

moneyfmt recipe in decimal documentation has error  (2004-11-23)
CLOSED http://python.org/sf/1071566  opened by  Anna Ravenscroft

coercing decimal to int doesn't work between -1 and 1  (2004-11-23)
       http://python.org/sf/1071588  opened by  Anna Ravenscroft

Windows msi doesn't install site-packages directory  (2004-11-23)
       http://python.org/sf/1071594  opened by  Anna Ravenscroft

configure problem on HP-UX 11.11  (2004-11-23)
       http://python.org/sf/1071597  opened by  Harri Pasanen

difflib HTML support description incorrect  (2004-11-23)
CLOSED http://python.org/sf/1072124  opened by  Dan Gass

bad arg type to isspace in struct module  (2004-11-23)
       http://python.org/sf/1072182  opened by  Greg McFarlane

re module segfaulting in large regular expression  (2004-11-23)
       http://python.org/sf/1072259  opened by  Erik Demaine

Bugs Closed
___________

Obsolete info in Tutorial 9.1  (2004-11-15)
       http://python.org/sf/1067018  closed by  rhettinger

Small typo  (2004-11-15)
       http://python.org/sf/1067023  closed by  rhettinger

subprocess works poorly on Windows with Python 2.3  (2004-10-29)
       http://python.org/sf/1057048  closed by  astrand

Overflow error seek()ing with float values > (2 ** 31) - 1  (2004-11-16)
       http://python.org/sf/1067728  closed by  rhettinger

Typo about PyErr_WriteUnraisable()  (2004-11-14)
       http://python.org/sf/1066036  closed by  rhettinger

confusing new method names for lists  (2004-11-18)
       http://python.org/sf/1068590  closed by  rhettinger

"Python/C API Reference Manual" lost one Function  (2004-11-19)
       http://python.org/sf/1069287  closed by  bcannon

test_socket fails  (2004-11-06)
       http://python.org/sf/1061429  closed by  bcannon

setup.py fails - cannot find byteyears.py  (2004-11-20)
       http://python.org/sf/1070121  closed by  bcannon

setup.py fails - cannot find byteyears.py  (2004-11-20)
       http://python.org/sf/1070121  closed by  bcannon

os.walk example for deleting a full tree is sometime wrong  (2004-11-22)
       http://python.org/sf/1071087  closed by  tim_one

Ctrl-C doesn't work with sleepy main thread  (2004-01-12)
       http://python.org/sf/875692  closed by  kbk

email: no way to add additional parameter to Content-Type  (2004-11-23)
       http://python.org/sf/1071459  closed by  bwarsaw

test_sutil fails on cygwin  (2004-11-23)
       http://python.org/sf/1071513  closed by  jlgijsbers

test_tcl fails on cygwin  (2004-11-23)
       http://python.org/sf/1071514  closed by  bcannon

moneyfmt recipe in decimal documentation has error  (2004-11-23)
       http://python.org/sf/1071566  closed by  rhettinger

difflib HTML support description incorrect  (2004-11-23)
       http://python.org/sf/1072124  closed by  jlgijsbers

From nick at craig-wood.com  Wed Nov 24 10:55:37 2004
From: nick at craig-wood.com (Nick Craig-Wood)
Date: Wed Nov 24 10:55:40 2004
Subject: [Python-Dev] Re: Two patches for the new subprocess module
In-Reply-To: <rowen-E0468C.12473623112004@sea.gmane.org>
References: <20041123150846.GA22827@craig-wood.com>
	<rowen-E0468C.12473623112004@sea.gmane.org>
Message-ID: <20041124095537.GA29790@craig-wood.com>

On Tue, Nov 23, 2004 at 12:47:37PM -0800, Russell E. Owen wrote:
> In article <20041123150846.GA22827@craig-wood.com>,
>  Nick Craig-Wood <nick@craig-wood.com> wrote:
> 
> > I have submitted two patches for the subprocess module against cvs
> > HEAD.  The first notifies the user of a common mistake...
> >
> > subprocess has a very easy to mistake error in it - forgetting to pass
> > the command as a list.... with the patch...
> > 
> > >>> subprocess.call("ls", "-l")
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> >   File "subprocess.py", line 428, in call
> >     return Popen(*args, **kwargs).wait()
> >   File "subprocess.py", line 508, in __init__
> >     raise TypeError("bufsize must be an integer - "
> > TypeError: bufsize must be an integer - did you forget to pass your arguments 
> > in a list?
> 
> I hope you didn't totally eliminate the ability for Popen and call to 
> accept args as a string? It is a useful feature and one I've been using.

No, you can still pass *ONE* string as an arg, and that works fine.
If you attempt to pass two (not in a list) it raises an exception as
above.

If you pass two arguments to Popen() or call() the second actually
sets the buffer size...

> If you have broken this, please consider some alternative to your patch.
> 
> Perhaps changing the default for shell to True if args is a string, 
> False if a list would suffice? I'm not sure this is really the right 
> thing to do on all platforms, but it'd be an easy fix (use shell=None in 
> the argument list and then convert it to True or False as needed).

That is a really bad idea IMHO!  Its exactly what perl does in its
system() built in function, and it is a huge source of security holes
in perl scripts.

Eg programmer writes system("prog").  Then realises that prog needs an
argument so writes system("prog $arg").  This is now a potential
security problem if $arg was supplied by a user, because this will be
interpreted by the shell.  Say the user makes $arg="blah ; rm -rf *"
then you are in trouble.  If you are thinking about security you'd
write system("prog", $arg), but the only way this error is caught in
perl is with code review.

Back to python - the subprocess module doesn't have this problem.
subprocess.call("prog "+arg) won't actually work because it won't find
the binary called "prog "+arg.  It will just throw an exception at
that point - a very definite clue to the programmer that there is a
mistake.

Eg

>>> subprocess.call("ls -l")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "subprocess.py", line 447, in call
    return Popen(*args, **kwargs).wait()
  File "subprocess.py", line 592, in __init__
    errread, errwrite)
  File "subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
>>> 

Having to write

  subprocess.call("prog "+arg, shell=True)

at least should give the programmer pause for thought, and hopefully
the shorter and better

  subprocess.call(["prog", arg])

would come to mind.

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick
From jlg at dds.nl  Wed Nov 24 11:39:11 2004
From: jlg at dds.nl (Johannes Gijsbers)
Date: Wed Nov 24 11:36:33 2004
Subject: [Python-Dev] Subversion on SourceForge
Message-ID: <20041124103911.GA9168@authsmtp.dds.nl>

FIY, I just received this in the monthly update from SourceForge:

"For year six, we have a lot of exciting things planned, including UI
updates, enhanced tools, new tools, and Subversion support (version
control). It will be an exciting year. We can't wait to show you."

Cheers,

Johannes
From jason at tishler.net  Wed Nov 24 22:23:42 2004
From: jason at tishler.net (Jason Tishler)
Date: Wed Nov 24 22:21:49 2004
Subject: [Python-Dev] Current CVS, Cygwin and "make test"
In-Reply-To: <41A3F9BA.3090702@ocf.berkeley.edu>
References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net>
	<419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net>
	<20041123124553.GB2096@tishler.net>
	<41A3F9BA.3090702@ocf.berkeley.edu>
Message-ID: <20041124212342.GA2140@tishler.net>

Brett,

On Tue, Nov 23, 2004 at 07:02:18PM -0800, Brett C. wrote:
> Jason Tishler wrote:
> >The above works because Cygwin's Tcl/Tk renders directly on the
> >Windows GDI instead of going through an X Server.  Can I submit a
> >patch to test_tcl to skip this test as is done for win and darwin?
> 
> You could, but I just fixed it anyway. =)  In revision 1.6 .

Thanks!

Jason

-- 
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D  8784 1AFD E4CC ECF4 8EF6
From mal at egenix.com  Fri Nov 26 12:23:03 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri Nov 26 12:23:06 2004
Subject: [Python-Dev] Multilib strikes back
In-Reply-To: <1400815C-3D0C-11D9-96B0-000A95A50FB2@fuhm.net>
References: <20041122185211.GL19174@abulafia.devel.redhat.com>	<1f7befae0411221104119c1a08@mail.gmail.com>	<20041122194939.GN19174@abulafia.devel.redhat.com>	<16802.17812.721194.512642@montanaro.dyndns.org>	<20041123021914.GC26050@abulafia.devel.redhat.com>
	<1400815C-3D0C-11D9-96B0-000A95A50FB2@fuhm.net>
Message-ID: <41A71217.7090609@egenix.com>

James Y Knight wrote:
> 
> On Nov 22, 2004, at 9:19 PM, Mihai Ibanescu wrote:
> 
>> On Mon, Nov 22, 2004 at 02:01:24PM -0600, Skip Montanaro wrote:
>>
>>>     Mihai> How about a _pyconfig_32.h and a _pyconfig_64.h and have
>>>     Mihai> pyconfig.h include one or the other, depending on the
>>>     Mihai> environment?
>>>
>>> How would that be detected?  As I understand the original bug report, 
>>> the
>>> user gave gcc a -m32 flag.  How would that be reflected in the 
>>> environment?
>>
>>
>> With the help of Jakub: pyconfig.h would have something close to:
>>
>> #include <limits.h>
>>
>> #if CHAR_BIT == 8 && LONG_MAX == 0x7fffffff
>> #include "_pyconfig_32.h"
>> #elif CHAR_BIT == 8 && LONG_MAX == 0x7fffffffffffffff
>> #include "_pyconfig_64.h"
>> #else
>> #error Unable to detect architecture word length
>> #endif
>>
>> This way, if the compiler is in 32-bit mode, the proper file is included.
> 
> 
> I think that is the wrong solution. The right solution is to put 
> pyconfig.h in an arch-specific directory.

+1

I wonder why 64-bit Unixes seem to ignore the fact that header
files may very well include platform specific settings. Currently,
only libs and binary files are separated according to platform.

However, this is a OS builder related problem and not one that
Python will need to solve.

> In current standards, that'd 
> probably have to be /usr/lib*: e.g. /usr/lib/python2.4/include/ and 
> /usr/lib64/python2.4/include/.  In the proposed multiarch standard 
> (http://www.linuxbase.org/~taggart/multiarch.html), that would be 
> /usr/include/i386-linux/ and /usr/include/x86_64-linux/.
 >
> Different, but slightly related: since .py files aren't platform 
> dependent, /usr/lib* is really the wrong place for them. They should be 
> going in /usr/share/python2.X/ not /usr/lib*/python2.X/. Obviously the 
> .so files still need to go in /usr/lib*, though, so that might be a bit 
> of a trick.

That won't work: extensions often place their .so files into their
package directories. As a result they need a platforma specific
location as well.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 26 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From mal at egenix.com  Fri Nov 26 12:37:30 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri Nov 26 12:37:33 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
Message-ID: <41A7157A.7030405@egenix.com>

I just tried to compile some extensions on Win XP with .NET 1.1
and Python 2.4c1 installed.

It turns out that the distutils recognition logic for finding
a suitable C compiler is not able to find the compiler by simply
looking at the PATH, INCLUDE and LIB settings of the OS instead
of poking around in the registry :-)

Now since extensions for 2.4 will have to be built using the
same compiler as Python itself (the one that comes with VC7.1
which is the same as the one in the freely downloadable
.NET 1.1 SDK), I would suggest to make distutils aware of the
compiler by looking on the PATH in case the registry doesn't
have the VC7.1 entries.

I think this would be a good patch candidate for 2.4.1 - not
sure whether it's important enough to squeeze something into
2.4c2.

What do you think ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 26 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From ncoghlan at iinet.net.au  Fri Nov 26 15:21:27 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Fri Nov 26 15:21:30 2004
Subject: [Python-Dev] String literal concatenation & docstrings
Message-ID: <41A73BE7.7020707@iinet.net.au>

A c.l.p question about docstring formatting got me curious about something.

http://www.python.org/doc/2.3.4/ref/string-catenation.html states that:
   Multiple adjacent string literals (delimited by whitespace), possibly using
   different quoting conventions, are allowed, and their meaning is the same as
   their concatenation. Thus, "hello" 'world' is equivalent to "helloworld".

This isn't quite true, since the following doesn't work:

   def some_func():
     """Doc string line 1 (the only line, surprisingly)\n"""
     """Doc string line 2, except it isn't."""

It seems like an odd quirk that the compile-time concatenation of string 
literals doesn't work for docstrings. I had a bit of trawl through the docs and 
the archive with Google, but couldn't find anything that stated whether this 
behaviour was deliberate or accidental.

So, can anyone satisfy my idle curiousity as to whether this was a deliberate 
design choice, or an accident of the implementation?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan@email.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://boredomandlaziness.skystorm.net
From theller at python.net  Fri Nov 26 15:26:18 2004
From: theller at python.net (Thomas Heller)
Date: Fri Nov 26 15:25:29 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A7157A.7030405@egenix.com> (M.'s message of "Fri, 26 Nov
	2004 12:37:30 +0100")
References: <41A7157A.7030405@egenix.com>
Message-ID: <3bywu1o5.fsf@python.net>

"M.-A. Lemburg" <mal@egenix.com> writes:

> I just tried to compile some extensions on Win XP with .NET 1.1
> and Python 2.4c1 installed.
>
> It turns out that the distutils recognition logic for finding
> a suitable C compiler is not able to find the compiler by simply
> looking at the PATH, INCLUDE and LIB settings of the OS instead
> of poking around in the registry :-)
>
> Now since extensions for 2.4 will have to be built using the
> same compiler as Python itself (the one that comes with VC7.1
> which is the same as the one in the freely downloadable
> .NET 1.1 SDK), I would suggest to make distutils aware of the
> compiler by looking on the PATH in case the registry doesn't
> have the VC7.1 entries.

I always wondered if non-present registry entries wouldn't mean that the
MSVC installation is broken.  OTOH, I always used the official MS IDEs,
and never tried their free compilers - is this desire related to the
free ones?

Thomas

From jim at zope.com  Fri Nov 26 16:01:06 2004
From: jim at zope.com (Jim Fulton)
Date: Fri Nov 26 16:01:10 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A7157A.7030405@egenix.com>
References: <41A7157A.7030405@egenix.com>
Message-ID: <41A74532.1050201@zope.com>

M.-A. Lemburg wrote:
> I just tried to compile some extensions on Win XP with .NET 1.1
> and Python 2.4c1 installed.
> 
> It turns out that the distutils recognition logic for finding
> a suitable C compiler is not able to find the compiler by simply
> looking at the PATH, INCLUDE and LIB settings of the OS instead
> of poking around in the registry :-)
> 
> Now since extensions for 2.4 will have to be built using the
> same compiler as Python itself (the one that comes with VC7.1
> which is the same as the one in the freely downloadable
> .NET 1.1 SDK), I would suggest to make distutils aware of the
> compiler by looking on the PATH in case the registry doesn't
> have the VC7.1 entries.
> 
> I think this would be a good patch candidate for 2.4.1

+1

  - not
> sure whether it's important enough to squeeze something into
> 2.4c2.

I think it isn't.

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org
From mwh at python.net  Fri Nov 26 16:06:01 2004
From: mwh at python.net (Michael Hudson)
Date: Fri Nov 26 16:06:02 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <41A73BE7.7020707@iinet.net.au> (Nick Coghlan's message of
	"Sat, 27 Nov 2004 00:21:27 +1000")
References: <41A73BE7.7020707@iinet.net.au>
Message-ID: <2mpt20veee.fsf@starship.python.net>

Nick Coghlan <ncoghlan@iinet.net.au> writes:

> A c.l.p question about docstring formatting got me curious about something.
>
> http://www.python.org/doc/2.3.4/ref/string-catenation.html states that:
>    Multiple adjacent string literals (delimited by whitespace), possibly using
>    different quoting conventions, are allowed, and their meaning is the same as
>    their concatenation. Thus, "hello" 'world' is equivalent to "helloworld".
>
> This isn't quite true, since the following doesn't work:
>
>    def some_func():
>      """Doc string line 1 (the only line, surprisingly)\n"""
>      """Doc string line 2, except it isn't."""

I haven't actually checked or anything rash like that, but I'd imagine
the answer is something like:

   The two strings are separate statements as far as the parser is
   concerned, and the "concatenating adjacent strings" thing only
   happens within an expression.

You can do this:

>>> "con"\
... "cat"
'concat'


> So, can anyone satisfy my idle curiousity as to whether this was a
> deliberate design choice, or an accident of the implementation?

Well, it surprises me not at all.

Cheers,
mwh

-- 
  ARTHUR:  Why should he want to know where his towel is?
    FORD:  Everybody should know where his towel is.
  ARTHUR:  I think your head's come undone.
                    -- The Hitch-Hikers Guide to the Galaxy, Episode 7
From ncoghlan at iinet.net.au  Fri Nov 26 16:19:10 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Fri Nov 26 16:19:18 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <2mpt20veee.fsf@starship.python.net>
References: <41A73BE7.7020707@iinet.net.au>
	<2mpt20veee.fsf@starship.python.net>
Message-ID: <41A7496E.9020309@iinet.net.au>

Michael Hudson wrote:
> Nick Coghlan <ncoghlan@iinet.net.au> writes:
> I haven't actually checked or anything rash like that, but I'd imagine
> the answer is something like:
> 
>    The two strings are separate statements as far as the parser is
>    concerned, and the "concatenating adjacent strings" thing only
>    happens within an expression.

That would certainly be a sensible explanation. The only time I've ever actually 
made use of the feature is when assigning a long string, and even then only 
rarely (I'm more likely to use triple quotes and left align the whole thing)

> You can do this:
>>>>"con"\
> 
> ... "cat"
> 'concat'

Which actually does work for combining multiple strings into a single docstring.

>>So, can anyone satisfy my idle curiousity as to whether this was a
>>deliberate design choice, or an accident of the implementation?
> 
> Well, it surprises me not at all.

I think the key distinction I'd missed was that in the doc string case, the two 
strings were actually separate statements. Once that distinction is noted, the 
behaviour is, as you say, unsurprising. It also makes it obvious why escaping 
the newline has the effect it does.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan@email.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://boredomandlaziness.skystorm.net
From martin at v.loewis.de  Fri Nov 26 17:27:42 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri Nov 26 17:27:42 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A7157A.7030405@egenix.com>
References: <41A7157A.7030405@egenix.com>
Message-ID: <41A7597E.2090805@v.loewis.de>

M.-A. Lemburg wrote:
> Now since extensions for 2.4 will have to be built using the
> same compiler as Python itself (the one that comes with VC7.1
> which is the same as the one in the freely downloadable
> .NET 1.1 SDK)

My understanding is that these are not exactly the same compilers;
I recall that the freely-redistributable one lacks support for
optimization.

Regards,
Martin
From ronaldoussoren at mac.com  Fri Nov 26 17:36:54 2004
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri Nov 26 17:37:03 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A7597E.2090805@v.loewis.de>
References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de>
Message-ID: <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>


On 26-nov-04, at 17:27, Martin v. L?wis wrote:

> M.-A. Lemburg wrote:
>> Now since extensions for 2.4 will have to be built using the
>> same compiler as Python itself (the one that comes with VC7.1
>> which is the same as the one in the freely downloadable
>> .NET 1.1 SDK)
>
> My understanding is that these are not exactly the same compilers;
> I recall that the freely-redistributable one lacks support for
> optimization.

That's not what their download page says:

<quote href="http://msdn.microsoft.com/visualc/vctoolkit2003/">
Microsoft Visual C++ Toolkit 2003

  The Microsoft Visual C++ Toolkit 2003 includes the core tools 
developers need to compile and link C++-based applications for Windows 
and the .NET Common Language Runtime:

	? 	Microsoft C/C++ Optimizing Compiler and Linker.? These are the same 
compiler and linker that ship with Visual Studio .NET 2003 
Professional!
</quote>

Ronald
--
X|support bv            http://www.xsupport.nl/
T:  +31 610271479       F:  +31 204416173

From mal at egenix.com  Fri Nov 26 17:49:16 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri Nov 26 17:49:15 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <3bywu1o5.fsf@python.net>
References: <41A7157A.7030405@egenix.com> <3bywu1o5.fsf@python.net>
Message-ID: <41A75E8C.3030301@egenix.com>

Thomas Heller wrote:
> "M.-A. Lemburg" <mal@egenix.com> writes:
> 
> 
>>I just tried to compile some extensions on Win XP with .NET 1.1
>>and Python 2.4c1 installed.
>>
>>It turns out that the distutils recognition logic for finding
>>a suitable C compiler is not able to find the compiler by simply
>>looking at the PATH, INCLUDE and LIB settings of the OS instead
>>of poking around in the registry :-)
>>
>>Now since extensions for 2.4 will have to be built using the
>>same compiler as Python itself (the one that comes with VC7.1
>>which is the same as the one in the freely downloadable
>>.NET 1.1 SDK), I would suggest to make distutils aware of the
>>compiler by looking on the PATH in case the registry doesn't
>>have the VC7.1 entries.
> 
> I always wondered if non-present registry entries wouldn't mean that the
> MSVC installation is broken. 

Not at all: their C compiler works just fine once you setup
the OS environment by loading the sdkvars.bat file.

> OTOH, I always used the official MS IDEs,
> and never tried their free compilers - is this desire related to the
> free ones?

Yes. distutils doesn't need any IDE to compile extensions,
just a compiler and a command line :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 26 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From theller at python.net  Fri Nov 26 17:53:05 2004
From: theller at python.net (Thomas Heller)
Date: Fri Nov 26 17:52:15 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> (Ronald
	Oussoren's message of "Fri, 26 Nov 2004 17:36:54 +0100")
References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de>
	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>
Message-ID: <653ssgb2.fsf@python.net>

Ronald Oussoren <ronaldoussoren@mac.com> writes:

> On 26-nov-04, at 17:27, Martin v. L?wis wrote:
>
>> M.-A. Lemburg wrote:
>>> Now since extensions for 2.4 will have to be built using the
>>> same compiler as Python itself (the one that comes with VC7.1
>>> which is the same as the one in the freely downloadable
>>> .NET 1.1 SDK)
>>
>> My understanding is that these are not exactly the same compilers;
>> I recall that the freely-redistributable one lacks support for
>> optimization.
>
> That's not what their download page says:
>
> <quote href="http://msdn.microsoft.com/visualc/vctoolkit2003/">
> Microsoft Visual C++ Toolkit 2003
>
>   The Microsoft Visual C++ Toolkit 2003 includes the core tools
>   developers need to compile and link C++-based applications for
>   Windows and the .NET Common Language Runtime:
>
> 	? 	Microsoft C/C++ Optimizing Compiler and Linker.? These
> 	are the same compiler and linker that ship with Visual Studio
> 	.NET 2003 Professional!
> </quote>

Wasn't the difference that no msvcrt import libraries are included,
neither the license to redistribute the msvcrt runtime dlls (although
the missing license probably doesn't matter, because they are already in
the Python distribution)?

Thomas

From anthony at interlink.com.au  Fri Nov 26 17:52:23 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri Nov 26 17:52:48 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A7157A.7030405@egenix.com>
References: <41A7157A.7030405@egenix.com>
Message-ID: <200411270352.24397.anthony@interlink.com.au>

On Friday 26 November 2004 22:37, M.-A. Lemburg wrote:
> I think this would be a good patch candidate for 2.4.1 - not
> sure whether it's important enough to squeeze something into
> 2.4c2.

I don't think it's important enough for 2.4c2/2.4 final, but I'd be
happy to see this in 2.4.1. 

An update: as far as I know, we're still without a fix for the problem
Tim identified last week - he seemed to think that it wasn't dangerous 
enough that we'd need a 2.4c2. I don't know of any other reasons 
that we'd want a c2, so I'll probably go with the final release next,
unless someone has a strong reason not to do so.

I'm not sure whether the bug Tim identified is bad enough to hold
up 2.4 final for - as far as I recall (I've just switched mailers, and
haven't imported my old mail folders yet) it was a function that 
no-one was using... 

Anthony
From mal at egenix.com  Fri Nov 26 18:07:45 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri Nov 26 18:07:45 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <653ssgb2.fsf@python.net>
References: <41A7157A.7030405@egenix.com>
	<41A7597E.2090805@v.loewis.de>	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>
	<653ssgb2.fsf@python.net>
Message-ID: <41A762E1.7040404@egenix.com>

Thomas Heller wrote:
> Ronald Oussoren <ronaldoussoren@mac.com> writes:
> 
> 
>>On 26-nov-04, at 17:27, Martin v. L?wis wrote:
>>
>>
>>>M.-A. Lemburg wrote:
>>>
>>>>Now since extensions for 2.4 will have to be built using the
>>>>same compiler as Python itself (the one that comes with VC7.1
>>>>which is the same as the one in the freely downloadable
>>>>.NET 1.1 SDK)
>>>
>>>My understanding is that these are not exactly the same compilers;
>>>I recall that the freely-redistributable one lacks support for
>>>optimization.
>>
>>That's not what their download page says:
>>
>><quote href="http://msdn.microsoft.com/visualc/vctoolkit2003/">
>>Microsoft Visual C++ Toolkit 2003
>>
>>  The Microsoft Visual C++ Toolkit 2003 includes the core tools
>>  developers need to compile and link C++-based applications for
>>  Windows and the .NET Common Language Runtime:
>>
>>	? 	Microsoft C/C++ Optimizing Compiler and Linker.  These
>>	are the same compiler and linker that ship with Visual Studio
>>	.NET 2003 Professional!
>></quote>

The compiler works happily with the optimization flags...

> Wasn't the difference that no msvcrt import libraries are included,
> neither the license to redistribute the msvcrt runtime dlls (although
> the missing license probably doesn't matter, because they are already in
> the Python distribution)?

... and links again MSVCRT.LIB which is included in .NET 1.1.

The DLLs come with the .NET framework, so there's no need to
redistribute them.

Anyway, the point is not creating binaries that you distribute,
but simply to be able to compile extensions yourself as necessary.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 26 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From ncoghlan at iinet.net.au  Fri Nov 26 18:19:08 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Fri Nov 26 18:19:15 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <653ssgb2.fsf@python.net>
References: <41A7157A.7030405@egenix.com>
	<41A7597E.2090805@v.loewis.de>	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>
	<653ssgb2.fsf@python.net>
Message-ID: <41A7658C.80504@iinet.net.au>

Thomas Heller wrote:
> Wasn't the difference that no msvcrt import libraries are included,
> neither the license to redistribute the msvcrt runtime dlls (although
> the missing license probably doesn't matter, because they are already in
> the Python distribution)?

The main elements needed are the free .NET SDK (which provides a complete C++ 
toolchain, but has a non-optimising C compiler and a statically-linked only C 
runtime) and the Visual C++ Toolkit (which replaces the .NET SDK compiler and 
linker with ones which are actually useful).

The toolkit is useless on its own, since it includes the compiler & linker only, 
and no other required build tools (like, oh, say, nmake).

I believe it's also necessary to install the platform SDK. I know I needed it to 
get the "windows.h" header (since I was building Python itself, rather than an 
exension) - I think it also included the stub libraries to link to the MSVCRT DLL's.

Unfortunately, the Windows partition I had all this set up on died a couple of 
months back, or I'd be able to just write up a summary of what was required :(

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan@email.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://boredomandlaziness.skystorm.net
From martin at v.loewis.de  Fri Nov 26 18:57:11 2004
From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri Nov 26 18:57:10 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A762E1.7040404@egenix.com>
References: <41A7157A.7030405@egenix.com>	<41A7597E.2090805@v.loewis.de>	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>	<653ssgb2.fsf@python.net>
	<41A762E1.7040404@egenix.com>
Message-ID: <41A76E77.6000901@v.loewis.de>

M.-A. Lemburg wrote:
>> Wasn't the difference that no msvcrt import libraries are included,
>> neither the license to redistribute the msvcrt runtime dlls (although
>> the missing license probably doesn't matter, because they are already in
>> the Python distribution)?
> 
> 
> ... and links again MSVCRT.LIB which is included in .NET 1.1.

Python 2.4 is linked against msvcr71.dll, though.

> The DLLs come with the .NET framework, so there's no need to
> redistribute them.

Not msvcr71.dll.

> Anyway, the point is not creating binaries that you distribute,
> but simply to be able to compile extensions yourself as necessary.

Yes, but can you also *link* correctly?

Regards,
Martin

From jim at zope.com  Fri Nov 26 19:12:03 2004
From: jim at zope.com (Jim Fulton)
Date: Fri Nov 26 19:12:08 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A762E1.7040404@egenix.com>
References: <41A7157A.7030405@egenix.com>	<41A7597E.2090805@v.loewis.de>	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>	<653ssgb2.fsf@python.net>
	<41A762E1.7040404@egenix.com>
Message-ID: <41A771F3.3060006@zope.com>

M.-A. Lemburg wrote:
...
> Anyway, the point is not creating binaries that you distribute,
> but simply to be able to compile extensions yourself as necessary.

Or to distribute sources, especially via cvs or svn that *other*
people can compile.  Even though I don't use windows myself, it
would make my life a lot easier if people could compile Zope
exptensions themselves rather than waiting for us or someone else
to build windows binaries for them.

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org
From mal at egenix.com  Fri Nov 26 19:41:37 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri Nov 26 19:41:39 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A76E77.6000901@v.loewis.de>
References: <41A7157A.7030405@egenix.com>	<41A7597E.2090805@v.loewis.de>	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>	<653ssgb2.fsf@python.net>	<41A762E1.7040404@egenix.com>
	<41A76E77.6000901@v.loewis.de>
Message-ID: <41A778E1.7060006@egenix.com>

Martin v. L?wis wrote:
> M.-A. Lemburg wrote:
> 
>>> Wasn't the difference that no msvcrt import libraries are included,
>>> neither the license to redistribute the msvcrt runtime dlls (although
>>> the missing license probably doesn't matter, because they are already in
>>> the Python distribution)?
>>
>>
>>
>> ... and links again MSVCRT.LIB which is included in .NET 1.1.
> 
> 
> Python 2.4 is linked against msvcr71.dll, though.

The MSVCRT.LIB from .NET 1.1 is an import lib for msvcr71.dll.

>> The DLLs come with the .NET framework, so there's no need to
>> redistribute them.
> 
> Not msvcr71.dll.

It was installed as part of the SDK that came MS update (v1.1.4322).

>> Anyway, the point is not creating binaries that you distribute,
>> but simply to be able to compile extensions yourself as necessary.
> 
> Yes, but can you also *link* correctly?

Yes. HelloWorld.c compiles&links just fine - even the
optimized version :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 26 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From bac at OCF.Berkeley.EDU  Fri Nov 26 20:56:05 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Fri Nov 26 20:56:13 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <41A7496E.9020309@iinet.net.au>
References: <41A73BE7.7020707@iinet.net.au>	<2mpt20veee.fsf@starship.python.net>
	<41A7496E.9020309@iinet.net.au>
Message-ID: <41A78A55.8030802@ocf.berkeley.edu>

Nick Coghlan wrote:
> Michael Hudson wrote:
> 
>> Nick Coghlan <ncoghlan@iinet.net.au> writes:
>> I haven't actually checked or anything rash like that, but I'd imagine
>> the answer is something like:
>>
>>    The two strings are separate statements as far as the parser is
>>    concerned, and the "concatenating adjacent strings" thing only
>>    happens within an expression.
> 
> 
> That would certainly be a sensible explanation. The only time I've ever 
> actually made use of the feature is when assigning a long string, and 
> even then only rarely (I'm more likely to use triple quotes and left 
> align the whole thing)
> 

And the sensible explanation is correct.  Just checked out the compiler and the 
string concatenation (in parsestrplus()) takes a node and then proceeds to 
concatenate all of its children that are strings right in a row starting at 
child 0.  With statements this won't trigger anything since the statements will 
only have the string as their child, unlike an expression, which will just have 
all the string pieces.

[SNIP]
> I think the key distinction I'd missed was that in the doc string case, 
> the two strings were actually separate statements. Once that distinction 
> is noted, the behaviour is, as you say, unsurprising. It also makes it 
> obvious why escaping the newline has the effect it does.
> 

Should probably change the wording on that unless people actually want the 
literal string concatenation to work with statements (docstrings seem like the 
only place that would be reasonable) unless you want to start allowing print 
statements to have a string part span multiple lines.  =)

-Brett
From mal at egenix.com  Sat Nov 27 00:26:38 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat Nov 27 00:26:36 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A762E1.7040404@egenix.com>
References: <41A7157A.7030405@egenix.com>
	<41A7597E.2090805@v.loewis.de>	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>
	<653ssgb2.fsf@python.net> <41A762E1.7040404@egenix.com>
Message-ID: <41A7BBAE.6080907@egenix.com>

M.-A. Lemburg wrote:
> Thomas Heller wrote:
> 
>> Ronald Oussoren <ronaldoussoren@mac.com> writes:
>>
>>
>>> On 26-nov-04, at 17:27, Martin v. L?wis wrote:
>>>
>>>
>>>> M.-A. Lemburg wrote:
>>>>
>>>>> Now since extensions for 2.4 will have to be built using the
>>>>> same compiler as Python itself (the one that comes with VC7.1
>>>>> which is the same as the one in the freely downloadable
>>>>> .NET 1.1 SDK)
>>>>
>>>>
>>>> My understanding is that these are not exactly the same compilers;
>>>> I recall that the freely-redistributable one lacks support for
>>>> optimization.
>>>
>>>
>>> That's not what their download page says:
>>>
>>> <quote href="http://msdn.microsoft.com/visualc/vctoolkit2003/">
>>> Microsoft Visual C++ Toolkit 2003
>>>
>>>  The Microsoft Visual C++ Toolkit 2003 includes the core tools
>>>  developers need to compile and link C++-based applications for
>>>  Windows and the .NET Common Language Runtime:
>>>
>>>     ?     Microsoft C/C++ Optimizing Compiler and Linker.  These
>>>     are the same compiler and linker that ship with Visual Studio
>>>     .NET 2003 Professional!
>>> </quote>
> 
> 
> The compiler works happily with the optimization flags...

After doing some extra checks, looking at the generated
assembler and trying the /O2 flag, I have to correct this:
even though the compiler from .NET 1.1 supports the /G optimization
flags it doesn't seem to do any of the more complicated optimizations
which you get with /O2.

Mike Fletcher told me that the Microsoft Visual C++ Toolkit 2003
comes with the optimizing version of the compiler which is also
the one mentioned in the quote.

He also pointed me at:

	http://www.thefreecountry.com/compilers/cpp.shtml

So in summary you need these things if you want an optimizing
compiler for Python 2.4 extensions:

* MS VC++ Toolkit 2003

* MS Platform SDK (if you want to build Windows specific extensions)

In any, the compiler uses the same command line name, so a PATH based
search for the compiler would provide a useful extension for
distutils.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 27 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From martin at v.loewis.de  Sat Nov 27 02:00:35 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov 27 02:00:35 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A771F3.3060006@zope.com>
References: <41A7157A.7030405@egenix.com>	<41A7597E.2090805@v.loewis.de>	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>	<653ssgb2.fsf@python.net>	<41A762E1.7040404@egenix.com>
	<41A771F3.3060006@zope.com>
Message-ID: <41A7D1B3.7070306@v.loewis.de>

Jim Fulton wrote:
> Or to distribute sources, especially via cvs or svn that *other*
> people can compile.  Even though I don't use windows myself, it
> would make my life a lot easier if people could compile Zope
> exptensions themselves rather than waiting for us or someone else
> to build windows binaries for them.

Of course, in that case, it will be sufficient if the other people
have VC.NET 2003 installed. Distutils will properly locate that
installation, and it will then produce proper extension modules.

Regards,
Martin
From carribeiro at gmail.com  Sat Nov 27 04:53:48 2004
From: carribeiro at gmail.com (Carlos Ribeiro)
Date: Sat Nov 27 04:53:53 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <41A78A55.8030802@ocf.berkeley.edu>
References: <41A73BE7.7020707@iinet.net.au>
	<2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au>
	<41A78A55.8030802@ocf.berkeley.edu>
Message-ID: <864d370904112619532ee559d4@mail.gmail.com>

On Fri, 26 Nov 2004 11:56:05 -0800, Brett C. <bac@ocf.berkeley.edu> wrote:
> Should probably change the wording on that unless people actually want the
> literal string concatenation to work with statements (docstrings seem like the
> only place that would be reasonable) unless you want to start allowing print
> statements to have a string part span multiple lines.  =)

It means that:

    print "this line continues"
          "on the next line"

does not work, while the following works:

    a = "this line continues"
        "on the next line"

Kind of weird, but anyway, that's not a common idiom. One more reason
to use triple-quoted-strings when printing long strings.

-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro@gmail.com
mail: carribeiro@yahoo.com
From orbitz at drorbitz.ath.cx  Sat Nov 27 05:01:24 2004
From: orbitz at drorbitz.ath.cx (ort)
Date: Sat Nov 27 05:01:32 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <864d370904112619532ee559d4@mail.gmail.com>
References: <41A73BE7.7020707@iinet.net.au>	<2mpt20veee.fsf@starship.python.net>
	<41A7496E.9020309@iinet.net.au>	<41A78A55.8030802@ocf.berkeley.edu>
	<864d370904112619532ee559d4@mail.gmail.com>
Message-ID: <41A7FC14.8090209@drorbitz.ath.cx>

Like anything, if you need to wrap a statement around multiple lines, 
you surround it in ()'s

Now the question is why does:
 >>> def foo():
...     ("""blah"""
...     """fejlfe""")
...     pass
...
 >>> help(foo)

Not show that as the doc string. Just because it has () doesn't mean it 
evaluates to anything other than a string as far as I know.



Carlos Ribeiro wrote:

>On Fri, 26 Nov 2004 11:56:05 -0800, Brett C. <bac@ocf.berkeley.edu> wrote:
>  
>
>>Should probably change the wording on that unless people actually want the
>>literal string concatenation to work with statements (docstrings seem like the
>>only place that would be reasonable) unless you want to start allowing print
>>statements to have a string part span multiple lines.  =)
>>    
>>
>
>It means that:
>
>    print "this line continues"
>          "on the next line"
>
>does not work, while the following works:
>
>    a = "this line continues"
>        "on the next line"
>
>Kind of weird, but anyway, that's not a common idiom. One more reason
>to use triple-quoted-strings when printing long strings.
>
>  
>

From goodger at python.org  Sat Nov 27 16:07:15 2004
From: goodger at python.org (David Goodger)
Date: Sat Nov 27 16:07:25 2004
Subject: [Python-Dev] Re: String literal concatenation & docstrings
In-Reply-To: <864d370904112619532ee559d4@mail.gmail.com>
References: <41A73BE7.7020707@iinet.net.au>
	<2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au>
	<41A78A55.8030802@ocf.berkeley.edu>
	<864d370904112619532ee559d4@mail.gmail.com>
Message-ID: <41A89823.1040403@python.org>

[Carlos Ribeiro]
>  while the following works:
> 
>     a = "this line continues"
>         "on the next line"

Are you sure about that?  Doesn't work on my machine:

$ cat x.py
a = "this line continues "
     "on the next line"
$ python x.py
   File "x.py", line 2
     "on the next line"
     ^
SyntaxError: invalid syntax

If you add a trailing backslash, it does work:

$ cat x2.py
a = "this line continues " \
     "on the next line"
print a
$ python x2.py
this line continues on the next line

 > Kind of weird

Not weird at all ;-)

-- 
David Goodger <http://python.net/~goodger>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 256 bytes
Desc: OpenPGP digital signature
Url : http://mail.python.org/pipermail/python-dev/attachments/20041127/b6e5ec3c/signature.pgp
From jim at zope.com  Sat Nov 27 16:38:45 2004
From: jim at zope.com (Jim Fulton)
Date: Sat Nov 27 16:38:52 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A7D1B3.7070306@v.loewis.de>
References: <41A7157A.7030405@egenix.com>	<41A7597E.2090805@v.loewis.de>	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>	<653ssgb2.fsf@python.net>	<41A762E1.7040404@egenix.com>
	<41A771F3.3060006@zope.com> <41A7D1B3.7070306@v.loewis.de>
Message-ID: <41A89F85.9080703@zope.com>

Martin v. L?wis wrote:
> Jim Fulton wrote:
> 
>> Or to distribute sources, especially via cvs or svn that *other*
>> people can compile.  Even though I don't use windows myself, it
>> would make my life a lot easier if people could compile Zope
>> exptensions themselves rather than waiting for us or someone else
>> to build windows binaries for them.
> 
> 
> Of course, in that case, it will be sufficient if the other people
> have VC.NET 2003 installed.

Is that free?  If not, that's a big "if".

Jim

-- 
Jim Fulton           mailto:jim@zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org
From astrand at lysator.liu.se  Sat Nov 27 17:09:05 2004
From: astrand at lysator.liu.se (Peter Astrand)
Date: Sat Nov 27 17:09:13 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A7597E.2090805@v.loewis.de>
References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de>
Message-ID: <Pine.GSO.4.51L2.0411270847260.28277@koeberg.lysator.liu.se>

On Fri, 26 Nov 2004, "Martin v. L?wis" wrote:

> M.-A. Lemburg wrote:
> > Now since extensions for 2.4 will have to be built using the
> > same compiler as Python itself (the one that comes with VC7.1
> > which is the same as the one in the freely downloadable
> > .NET 1.1 SDK)
>
> My understanding is that these are not exactly the same compilers;
> I recall that the freely-redistributable one lacks support for
> optimization.

On the contrary: The commercial, boxed "Visual Studio .NET 2003 Standard"
lacks support for optimization, but the free "Visual C++ Toolkit 2003" can
optimize. It is possible to use the free, optimizing compiler with Visual
Studio .NET 2003 Standard, though: See
http://www.sawtoothdistortion.com/Articles/FreeOptimizingCompiler.html


/Peter ?strand <astrand@lysator.liu.se>
From gvanrossum at gmail.com  Sat Nov 27 18:13:25 2004
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sat Nov 27 18:13:29 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <864d370904112619532ee559d4@mail.gmail.com>
References: <41A73BE7.7020707@iinet.net.au>
	<2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au>
	<41A78A55.8030802@ocf.berkeley.edu>
	<864d370904112619532ee559d4@mail.gmail.com>
Message-ID: <ca471dc20411270913780b7e63@mail.gmail.com>

> It means that:
> 
>     print "this line continues"
>           "on the next line"
> 
> does not work, while the following works:
> 
>     a = "this line continues"
>         "on the next line"

As has been pointed out already, it doesn't.

The right way to look at this is not using the statement/expression
distinction, but to look at whether the newline between the two
literals is significant to the parser or not. A significant newline
ends a statement; an insignificant one is equivalent to a space. The
rule is that newlines (outside string quotes anyway) are significant
unless either escaped with \, or contained within matching
parentheses.

So if you want to print or assign a long string broken across multiple
lines without having to use triple-quoted strings (which have their
own set of issues), you can write this:

    print ("this line continues"
           " on the next line")

Note that I fixed a common buglet in your example: the words
"continues" and "on" should be separated by a space. Not important for
the example, of course, but (having used this idiom and similar ones a
lot) something to keep in mind when splitting a long string across a
line this way.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
From bac at OCF.Berkeley.EDU  Sat Nov 27 20:08:34 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Sat Nov 27 20:08:42 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <ca471dc20411270913780b7e63@mail.gmail.com>
References: <41A73BE7.7020707@iinet.net.au>	<2mpt20veee.fsf@starship.python.net>
	<41A7496E.9020309@iinet.net.au>	<41A78A55.8030802@ocf.berkeley.edu>	<864d370904112619532ee559d4@mail.gmail.com>
	<ca471dc20411270913780b7e63@mail.gmail.com>
Message-ID: <41A8D0B2.6060800@ocf.berkeley.edu>

Guido van Rossum wrote:
>>It means that:
>>
>>    print "this line continues"
>>          "on the next line"
>>
>>does not work, while the following works:
>>
>>    a = "this line continues"
>>        "on the next line"
> 
> 
> As has been pointed out already, it doesn't.
> 
> The right way to look at this is not using the statement/expression
> distinction, but to look at whether the newline between the two
> literals is significant to the parser or not. A significant newline
> ends a statement; an insignificant one is equivalent to a space. The
> rule is that newlines (outside string quotes anyway) are significant
> unless either escaped with \, or contained within matching
> parentheses.
> 

So how is this for new wording?

"Multiple adjacent string literals (delimited by whitespace), possibly using 
different quoting conventions, are allowed, and their meaning is the same as 
their concatenation as long as the newline separating them is not signifcant to 
the parser."
From gvanrossum at gmail.com  Sat Nov 27 22:05:42 2004
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sat Nov 27 22:05:46 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <41A8D0B2.6060800@ocf.berkeley.edu>
References: <41A73BE7.7020707@iinet.net.au>
	<2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au>
	<41A78A55.8030802@ocf.berkeley.edu>
	<864d370904112619532ee559d4@mail.gmail.com>
	<ca471dc20411270913780b7e63@mail.gmail.com>
	<41A8D0B2.6060800@ocf.berkeley.edu>
Message-ID: <ca471dc2041127130529ecdf98@mail.gmail.com>

> So how is this for new wording?
> 
> "Multiple adjacent string literals (delimited by whitespace), possibly using
> different quoting conventions, are allowed, and their meaning is the same as 
> their concatenation as long as the newline separating them is not signifcant to
> the parser."

I'm not sure it needs clarifying; it's the reference manual after all,
not a tutorial.

I'd rather let the grammar speak for itself; there's no ambiguity in
that, and the words are just there to clarify the *semantics*.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
From python at rcn.com  Sat Nov 27 22:10:14 2004
From: python at rcn.com (Raymond Hettinger)
Date: Sat Nov 27 22:13:28 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <ca471dc2041127130529ecdf98@mail.gmail.com>
Message-ID: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer>

> > "Multiple adjacent string literals (delimited by whitespace),
possibly
> using
> > different quoting conventions, are allowed, and their meaning is the
> same as
> > their concatenation as long as the newline separating them is not
> signifcant to
> > the parser."
> 
> I'm not sure it needs clarifying; it's the reference manual after all,
> not a tutorial.

Right.  Over-clarification results in docs that read like the
instructions for the holy hand grenade ;-)


Raymond

From martin at v.loewis.de  Sat Nov 27 23:27:12 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat Nov 27 23:27:10 2004
Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils
In-Reply-To: <41A89F85.9080703@zope.com>
References: <41A7157A.7030405@egenix.com>	<41A7597E.2090805@v.loewis.de>	<61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com>	<653ssgb2.fsf@python.net>	<41A762E1.7040404@egenix.com>
	<41A771F3.3060006@zope.com> <41A7D1B3.7070306@v.loewis.de>
	<41A89F85.9080703@zope.com>
Message-ID: <41A8FF40.4040106@v.loewis.de>

Jim Fulton wrote:
>> Of course, in that case, it will be sufficient if the other people
>> have VC.NET 2003 installed.
> 
> Is that free?  If not, that's a big "if".

No, it is a product sold by Microsoft. However, many people
do have VC.NET installed, despite it not being free.

Regards,
Martin
From ncoghlan at iinet.net.au  Sun Nov 28 02:56:49 2004
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sun Nov 28 02:56:56 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <ca471dc2041127130529ecdf98@mail.gmail.com>
References: <41A73BE7.7020707@iinet.net.au>	<2mpt20veee.fsf@starship.python.net>
	<41A7496E.9020309@iinet.net.au>	<41A78A55.8030802@ocf.berkeley.edu>	<864d370904112619532ee559d4@mail.gmail.com>	<ca471dc20411270913780b7e63@mail.gmail.com>	<41A8D0B2.6060800@ocf.berkeley.edu>
	<ca471dc2041127130529ecdf98@mail.gmail.com>
Message-ID: <41A93061.4080701@iinet.net.au>

Guido van Rossum wrote:
> I'm not sure it needs clarifying; it's the reference manual after all,
> not a tutorial.
> 
> I'd rather let the grammar speak for itself; there's no ambiguity in
> that, and the words are just there to clarify the *semantics*.
> 

Also, I'll freely admit that I made the mistake of reading the "string literal 
concatenation" bit out of context. Reading the preceding section on string 
literals in general first (which includes the relevant snippet from the 
grammar), probably would have made it clearer what was going on.

http://www.python.org/dev/doc/devel/ref/strings.html

Cheers,
Nick.
I guess there was a reason K&R used ';' after all. . .

-- 
Nick Coghlan   |   ncoghlan@email.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://boredomandlaziness.skystorm.net
From anthony at interlink.com.au  Sun Nov 28 13:16:54 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sun Nov 28 13:17:43 2004
Subject: [Python-Dev] state of 2.4 final release
Message-ID: <200411282316.55353.anthony@interlink.com.au>

I didn't see any replies to the last post, so I'll ask again with a 
better subject line - as I said last time, as far as I'm aware, I'm
not aware of anyone having done a fix for the issue Tim identified
( http://www.python.org/sf/1069160 )

So, my question is: Is this important enough to delay a 2.4 final for?
My plan is currently to release it _this_ _Tuesday_, so I really need
an answer soon...

I've attached Tim's original message at the end here. At the moment,
I'm inclined to say "if it's not fixed, it won't kill us". But that's
admittedly my own biases - threading bugs annoy me <wink>

I'm happy to defer to more knowlegable types, though - is this so
bad that it merits delaying the release? I can't make time to look
at it before then - I'm still writing slides for a couple of talks at OSDC.

Anthony



From: Tim Peters <tim.peters@gmail.com>
To: Python Dev <python-dev@python.org>
Date: 2004-11-19 13:08


This one is a puzzler.  See

    http://www.python.org/sf/1069160

for details.  The short course is that the PyThreadState_SetAsyncExc()
implementation fell into a common trap, and can cause segfaults under
rare conditions (like every other Python thread segfault bug we've
ever had).

This is easily repaired (although I've got no interest in doing the
coding, or even in contriving a test case -- this was an obvious "by
eyeball" bug).

The puzzle is how to treat this wrt 2.4.  Since it's a critical bug, I
suppose it "should" force another release candidate.  OTOH, this is a
C-only API (there's no exposure from Python) that's never used in the
core.  We could add code to make it segfault every time <wink>, and
nothing in the distribution would notice.

OTOH, if we broke its intended behavior while fixing the bug, we'd
never know that either.  "Never used in the core" means never -- the
function isn't tested.

On the third hand, it's a simple function with an obvious segfault
mode that has an obvious fix.

I'll leave it to the release manager <wink>.
From aurora00 at gmail.com  Sun Nov 28 04:38:58 2004
From: aurora00 at gmail.com (aurora)
Date: Sun Nov 28 16:54:15 2004
Subject: [Python-Dev] server modules class diagram
Message-ID: <cbd17751041127193872244652@mail.gmail.com>

Hi I don't know if I'm sending this to the right mailing list. Anyway
I have gone through the Python server module code so many times. I
actually come up with a class diagram to help me navigate it. I wonder
if this would be useful in offical documentation. Please let me know
if this is not the right mailing list for documentations. (The class
diagram show be viewed with monospace fonts.)

Wai Yip Tung


---------------------------------------------------------------------------------------
Standard Python Modules for implementing network servers

...Some description on the collection of classes...

There are two important line of classes. The BaseServer and its
subclasses handle network connections. The BaseRequestHandler and its
subclasses handle the message.

...ForkingMixIn v.s. synchronized...

...mention asyncore, mention cgi...



                       ***************************
                       * The SocketServer module *
                       ***************************

-------------------------------
BaseServer
-------------------------------
An abstract class that defines
the main method handle_request()
with pseudo code like below:
                                     -------------------------------
def handle_request():                ForkingMixIn
  req = get_request()                -------------------------------
  if verify_request(req):            Overrides process_request() to
  try:                               fork process to handle each
    process_request(req)             request. Handles error in the
  except:                            child process.
    handle_error(req)                -------------------------------
    close_request(req)

def process_request(req):            -------------------------------
  finish_request(req)                ThreadingMixIn
  close_request(req)                 -------------------------------
                                     Overrides process_request() to
def finish_request(req):             spawn new thread to handle each
  instantiates                       request. Handles error in the
    RequestHandlerClass(req)         child thread.
                                     -------------------------------
Other than handle_request(),
all methods may be overridden.
-------------------------------
            ^
            |
-------------------------------
TCPServer
-------------------------------          -------------------------------
Creates a STREAM socket, then            BaseRequestHandler
bind() and listen() to it.               -------------------------------
                                         Abstract base class that calls
Implements get_request() to use          setup(), handle() and finish()
socket.accept()                          methods.
-------------------------------          -------------------------------
  ^         ^                                        ^
  |         |                                        |
  |   ----------------------------       -------------------------------
  |   UDPServer                          StreamRequestHandler
  |   ----------------------------       -------------------------------
  |   Creates DGRAM socket.              Implements setup() and finish()
  |                                      to create the rfile/wfile
  |   Overrides get_request() to         object from the socket
  |   use socket.recvfrom().             connection.
  |   ----------------------------       -------------------------------
  |                                                  ^
  |                                                  |
  |                                                  |
  |                                                  |
  |                *****************************     |
  |                * The BaseHTTPServer module *     |
  |                *****************************     |
  |                                                  |
  |                                                  |
-------------------------------          -------------------------------
HTTPServer                               BaseHTTPRequestHandler
-------------------------------          -------------------------------
Overrides server_bind to store           Implements handle(). Calls
the server name.                         handle_one_request() to parse
-------------------------------          HTTP request and invoke the
                                         corresponding do_XXX methods.

                                         handle_one_request() are called
                                         once for each HTTP request in a
                                         persistent connection.
                                         -------------------------------
                                                     ^
                                                     |
                                                     |
                    ************************         |
                    * The SimpleHTTPServer *         |
                    ************************         |
                                                     |
                                                     |
                                         -------------------------------
                                         SimpleHTTPRequestHandler
                                         -------------------------------
                                         Implements do_GET() to serve
                                         files mapped to URL path.
                                         -------------------------------
                                                     ^
                                                     |
                                                     |
                   ****************************      |
                   * The CGIHTTPServer module *      |
                   ****************************      |
                                                     |
                                                     |
                                         -------------------------------
                                         CGIHTTPRequestHandler
                                         -------------------------------
                                         Implements do_POST() to find,
                                         prepare and invoke CGI programs.
                                         -------------------------------


Some observations

1. One exception to the Server / RequestHandler division is the
   BaseHTTPRequestHandler.handle() method. The request handler is
   responsible to break down multiple HTTP messages from a persistent
   TCP connection (See RFC 2616 section 8.1).

2. Strictly speaking UDPServer should not be a specialization of
   TCPServer.

3. The BaseServer.finish_request() is a bit of a misnomer. From there it
   instantiates a RequestHandler where most application logic actually
   begins.

4. Note that BaseRequestHandler put most application logic in the
   __init__ method. (Unfortunately this does not give the caller much
   fine control. Also when an exception is thrown in __init__, the
   object reference is not available at all). Also notice that finish()
   is not in a finally clause and may not be called when an exception is
   thrown.
From mwh at python.net  Sun Nov 28 18:20:18 2004
From: mwh at python.net (Michael Hudson)
Date: Sun Nov 28 18:20:28 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer> (Raymond
	Hettinger's message of "Sat, 27 Nov 2004 16:10:14 -0500")
References: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer>
Message-ID: <2msm6tubzh.fsf@starship.python.net>

"Raymond Hettinger" <python@rcn.com> writes:

> Right.  Over-clarification results in docs that read like the
> instructions for the holy hand grenade ;-)

Well said.

Cheers,
mwh

-- 
  Famous remarks are very seldom quoted correctly.
                                                    -- Simeon Strunsky
From martin at v.loewis.de  Sun Nov 28 23:27:13 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun Nov 28 23:27:11 2004
Subject: [Python-Dev] state of 2.4 final release
In-Reply-To: <200411282316.55353.anthony@interlink.com.au>
References: <200411282316.55353.anthony@interlink.com.au>
Message-ID: <41AA50C1.4010102@v.loewis.de>

Anthony Baxter wrote:
> So, my question is: Is this important enough to delay a 2.4 final for?

I think we should release the code with that bug. I feel that this
async exceptions thing has many more problems in it, and we may
have to revert the entire feature.

Regards,
Martin
From gvanrossum at gmail.com  Mon Nov 29 01:55:54 2004
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Mon Nov 29 01:55:58 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <2msm6tubzh.fsf@starship.python.net>
References: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer>
	<2msm6tubzh.fsf@starship.python.net>
Message-ID: <ca471dc2041128165530532ab4@mail.gmail.com>

> > Right.  Over-clarification results in docs that read like the
> > instructions for the holy hand grenade ;-)
> 
> Well said.

Except that now I can't find the adjacent string literals in the
grammar any more!

I'm looking al http://www.python.org/dev/doc/devel/ref/grammar.txt

The path goes from primary to atom to literal to stringliteral (and
from there on into lexical detail) and nowhere does the grammar show
that multiple string literals are allowed. Adding a single + after
stringliteral in the expansion for literal would fix this. Once that
is fixed, we could probably reduce the text of the offending section
somewhat to use the phrase "where allowed by the grammar" and skip the
mentioning of different quoting conventions or intervening whitespace.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
From anthony at interlink.com.au  Mon Nov 29 02:39:20 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Mon Nov 29 02:40:48 2004
Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/email
	__init__.py, 1.34, 1.35
In-Reply-To: <E1CYa3V-0004xH-OH@sc8-pr-cvs1.sourceforge.net>
References: <E1CYa3V-0004xH-OH@sc8-pr-cvs1.sourceforge.net>
Message-ID: <200411291239.20848.anthony@interlink.com.au>

On Monday 29 November 2004 12:10, bwarsaw@users.sourceforge.net wrote:
> Update of /cvsroot/python/python/dist/src/Lib/email
> In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv19040
>
> Modified Files:
>  __init__.py
> Log Message:
> There's likely nothing more to do to the email package before Python 2.4 is
> final, so I'm marking email's version number as "3.0" (e.g. final).


Are there any other packages that have their own version numbers that
should be updated?

From anthony at interlink.com.au  Mon Nov 29 02:50:18 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Mon Nov 29 03:24:07 2004
Subject: [Python-Dev] TRUNK FREEZE for 2.4 final from 2100 UTC, 29-11-2004
Message-ID: <200411291250.18764.anthony@interlink.com.au>

Ok, we're about ready for the 2.4 final release. Please hold off any checkins
post 21:00 UTC (so in about 19-20 hours from now). 

Thanks,
Anthony
From bac at OCF.Berkeley.EDU  Mon Nov 29 03:51:27 2004
From: bac at OCF.Berkeley.EDU (Brett C.)
Date: Mon Nov 29 03:51:30 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <ca471dc2041128165530532ab4@mail.gmail.com>
References: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer>	<2msm6tubzh.fsf@starship.python.net>
	<ca471dc2041128165530532ab4@mail.gmail.com>
Message-ID: <41AA8EAF.1050600@ocf.berkeley.edu>

Guido van Rossum wrote:
>>>Right.  Over-clarification results in docs that read like the
>>>instructions for the holy hand grenade ;-)
>>
>>Well said.
> 
> 
> Except that now I can't find the adjacent string literals in the
> grammar any more!
> 
> I'm looking al http://www.python.org/dev/doc/devel/ref/grammar.txt
> 
> The path goes from primary to atom to literal to stringliteral (and
> from there on into lexical detail) and nowhere does the grammar show
> that multiple string literals are allowed. Adding a single + after
> stringliteral in the expansion for literal would fix this
[SNIP]

But if you look at Grammar/Grammar you will notice that atom goes to STRING+ 
which should cover this.  Is that grammar.txt file generated from 
Grammar/Grammar or is it done by hand?

-Brett
From gvanrossum at gmail.com  Mon Nov 29 05:32:29 2004
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Mon Nov 29 05:32:32 2004
Subject: [Python-Dev] String literal concatenation & docstrings
In-Reply-To: <41AA8EAF.1050600@ocf.berkeley.edu>
References: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer>
	<2msm6tubzh.fsf@starship.python.net>
	<ca471dc2041128165530532ab4@mail.gmail.com>
	<41AA8EAF.1050600@ocf.berkeley.edu>
Message-ID: <ca471dc204112820321dee1a2a@mail.gmail.com>

> But if you look at Grammar/Grammar you will notice that atom goes to STRING+
> which should cover this.

Of course, otherwise it wouldn't work! :-)

> Is that grammar.txt file generated from
> Grammar/Grammar or is it done by hand?

By hand. The reference manual has more detail (Grammar/Grammar doesn't
say anything about what a STRING is, or other literals) and presents
some rules in a more readable form which wouldn't work for
Grammar/Grammar given the LL1 constraints of pgen. It also chooses
more readable names. But the translation process is fallible.


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
From aahz at pythoncraft.com  Mon Nov 29 06:01:14 2004
From: aahz at pythoncraft.com (Aahz)
Date: Mon Nov 29 06:01:17 2004
Subject: [Python-Dev] server modules class diagram
In-Reply-To: <cbd17751041127193872244652@mail.gmail.com>
References: <cbd17751041127193872244652@mail.gmail.com>
Message-ID: <20041129050114.GA4293@panix.com>

On Sat, Nov 27, 2004, aurora wrote:
>
> Hi I don't know if I'm sending this to the right mailing list. Anyway
> I have gone through the Python server module code so many times. I
> actually come up with a class diagram to help me navigate it. I wonder
> if this would be useful in offical documentation. Please let me know
> if this is not the right mailing list for documentations. (The class
> diagram show be viewed with monospace fonts.)

If you look at the Python docs, the bottom of every page says, "See
About the Python Documentation for information on suggesting changes".
-- 
Aahz (aahz@pythoncraft.com)           <*>         http://www.pythoncraft.com/

WiFi is the SCSI of the 21st Century -- there are fundamental technical
reasons for sacrificing a goat.  (with no apologies to John Woods)
From anthony at interlink.com.au  Mon Nov 29 10:03:58 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Mon Nov 29 10:05:29 2004
Subject: [Python-Dev] TRUNK FREEZE for 2.4 final from 2100 UTC, 29-11-2004
In-Reply-To: <200411291250.18764.anthony@interlink.com.au>
References: <200411291250.18764.anthony@interlink.com.au>
Message-ID: <200411292004.00001.anthony@interlink.com.au>

On Monday 29 November 2004 12:50, Anthony Baxter wrote:
> Ok, we're about ready for the 2.4 final release. Please hold off any
> checkins post 21:00 UTC (so in about 19-20 hours from now).

I should also note that shortly after the release is done and we've
confirmed that there's no brown-paper-bag bugs, I'll be branching
the release24-maint branch in CVS. 

Anthony
From niemeyer at conectiva.com  Mon Nov 29 20:04:48 2004
From: niemeyer at conectiva.com (Gustavo Niemeyer)
Date: Tue Nov 30 02:36:46 2004
Subject: [Python-Dev] File encodings
Message-ID: <20041129190448.GA23399@burma.localdomain>

Greetings,

Today, while trying to internationalize a program I'm working on,
I found an interesting side-effect of how we're dealing with
encoding of unicode strings while being written to files.

Suppose the following example:

  # -*- encoding: iso-8859-1 -*-
  print u"?"

This will correctly print the string '?', as expected. Now, what
surprises me, is that the following code won't work in an equivalent
way (unless using sys.setdefaultencoding()):

  # -*- encoding: iso-8859-1 -*-
  import sys
  sys.stdout.write(u"?\n")

This will raise the following error:

  Traceback (most recent call last):
    File "asd.py", line 3, in ?
      sys.stdout.write(u"?")
  UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1'
                      in position 0:ordinal not in range(128)

This difference may become a really annoying problem when trying to
internationalize programs, since it's usual to see third-party code
dealing with sys.stdout, instead of using 'print'. The standard
optparse module, for instance, has a reference to sys.stdout which
is used in the default --help handling mechanism.

Given the fact that files have an 'encoding' parameter, and that
any unicode strings with characters not in the 0-127 range will
raise an exception if being written to files, isn't it reasonable
to respect the 'encoding' attribute whenever writing data to a
file?

The workaround for that problem is to either use the evil-considered
sys.setdefaultencoding(), or to wrap sys.stdout. IMO, both options
seem unreasonable for such a common idiom.

-- 
Gustavo Niemeyer
http://niemeyer.net
From tim.peters at gmail.com  Tue Nov 30 04:18:40 2004
From: tim.peters at gmail.com (Tim Peters)
Date: Tue Nov 30 04:19:30 2004
Subject: [Python-Dev] state of 2.4 final release
In-Reply-To: <200411282316.55353.anthony@interlink.com.au>
References: <200411282316.55353.anthony@interlink.com.au>
Message-ID: <1f7befae04112919183144973b@mail.gmail.com>

[Anthony Baxter]
> I didn't see any replies to the last post, so I'll ask again with a
> better subject line - as I said last time, as far as I'm aware, I'm
> not aware of anyone having done a fix for the issue Tim identified
> ( http://www.python.org/sf/1069160 )
> 
> So, my question is: Is this important enough to delay a 2.4 final
> for?

Not according to me; said before I'd be happy if everyone pretended I
hadn't filed that report until a month after 2.4 final was released.
From bob at redivi.com  Tue Nov 30 04:45:26 2004
From: bob at redivi.com (Bob Ippolito)
Date: Tue Nov 30 04:45:37 2004
Subject: [Python-Dev] File encodings
In-Reply-To: <20041129190448.GA23399@burma.localdomain>
References: <20041129190448.GA23399@burma.localdomain>
Message-ID: <456E4FEC-4282-11D9-A720-000A9567635C@redivi.com>

On Nov 29, 2004, at 2:04 PM, Gustavo Niemeyer wrote:

> Today, while trying to internationalize a program I'm working on,
> I found an interesting side-effect of how we're dealing with
> encoding of unicode strings while being written to files.
>
> Suppose the following example:
>
>   # -*- encoding: iso-8859-1 -*-
>   print u"?"
>
> This will correctly print the string '?', as expected. Now, what
> surprises me, is that the following code won't work in an equivalent
> way (unless using sys.setdefaultencoding()):

That doesn't work here, where sys.getdefaultencoding() is 'ascii', as 
expected.

>   # -*- encoding: iso-8859-1 -*-
>   import sys
>   sys.stdout.write(u"?\n")
>
> This will raise the following error:
>
>   Traceback (most recent call last):
>     File "asd.py", line 3, in ?
>       sys.stdout.write(u"?")
>   UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1'
>                       in position 0:ordinal not in range(128)

That's expected.

> This difference may become a really annoying problem when trying to
> internationalize programs, since it's usual to see third-party code
> dealing with sys.stdout, instead of using 'print'. The standard
> optparse module, for instance, has a reference to sys.stdout which
> is used in the default --help handling mechanism.
>
> Given the fact that files have an 'encoding' parameter, and that
> any unicode strings with characters not in the 0-127 range will
> raise an exception if being written to files, isn't it reasonable
> to respect the 'encoding' attribute whenever writing data to a
> file?

No, because you don't know it's a file.  You're calling a function with 
a unicode object.  The function doesn't know that the object was some 
unicode object that came from a source file of some particular 
encoding.

> The workaround for that problem is to either use the evil-considered
> sys.setdefaultencoding(), or to wrap sys.stdout. IMO, both options
> seem unreasonable for such a common idiom.

There's no guaranteed correlation whatsoever between the claimed 
encoding of your source document and the encoding of the user's 
terminal, why do you want there to be?  What if you have some source 
files with 'foo' encoding and others with 'bar' encoding?  What about 
ascii encoded source documents that use escape sequences to represent 
non-ascii characters?  What you want doesn't make any sense so long as 
python strings and file objects deal in bytes not characters :)

Wrapping sys.stdout is the ONLY reasonable solution.

This is the idiom that I use.  It's painless and works quite well:

import sys
import codecs
sys.stdout = codecs.getwriter('utf-8')(sys.stdout)

-bob

From martin at v.loewis.de  Tue Nov 30 09:44:05 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue Nov 30 09:44:10 2004
Subject: [Python-Dev] File encodings
In-Reply-To: <20041129190448.GA23399@burma.localdomain>
References: <20041129190448.GA23399@burma.localdomain>
Message-ID: <41AC32D5.5000707@v.loewis.de>

Gustavo Niemeyer wrote:
> Given the fact that files have an 'encoding' parameter, and that
> any unicode strings with characters not in the 0-127 range will
> raise an exception if being written to files, isn't it reasonable
> to respect the 'encoding' attribute whenever writing data to a
> file?

In general, files don't have an encoding parameter - sys.stdout
is an exception.

The reason why this works for print and not for write is that
I considered "print unicodeobject" important, and wanted to
implement that. file.write is an entirely different code path,
so it doesn't currently consider Unicode objects; instead, it
only supports strings (or, more generally, buffers).

 > This difference may become a really annoying problem when trying to
 > internationalize programs, since it's usual to see third-party code
 > dealing with sys.stdout, instead of using 'print'.

Apparently, it isn't important enough that somebody had analysed this,
and offered a patch. In any case, it would be quite unreliable to
pass unicode strings to .write even *if* .write supported .encoding,
since most files don't have .encoding. Even sys.stdout does not
always have .encoding - only when it is a terminal, and only if we
managed to find out what the encoding of the terminal is.

Regards,
Martin
From mal at egenix.com  Tue Nov 30 09:52:34 2004
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue Nov 30 09:52:28 2004
Subject: [Python-Dev] File encodings
In-Reply-To: <20041129190448.GA23399@burma.localdomain>
References: <20041129190448.GA23399@burma.localdomain>
Message-ID: <41AC34D2.1060909@egenix.com>

Gustavo Niemeyer wrote:
> Greetings,
> 
> Today, while trying to internationalize a program I'm working on,
> I found an interesting side-effect of how we're dealing with
> encoding of unicode strings while being written to files.
> 
> Suppose the following example:
> 
>   # -*- encoding: iso-8859-1 -*-
>   print u"?"
> 
> This will correctly print the string '?', as expected. Now, what
> surprises me, is that the following code won't work in an equivalent
> way (unless using sys.setdefaultencoding()):
> 
>   # -*- encoding: iso-8859-1 -*-
>   import sys
>   sys.stdout.write(u"?\n")
> 
> This will raise the following error:
> 
>   Traceback (most recent call last):
>     File "asd.py", line 3, in ?
>       sys.stdout.write(u"?")
>   UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1'
>                       in position 0:ordinal not in range(128)
> 
> This difference may become a really annoying problem when trying to
> internationalize programs, since it's usual to see third-party code
> dealing with sys.stdout, instead of using 'print'. The standard
> optparse module, for instance, has a reference to sys.stdout which
> is used in the default --help handling mechanism.

You are mixing things here:

The source encoding is meant for the
parser and defines the way Unicode literals are converted
into Unicode objects.

The encoding used on the stdout stream doesn't have anything
to do with the source code encoding and has to be handled
differently.

The idiom presented by Bob is the right way to go: wrap
sys.stdout with a StreamEncoder.

Using sys.setdefaultencoding() is *not* the right solution
to the problem.

In general when writing programs that are targetted for
i18n, you should use Unicode for all text data and
convert from Unicode to 8-bit only at the IO/UI layer.

The various wrappers in the codecs module make this
rather easy.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 30 2004)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
From anthony at interlink.com.au  Tue Nov 30 13:31:34 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Tue Nov 30 13:33:02 2004
Subject: [Python-Dev] RELEASED Python 2.4 (final)
Message-ID: <200411302331.43246.anthony@interlink.com.au>

On behalf of the Python development team and the Python community, I'm
happy to announce the release of Python 2.4.

Python 2.4 is a final, stable release, and we can recommend that Python
users upgrade to this version.

Python 2.4 is the result of almost 18 month's worth of work on top 
of Python 2.3 and represents another stage in the careful evolution 
of Python. New language features have been kept to a minimum, many 
bugs have been fixed and a wide variety of improvements have been made.

Notable changes in Python 2.4 include improvements to the importing of
modules, generator expressions, function decorators, a number of new 
modules (including subprocess, decimal and cookielib) and countless 
numbers of fixed bugs and smaller enhancements. For more, see the 
(subjective) highlights, the release notes, or Andrew Kuchling's What's 
New In Python, all available from the 2.4 web page.

    http://www.python.org/2.4/

Please log any problems you have with this release in the SourceForge
bug tracker (noting that you're using Python 2.4):

    http://sourceforge.net/bugs/?group_id=5470

Enjoy the new (stable!) release,
Anthony

Anthony Baxter
anthony@python.org
Python Release Manager
(on behalf of the entire python-dev team)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20041130/13167337/attachment.pgp
From niemeyer at conectiva.com  Tue Nov 30 13:20:13 2004
From: niemeyer at conectiva.com (Gustavo Niemeyer)
Date: Tue Nov 30 14:20:50 2004
Subject: [Python-Dev] File encodings
In-Reply-To: <456E4FEC-4282-11D9-A720-000A9567635C@redivi.com>
References: <20041129190448.GA23399@burma.localdomain>
	<456E4FEC-4282-11D9-A720-000A9567635C@redivi.com>
Message-ID: <20041130122013.GA6610@burma.localdomain>

Hello Bob,

[...]
> >Given the fact that files have an 'encoding' parameter, and that
> >any unicode strings with characters not in the 0-127 range will
> >raise an exception if being written to files, isn't it reasonable
> >to respect the 'encoding' attribute whenever writing data to a
> >file?
> 
> No, because you don't know it's a file.  You're calling a function with 
> a unicode object.  The function doesn't know that the object was some 
> unicode object that came from a source file of some particular 
> encoding.

I don't understand what you're saying here. The file knows itself
is a file. The write function knows the parameter is unicode.

> >The workaround for that problem is to either use the evil-considered
> >sys.setdefaultencoding(), or to wrap sys.stdout. IMO, both options
> >seem unreasonable for such a common idiom.
> 
> There's no guaranteed correlation whatsoever between the claimed 
> encoding of your source document and the encoding of the user's 
> terminal, why do you want there to be?  What if you have some source 

I don't. I want the write() function of file objects to respect
the encoding attribute of these objects. This is already being
done when print is used. I'm proposing to extend that behavior to
the write function. That's all.

> files with 'foo' encoding and others with 'bar' encoding?  What about 
> ascii encoded source documents that use escape sequences to represent 
> non-ascii characters?  What you want doesn't make any sense so long as 
> python strings and file objects deal in bytes not characters :)

Please, take a long breath, and read my message again. :-)

> Wrapping sys.stdout is the ONLY reasonable solution.
[...]

No, it's not. But I'm glad to know other people is also doing
workarounds for that problem.

-- 
Gustavo Niemeyer
http://niemeyer.net
From anthony at interlink.com.au  Tue Nov 30 14:23:17 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Tue Nov 30 14:24:40 2004
Subject: [Python-Dev] TRUNK UNFROZEN; release24-maint branch has been cut
Message-ID: <200412010023.17728.anthony@interlink.com.au>

I've cut the release24-maint branch, and updated the Include/patchlevel.h
on trunk and branch (trunk is now 2.5a0, branch is 2.4+)

The trunk and the branch are now both unfrozen and suitable for checkins.
The feature freeze on the trunk is lifted. Remember - if you're checking 
bugfixes into the trunk, either backport them to the branch, or else mark 
the commit message with 'bugfix candidate' or 'backport candidate' or the
like.

Next up will be a 2.3.5 release. I'm going to be travelling for a large chunk
of December (at very short notice) so it's likely that this will happen at the
start of January. If someone else wants to cut a 2.3.5 sooner than that,
please feel free to volunteer! 2.3.5 will be the last 2.3.x release, barring
some almighty cockup - the next scheduled release will be 2.4.1, which will 
probably happen around May 2005.

Anthony
and yes, I'm drinking.
From niemeyer at conectiva.com  Tue Nov 30 13:29:08 2004
From: niemeyer at conectiva.com (Gustavo Niemeyer)
Date: Tue Nov 30 14:29:48 2004
Subject: [Python-Dev] File encodings
In-Reply-To: <41AC32D5.5000707@v.loewis.de>
References: <20041129190448.GA23399@burma.localdomain>
	<41AC32D5.5000707@v.loewis.de>
Message-ID: <20041130122908.GB6610@burma.localdomain>

> Gustavo Niemeyer wrote:
> >Given the fact that files have an 'encoding' parameter, and that
> >any unicode strings with characters not in the 0-127 range will
> >raise an exception if being written to files, isn't it reasonable
> >to respect the 'encoding' attribute whenever writing data to a
> >file?
> 
> In general, files don't have an encoding parameter - sys.stdout
> is an exception.

That's the only case I'd like to solve.

If there are platforms that don't know how to set it, we could make
the encoding attribute writable, and that would allow people to
easily set it to the encoding which is deemed correct in their
systems.

> The reason why this works for print and not for write is that
> I considered "print unicodeobject" important, and wanted to
> implement that. file.write is an entirely different code path,
> so it doesn't currently consider Unicode objects; instead, it
> only supports strings (or, more generally, buffers).

I understand your reasoning behind it, and would like to extend
your idea to the write function, allowing anyone to use the common
sys.stdout idiom to implement print-like functionality (like optparse
and many others). For normal files, the absence of the encoding
parameter would ensure the current behavior.

> > This difference may become a really annoying problem when trying to
> > internationalize programs, since it's usual to see third-party code
> > dealing with sys.stdout, instead of using 'print'.
> 
> Apparently, it isn't important enough that somebody had analysed this,
> and offered a patch. In any case, it would be quite unreliable to

That's what I'm doing here! :-)

> pass unicode strings to .write even *if* .write supported .encoding,
> since most files don't have .encoding. Even sys.stdout does not always
> have .encoding - only when it is a terminal, and only if we managed to
> find out what the encoding of the terminal is.

I think that's acceptable. The encoding parameter is meant for output
streams, and Python does its best to try to find a reasonable value
for showing output strings.

Thanks for your answer and clarifications,

-- 
Gustavo Niemeyer
http://niemeyer.net
From niemeyer at conectiva.com  Tue Nov 30 14:09:34 2004
From: niemeyer at conectiva.com (Gustavo Niemeyer)
Date: Tue Nov 30 15:10:40 2004
Subject: [Python-Dev] File encodings
In-Reply-To: <41AC34D2.1060909@egenix.com>
References: <20041129190448.GA23399@burma.localdomain>
	<41AC34D2.1060909@egenix.com>
Message-ID: <20041130130934.GC6610@burma.localdomain>

[...]
> You are mixing things here:
> 
> The source encoding is meant for the
> parser and defines the way Unicode literals are converted
> into Unicode objects.
> 
> The encoding used on the stdout stream doesn't have anything
> to do with the source code encoding and has to be handled
> differently.

Sorry. I probably wasn't clear enough in my message. I understand
the issue, and I'm not discussing source encoding at all. The
only problem I'd like to solve is that of output streams not
being able to have unicode strings written.

> The idiom presented by Bob is the right way to go: wrap
> sys.stdout with a StreamEncoder.

I don't see that as a good solution, since every Python software
that is internationalizaed will have do figure out this wrapping,
introducing extra overhead unnecessarily.

> Using sys.setdefaultencoding() is *not* the right solution
> to the problem.

I understand.

> In general when writing programs that are targetted for
> i18n, you should use Unicode for all text data and
> convert from Unicode to 8-bit only at the IO/UI layer.

That's what I think as well. I just would expect that Python was
kind enough to allow me to tell which output encoding I want,
instead of wrapping the sys.stdout object with a non-native-file.

IOW, being widely necessary, handling internationalization without
wrapping sys.stdout everytime seems like a good step for a language
like Python.

> The various wrappers in the codecs module make this
> rather easy.

Thanks for the suggestion!

-- 
Gustavo Niemeyer
http://niemeyer.net
From gvanrossum at gmail.com  Tue Nov 30 18:04:33 2004
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Tue Nov 30 18:05:17 2004
Subject: [Python-Dev] RELEASED Python 2.4 (final)
In-Reply-To: <200411302331.43246.anthony@interlink.com.au>
References: <200411302331.43246.anthony@interlink.com.au>
Message-ID: <ca471dc204113009041ce40ac4@mail.gmail.com>

On Tue, 30 Nov 2004 23:31:34 +1100, Anthony Baxter
<anthony@interlink.com.au> wrote:
> On behalf of the Python development team and the Python community, I'm
> happy to announce the release of Python 2.4.

Anthony, congratulations with this smooth release! Thanks for all the
hard work. Enjoy your beer, I'm celebrating with you in spirit.

Thanks also to all the other developers who contributed to this
release, and to the many thousands of users who helped with the the
beta testing.

Let's hope for record downloads,

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
From walter at livinglogic.de  Tue Nov 30 18:43:37 2004
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Tue Nov 30 18:43:40 2004
Subject: [Python-Dev] File encodings
In-Reply-To: <20041130130934.GC6610@burma.localdomain>
References: <20041129190448.GA23399@burma.localdomain>	<41AC34D2.1060909@egenix.com>
	<20041130130934.GC6610@burma.localdomain>
Message-ID: <41ACB149.6050300@livinglogic.de>

Gustavo Niemeyer wrote:

> [...]
> 
>>You are mixing things here:
>>
>>The source encoding is meant for the
>>parser and defines the way Unicode literals are converted
>>into Unicode objects.
>>
>>The encoding used on the stdout stream doesn't have anything
>>to do with the source code encoding and has to be handled
>>differently.
> 
> Sorry. I probably wasn't clear enough in my message. I understand
> the issue, and I'm not discussing source encoding at all. The
> only problem I'd like to solve is that of output streams not
> being able to have unicode strings written.
> 
>>The idiom presented by Bob is the right way to go: wrap
>>sys.stdout with a StreamEncoder.
> 
> I don't see that as a good solution, since every Python software
> that is internationalizaed will have do figure out this wrapping,
> introducing extra overhead unnecessarily.

This wrapping is probably necessary for stateful encodings. If you
had a sys.stdout.encoding=="utf-16", print would probably add the
BOM every time a unicode object is printed. This doesn't happen if
you wrap sys.stdout in a StreamWriter.

> [...]
> That's what I think as well. I just would expect that Python was
> kind enough to allow me to tell which output encoding I want,
> instead of wrapping the sys.stdout object with a non-native-file.
> 
> IOW, being widely necessary, handling internationalization without
> wrapping sys.stdout everytime seems like a good step for a language
> like Python.

You can't have stateful encodings without something that keeps
state. The only thing that does keep state in Python is a
StreamReader/StreamWriter.

Bye,
    Walter D?rwald


From ark at acm.org  Tue Nov 30 18:49:38 2004
From: ark at acm.org (Andrew Koenig)
Date: Tue Nov 30 18:49:42 2004
Subject: [Python-Dev] Trouble installing 2.4
Message-ID: <000001c4d704$f6d4f8d0$6402a8c0@arkdesktop>

I'm using Windows XP SP2.

Uninstalled 2.3, installed 2.4 (running as me, not as administrator).
No problems so far.

Tried installing pywin32-203.win32-py2.4.exe

When I try to install it as me, it gets as far as "ready to install."  When
I click Next, it says

	Can't load Python for pre-install script

and quits, even though earlier it said it had found Python 2.4 in the
registry.

When I try to install it as Administrator, it quits immediately, saying that
it couldn't locate a Python 2.4 installation.

My hypothesis: When I install 2.4 as me, it puts it in my user registry, not
the system-wide registry, and then pywin32 can't find it.

I'm going to unstall and try again as Administrator.


From niemeyer at conectiva.com  Tue Nov 30 18:11:53 2004
From: niemeyer at conectiva.com (Gustavo Niemeyer)
Date: Tue Nov 30 19:12:28 2004
Subject: [Python-Dev] File encodings
In-Reply-To: <41ACB149.6050300@livinglogic.de>
References: <20041129190448.GA23399@burma.localdomain>
	<41AC34D2.1060909@egenix.com>
	<20041130130934.GC6610@burma.localdomain>
	<41ACB149.6050300@livinglogic.de>
Message-ID: <20041130171153.GA8780@burma.localdomain>

Hello Walter,

> >I don't see that as a good solution, since every Python software
> >that is internationalizaed will have do figure out this wrapping,
> >introducing extra overhead unnecessarily.
> 
> This wrapping is probably necessary for stateful encodings. If you
> had a sys.stdout.encoding=="utf-16", print would probably add the
> BOM every time a unicode object is printed. This doesn't happen if
> you wrap sys.stdout in a StreamWriter.

I'm not sure this is an issue for a terminal output stream, which
is the case I'm trying to find a solution for. Otherwise, Python
would already be in trouble for using this scheme in the print
statement. Can you show an example of the print statement not
working?

-- 
Gustavo Niemeyer
http://niemeyer.net
From ark-mlist at att.net  Tue Nov 30 19:16:34 2004
From: ark-mlist at att.net (Andrew Koenig)
Date: Tue Nov 30 19:16:36 2004
Subject: [Python-Dev] Trouble installing 2.4
In-Reply-To: <000001c4d704$f6d4f8d0$6402a8c0@arkdesktop>
Message-ID: <000401c4d708$ba149dc0$6402a8c0@arkdesktop>

Follow-up:  When I install Python as Administrator, all is well.  In that
case (but not when installing it as me), it asks whether I want to install
it for all users or for myself only.  I then install pywin32 and it works.

So it may be that a caveat is in order to people who do not install 2.4 as
Administrator.


From walter at livinglogic.de  Tue Nov 30 20:26:36 2004
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Tue Nov 30 20:26:40 2004
Subject: [Python-Dev] File encodings
In-Reply-To: <20041130171153.GA8780@burma.localdomain>
References: <20041129190448.GA23399@burma.localdomain>
	<41AC34D2.1060909@egenix.com>
	<20041130130934.GC6610@burma.localdomain>
	<41ACB149.6050300@livinglogic.de>
	<20041130171153.GA8780@burma.localdomain>
Message-ID: <41ACC96C.5030102@livinglogic.de>

Gustavo Niemeyer wrote:

> Hello Walter,
> 
>>>I don't see that as a good solution, since every Python software
>>>that is internationalizaed will have do figure out this wrapping,
>>>introducing extra overhead unnecessarily.
>>
>>This wrapping is probably necessary for stateful encodings. If you
>>had a sys.stdout.encoding=="utf-16", print would probably add the
>>BOM every time a unicode object is printed. This doesn't happen if
>>you wrap sys.stdout in a StreamWriter.
> 
> I'm not sure this is an issue for a terminal output stream, which
> is the case I'm trying to find a solution for. Otherwise, Python
> would already be in trouble for using this scheme in the print
> statement. Can you show an example of the print statement not
> working?

No, I can't. Python doesn't accept UTF-16 as encoding.

This works:
 > LANG=de_DE.UTF-8 python2.4
Python 2.4 (#1, Nov 30 2004, 14:16:24)
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> import sys
 >>> sys.stdout.encoding
'UTF-8'

This doesn't:
 > LANG=de_DE.UTF-16 python2.4
Python 2.4 (#1, Nov 30 2004, 14:16:24)
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> import sys
 >>> sys.stdout.encoding
'ANSI_X3.4-1968'

Bye,
    Walter D?rwald


From theller at python.net  Tue Nov 30 20:46:51 2004
From: theller at python.net (Thomas Heller)
Date: Tue Nov 30 20:45:58 2004
Subject: [Python-Dev] Python.org current docs
Message-ID: <8y8jrufo.fsf@python.net>

http://www.python.org/doc/current/
and
http://docs.python.org/

still point to 2.3.4 docs.

Thomas

From astrand at lysator.liu.se  Tue Nov 30 21:33:06 2004
From: astrand at lysator.liu.se (=?iso-8859-1?Q?Peter_=C5strand?=)
Date: Tue Nov 30 21:33:14 2004
Subject: [Python-Dev] Small subprocess patch
Message-ID: <Pine.GSO.4.51L2.0411302123220.18754@koeberg.lysator.liu.se>


I'm planning to change the signature for subprocess.call slightly:

-def call(*args, **kwargs):
+def call(*popenargs, **kwargs):

The purpose is to make it clearer that "args" in this context is not the
same as the "args" argument to the Popen constructor. Two questions:

1) Is it OK to commit changes like this on the 2.4 branch, in addition to
trunk?

2) Anyone that thinks that "kwargs" should be changed into "popenkwargs"?


/Peter ?strand <astrand@lysator.liu.se>

From martin at v.loewis.de  Tue Nov 30 22:06:28 2004
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue Nov 30 22:06:30 2004
Subject: [Python-Dev] Trouble installing 2.4
In-Reply-To: <000401c4d708$ba149dc0$6402a8c0@arkdesktop>
References: <000401c4d708$ba149dc0$6402a8c0@arkdesktop>
Message-ID: <41ACE0D4.3090507@v.loewis.de>

Andrew Koenig wrote:
> So it may be that a caveat is in order to people who do not install 2.4 as
> Administrator.

I think the trouble is not with 2.4, here - the trouble is with
installing pywin32. As you said, the installation of Python itself
went fine.

 > My hypothesis: When I install 2.4 as me, it puts it in my user
 > registry, not the system-wide registry,

I can confirm this hypothesis. In a per-user installation, the registry
settings are deliberately change for the user, not for the entire
system. Otherwise, it wouldn't be per-user. Also, the user might not
be able to write to the machine registry (unless he is a member of
the Power Users group).

 > and then pywin32 can't find it.

That sounds likely, but I cannot confirm it. If it is, it is a bug
in pywin32 (and, in turn, possibly in distutils).

Regards,
Martin
From fdrake at acm.org  Tue Nov 30 22:12:37 2004
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Tue Nov 30 22:12:59 2004
Subject: [Python-Dev] Python.org current docs
In-Reply-To: <8y8jrufo.fsf@python.net>
References: <8y8jrufo.fsf@python.net>
Message-ID: <200411301612.37180.fdrake@acm.org>

On Tuesday 30 November 2004 02:46 pm, Thomas Heller wrote:
 > http://www.python.org/doc/current/
 > and
 > http://docs.python.org/
 >
 > still point to 2.3.4 docs.


I'll be fixing that up tonight.


  -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>

From astrand at lysator.liu.se  Tue Nov 30 23:00:20 2004
From: astrand at lysator.liu.se (Peter Astrand)
Date: Tue Nov 30 23:00:27 2004
Subject: [Python-Dev] Re: Small subprocess patch
In-Reply-To: <Pine.GSO.4.51L2.0411302123220.18754@koeberg.lysator.liu.se>
References: <Pine.GSO.4.51L2.0411302123220.18754@koeberg.lysator.liu.se>
Message-ID: <Pine.GSO.4.51L2.0411302255440.21491@koeberg.lysator.liu.se>

On Tue, 30 Nov 2004, Peter ?strand wrote:

> 1) Is it OK to commit changes like this on the 2.4 branch, in addition to
> trunk?

I'm also wondering if patch 1071755 and 1071764 should go into
release24-maint:

* 1071755 makes subprocess raise TypeError if Popen is called with a
bufsize that is not an integer.

* 1071764 adds a new, small utility function.


/Peter ?strand <astrand@lysator.liu.se>