From mal at egenix.com  Tue Jan  2 23:41:15 2007
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 02 Jan 2007 23:41:15 +0100
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <20070102000242.331871E4007@bag.python.org>
References: <20070102000242.331871E4007@bag.python.org>
Message-ID: <459ADF8B.1080506@egenix.com>

On 2007-01-02 01:02, brett.cannon wrote:
> Author: brett.cannon
> Date: Tue Jan  2 01:02:41 2007
> New Revision: 53204
> 
> Added:
>    peps/trunk/pep-3108.txt   (contents, props changed)
> Modified:
>    peps/trunk/pep-0000.txt
> Log:
> Add PEP 3108: Standard Library Reorganization.
> 
>...
>
> +Open Issues
> +===========
> +
> +Consolidate dependent modules together into a single module or package?
> ...
> +Consolidate certain modules with similar themes together in a package?
> +----------------------------------------------------------------------
> ...

If you do follow this route, please take the chance to place
the whole Python stdlib under a single package. That way we'll
avoid name clashes with existing packages and modules now and
in the future.

Together with absolute imports this also improves the readability
of modules since it becomes immediately clear where the imported code
is coming from.

Note that as side-effect of this it becomes a lot harder to manipulate
PYTHONPATH to trick Python into loading a standard module from a
non-standard location, improving security and robustness of the
Python installations.

> +Packages are often used to group together modules that have a similar
> +theme but do not have any direct relationship or dependency upon each
> +other.  For Python 3.0 obvious groupings could be done since renaming
> +of various modules is already occurring.
> +
> +* collections
> +    + heapq
> +    + Queue
> +    + sets
> +    + UserDist
> +    + UserList
> +    + What to do with UserString?
> +        - Have a package for Python implementations of built-in types
> +          instead of putting the User* modules into 'collections'?
> +* mac
> +    + Various Mac-specific modules.
> +    + Same can be done for other platform-specific code.
> +* Profiling
> +    + cProfile
> +    + profile
> +    + hotshot
> +    + pstats
> +* email
> +    + mailbox
> +    + mhlib
> +* Databases
> +    + anydbm
> +    + dbhash
> +    + dbm
> +    + bsddb
> +    + dumbdbm
> +    + gdbm
> +    + whichdb
> +* Audio
> +    + aifc
> +    + audioop
> +    + chunk
> +    + ossaudiodev
> +    + sunau
> +    + wave
> +    + winsound
> +* Servers
> +    + BaseHTTPServer
> +    + CGIHTTPServer
> +    + DocXMLRPCServer
> +    + SimpleHTTPServer
> +    + SimpleXMLRPCServer
> +    + SocketServer

The package names should probably be converted to lower-case to
follow PEP 8.

Thanks and Happy New Year,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 02 2007)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From brett at python.org  Tue Jan  2 23:54:59 2007
From: brett at python.org (Brett Cannon)
Date: Tue, 2 Jan 2007 14:54:59 -0800
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <459ADF8B.1080506@egenix.com>
References: <20070102000242.331871E4007@bag.python.org>
	<459ADF8B.1080506@egenix.com>
Message-ID: <bbaeab100701021454i369192f4heb7bd63c246909d5@mail.gmail.com>

On 1/2/07, M.-A. Lemburg <mal at egenix.com> wrote:
>
> On 2007-01-02 01:02, brett.cannon wrote:
> > Author: brett.cannon
> > Date: Tue Jan  2 01:02:41 2007
> > New Revision: 53204
> >
> > Added:
> >    peps/trunk/pep-3108.txt   (contents, props changed)
> > Modified:
> >    peps/trunk/pep-0000.txt
> > Log:
> > Add PEP 3108: Standard Library Reorganization.
> >
> >...
> >
> > +Open Issues
> > +===========
> > +
> > +Consolidate dependent modules together into a single module or package?
> > ...
> > +Consolidate certain modules with similar themes together in a package?
> > +----------------------------------------------------------------------
> > ...
>
> If you do follow this route, please take the chance to place
> the whole Python stdlib under a single package. That way we'll
> avoid name clashes with existing packages and modules now and
> in the future.


That has been suggested before (including by me) and Guido has always shot
it down.  That's why I left it out of this proposal.

Together with absolute imports this also improves the readability
> of modules since it becomes immediately clear where the imported code
> is coming from.
>
> Note that as side-effect of this it becomes a lot harder to manipulate
> PYTHONPATH to trick Python into loading a standard module from a
> non-standard location, improving security and robustness of the
> Python installations.
>
> > +Packages are often used to group together modules that have a similar
> > +theme but do not have any direct relationship or dependency upon each
> > +other.  For Python 3.0 obvious groupings could be done since renaming
> > +of various modules is already occurring.
> > +
> > +* collections
> > +    + heapq
> > +    + Queue
> > +    + sets
> > +    + UserDist
> > +    + UserList
> > +    + What to do with UserString?
> > +        - Have a package for Python implementations of built-in types
> > +          instead of putting the User* modules into 'collections'?
> > +* mac
> > +    + Various Mac-specific modules.
> > +    + Same can be done for other platform-specific code.
> > +* Profiling
> > +    + cProfile
> > +    + profile
> > +    + hotshot
> > +    + pstats
> > +* email
> > +    + mailbox
> > +    + mhlib
> > +* Databases
> > +    + anydbm
> > +    + dbhash
> > +    + dbm
> > +    + bsddb
> > +    + dumbdbm
> > +    + gdbm
> > +    + whichdb
> > +* Audio
> > +    + aifc
> > +    + audioop
> > +    + chunk
> > +    + ossaudiodev
> > +    + sunau
> > +    + wave
> > +    + winsound
> > +* Servers
> > +    + BaseHTTPServer
> > +    + CGIHTTPServer
> > +    + DocXMLRPCServer
> > +    + SimpleHTTPServer
> > +    + SimpleXMLRPCServer
> > +    + SocketServer
>
> The package names should probably be converted to lower-case to
> follow PEP 8.


Oops, I should have clarified that was not package name suggestsions beyond
'collections'.  It was just meant to act as what the type of grouping was.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070102/f867294b/attachment.html 

From mal at egenix.com  Wed Jan  3 00:02:50 2007
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 03 Jan 2007 00:02:50 +0100
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <bbaeab100701021454i369192f4heb7bd63c246909d5@mail.gmail.com>
References: <20070102000242.331871E4007@bag.python.org>	<459ADF8B.1080506@egenix.com>
	<bbaeab100701021454i369192f4heb7bd63c246909d5@mail.gmail.com>
Message-ID: <459AE49A.5040704@egenix.com>

On 2007-01-02 23:54, Brett Cannon wrote:
> On 1/2/07, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>> On 2007-01-02 01:02, brett.cannon wrote:
>> > Author: brett.cannon
>> > Date: Tue Jan  2 01:02:41 2007
>> > New Revision: 53204
>> >
>> > Added:
>> >    peps/trunk/pep-3108.txt   (contents, props changed)
>> > Modified:
>> >    peps/trunk/pep-0000.txt
>> > Log:
>> > Add PEP 3108: Standard Library Reorganization.
>> >
>> >...
>> >
>> > +Open Issues
>> > +===========
>> > +
>> > +Consolidate dependent modules together into a single module or
>> package?
>> > ...
>> > +Consolidate certain modules with similar themes together in a package?
>> > +----------------------------------------------------------------------
>> > ...
>>
>> If you do follow this route, please take the chance to place
>> the whole Python stdlib under a single package. That way we'll
>> avoid name clashes with existing packages and modules now and
>> in the future.
> 
> 
> That has been suggested before (including by me) and Guido has always shot
> it down.  That's why I left it out of this proposal.

Even if it is shot down again, it still deserves to be documented
together with the reasons for being shot down.

This is a one-in-a-lifetime chance, so it would be sad if it were
not taken into account.

The extra effort would be minimal - the renaming would have to be
done using a script anyway and adding an extra 'from py import '
prefix to the modules wouldn't really make the renaming more
complicated ;-)

> Together with absolute imports this also improves the readability
>> of modules since it becomes immediately clear where the imported code
>> is coming from.
>>
>> Note that as side-effect of this it becomes a lot harder to manipulate
>> PYTHONPATH to trick Python into loading a standard module from a
>> non-standard location, improving security and robustness of the
>> Python installations.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 02 2007)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From brett at python.org  Wed Jan  3 00:09:44 2007
From: brett at python.org (Brett Cannon)
Date: Tue, 2 Jan 2007 15:09:44 -0800
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <459AE49A.5040704@egenix.com>
References: <20070102000242.331871E4007@bag.python.org>
	<459ADF8B.1080506@egenix.com>
	<bbaeab100701021454i369192f4heb7bd63c246909d5@mail.gmail.com>
	<459AE49A.5040704@egenix.com>
Message-ID: <bbaeab100701021509x2a3a3a8fr5cded6b199a0f88b@mail.gmail.com>

On 1/2/07, M.-A. Lemburg <mal at egenix.com> wrote:
>
> On 2007-01-02 23:54, Brett Cannon wrote:
> > On 1/2/07, M.-A. Lemburg <mal at egenix.com> wrote:
> >>
> >> On 2007-01-02 01:02, brett.cannon wrote:
> >> > Author: brett.cannon
> >> > Date: Tue Jan  2 01:02:41 2007
> >> > New Revision: 53204
> >> >
> >> > Added:
> >> >    peps/trunk/pep-3108.txt   (contents, props changed)
> >> > Modified:
> >> >    peps/trunk/pep-0000.txt
> >> > Log:
> >> > Add PEP 3108: Standard Library Reorganization.
> >> >
> >> >...
> >> >
> >> > +Open Issues
> >> > +===========
> >> > +
> >> > +Consolidate dependent modules together into a single module or
> >> package?
> >> > ...
> >> > +Consolidate certain modules with similar themes together in a
> package?
> >> >
> +----------------------------------------------------------------------
> >> > ...
> >>
> >> If you do follow this route, please take the chance to place
> >> the whole Python stdlib under a single package. That way we'll
> >> avoid name clashes with existing packages and modules now and
> >> in the future.
> >
> >
> > That has been suggested before (including by me) and Guido has always
> shot
> > it down.  That's why I left it out of this proposal.
>
> Even if it is shot down again, it still deserves to be documented
> together with the reasons for being shot down.


Aw, but that means I have to go find why Guido didn't like it.  =)  But yes,
it should be either an open issue or rejected idea.

This is a one-in-a-lifetime chance, so it would be sad if it were
> not taken into account.
>
> The extra effort would be minimal - the renaming would have to be
> done using a script anyway and adding an extra 'from py import '
> prefix to the modules wouldn't really make the renaming more
> complicated ;-)


=)

> Together with absolute imports this also improves the readability
> >> of modules since it becomes immediately clear where the imported code
> >> is coming from.
> >>
> >> Note that as side-effect of this it becomes a lot harder to manipulate
> >> PYTHONPATH to trick Python into loading a standard module from a
> >> non-standard location, improving security and robustness of the
> >> Python installations.


Good point.  Could actually have the py namespace be special and use a
separate path list (e.g., sys.library_path) for that specific namespace.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070102/9c876504/attachment.html 

From barry at python.org  Wed Jan  3 00:35:11 2007
From: barry at python.org (Barry Warsaw)
Date: Tue, 2 Jan 2007 18:35:11 -0500
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <459ADF8B.1080506@egenix.com>
References: <20070102000242.331871E4007@bag.python.org>
	<459ADF8B.1080506@egenix.com>
Message-ID: <A92C8848-9DF2-4EF5-A51D-88F87D19B6F4@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 2, 2007, at 5:41 PM, M.-A. Lemburg wrote:

> Note that as side-effect of this it becomes a lot harder to manipulate
> PYTHONPATH to trick Python into loading a standard module from a
> non-standard location, improving security and robustness of the
> Python installations.

Sometimes though you want to do this, as when you want your  
application to ensure it gets a particular version of a standard  
library module, regardless of the version of Python being used.  And  
now we're back to application-specific site-packages ;).

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRZrsL3EjvBPtnXfVAQKVnAQAkJBlZ0nijuD062qu1Z97WTt0To07nLEw
Bq4fWsdQ1OCmBq7SREnLup/pnu17N0zEvqP30sRan1+C9Tj4rj22Ohy1tBBqQ0Fc
Bn7AI634gAt0n05bM3u5RErkj1SUqFksxExcarFVHwVT929e2ljiqUngr8OHHYSk
KaEO/3OhPjg=
=J34N
-----END PGP SIGNATURE-----

From brett at python.org  Wed Jan  3 01:42:01 2007
From: brett at python.org (Brett Cannon)
Date: Tue, 2 Jan 2007 16:42:01 -0800
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <459AE49A.5040704@egenix.com>
References: <20070102000242.331871E4007@bag.python.org>
	<459ADF8B.1080506@egenix.com>
	<bbaeab100701021454i369192f4heb7bd63c246909d5@mail.gmail.com>
	<459AE49A.5040704@egenix.com>
Message-ID: <bbaeab100701021642o4bce08ffy729c15507c4e2c55@mail.gmail.com>

On 1/2/07, M.-A. Lemburg <mal at egenix.com> wrote:
>
> On 2007-01-02 23:54, Brett Cannon wrote:
> > On 1/2/07, M.-A. Lemburg <mal at egenix.com> wrote:
> >>
> >> On 2007-01-02 01:02, brett.cannon wrote:
> >> > Author: brett.cannon
> >> > Date: Tue Jan  2 01:02:41 2007
> >> > New Revision: 53204
> >> >
> >> > Added:
> >> >    peps/trunk/pep-3108.txt   (contents, props changed)
> >> > Modified:
> >> >    peps/trunk/pep-0000.txt
> >> > Log:
> >> > Add PEP 3108: Standard Library Reorganization.
> >> >
> >> >...
> >> >
> >> > +Open Issues
> >> > +===========
> >> > +
> >> > +Consolidate dependent modules together into a single module or
> >> package?
> >> > ...
> >> > +Consolidate certain modules with similar themes together in a
> package?
> >> >
> +----------------------------------------------------------------------
> >> > ...
> >>
> >> If you do follow this route, please take the chance to place
> >> the whole Python stdlib under a single package. That way we'll
> >> avoid name clashes with existing packages and modules now and
> >> in the future.
> >
> >
> > That has been suggested before (including by me) and Guido has always
> shot
> > it down.  That's why I left it out of this proposal.
>
> Even if it is shot down again, it still deserves to be documented
> together with the reasons for being shot down.
>
> This is a one-in-a-lifetime chance, so it would be sad if it were
> not taken into account.
>
> The extra effort would be minimal - the renaming would have to be
> done using a script anyway and adding an extra 'from py import '
> prefix to the modules wouldn't really make the renaming more
> complicated ;-)


I was about to start writing an open issue on this since the biggest
objection from Guido I could find on this topic is
http://mail.python.org/pipermail/python-dev/2002-July/026409.html , but then
it started to feel like a separate PEP to me.  So I think I am going to pass
on taking on this topic and let someone else tackle it in a PEP.  Sorry,
MAL, but I need to worry about my sanity on this one.  =)

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070102/bbeb9ef9/attachment.html 

From dustin at v.igoro.us  Wed Jan  3 06:27:40 2007
From: dustin at v.igoro.us (dustin at v.igoro.us)
Date: Tue, 2 Jan 2007 23:27:40 -0600
Subject: [Python-Dev] smarter temporary file object (SF #415692)
Message-ID: <20070103052739.GF19957@v.igoro.us>

I came across a complaint that PEP 0042 had become a graveyard of
neglected ideas, and decided to have a look through and implement
something.  Creating a smarter temporary file object seemed simple
enough.

Oddly, even after GvR re-opened it, I can't post an attachment to that
tracker item (it's under "Feature Requests" -- does it need to get moved
to "Patches" first?), but the implementation is short, so it's included
below.  This is intended to be appended to Lib/tempfile.py (and thus
assumes that module's globals are present).

I would appreciate it if the gurus of python-dev could take a peek and
let me know if this is unsuitable or incorrect for any reason.  It's not
the most straightforward implementatio -- I used the optimization
techniques I found in TemporaryFile.

If this looks good, I'll prepare a patch against trunk, including an
additional chunk of documentation and a unit test.

Dustin

                     -----cut-here-----

try:
  from cStringIO import StringIO
except:
  from StringIO import StringIO

class SpooledTemporaryFile:
    """Temporary file wrapper, specialized to switch from
    StringIO to a real file when it exceeds a certain size or
    when a fileno is needed.
    """
    _rolled = False

    def __init__(self, max_size=0, mode='w+b', bufsize=-1,
                 suffix="", prefix=template, dir=None):
        self._file = StringIO()
        self._max_size = max_size
        self._TemporaryFileArgs = (mode, bufsize, suffix, prefix, dir)

    def _check(self, file):
        if self._rolled: return
        if file.tell() > self.__dict__['_max_size']:
            self._rollover(file)

    def _rollover(self, file):
        args = self.__dict__['_TemporaryFileArgs']
        self.__dict__.clear() # clear attributes cached by __getattr__
        newfile = self._file = TemporaryFile(*args)
        newfile.write(file.getvalue())
        newfile.seek(file.tell(), 0)
        self._rolled = True
        
        # replace patched functions with the new file's methods
        self.write = newfile.write
        self.writelines = newfile.writelines
        self.fileno = newfile.fileno

    def write(self, s):
        file = self.__dict__['_file']
        rv = file.write(s)
        self._check(file)
        return rv

    def writelines(self, iterable):
        file = self.__dict__['_file']
        rv = file.writelines(iterable)
        self._check(file)
        return rv

    def fileno(self):
        self._rollover(self.__dict__['_file'])
        return self.fileno()

    def __getattr__(self, name):
        file = self.__dict__['_file']
        a = getattr(file, name)
        if type(a) != type(0):
            setattr(self, name, a)
        return a

From nnorwitz at gmail.com  Wed Jan  3 07:07:58 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 2 Jan 2007 22:07:58 -0800
Subject: [Python-Dev] smarter temporary file object (SF #415692)
In-Reply-To: <20070103052739.GF19957@v.igoro.us>
References: <20070103052739.GF19957@v.igoro.us>
Message-ID: <ee2a432c0701022207s7b361842jdc6acd11f8bef4a0@mail.gmail.com>

On 1/2/07, dustin at v.igoro.us <dustin at v.igoro.us> wrote:
> I came across a complaint that PEP 0042 had become a graveyard of
> neglected ideas, and decided to have a look through and implement
> something.  Creating a smarter temporary file object seemed simple
> enough.
>
> Oddly, even after GvR re-opened it, I can't post an attachment to that
> tracker item

Hi Dustin.

Thanks for your patch!

Unfortunately, SF doesn't allow anyone except developers and the
tracker item owner to submit attachments.  The best thing to do would
be to create a new patch.  Then you can make a comment in the old RFE
that references your new patch.  Something like this is sufficient:

  Patch available here:  http://python.org/sf/NEW_PATCH_NUMBER

Cheers,
n

From mal at egenix.com  Wed Jan  3 12:07:48 2007
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 03 Jan 2007 12:07:48 +0100
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <A92C8848-9DF2-4EF5-A51D-88F87D19B6F4@python.org>
References: <20070102000242.331871E4007@bag.python.org>	<459ADF8B.1080506@egenix.com>
	<A92C8848-9DF2-4EF5-A51D-88F87D19B6F4@python.org>
Message-ID: <459B8E84.3040000@egenix.com>

On 2007-01-03 00:35, Barry Warsaw wrote:
> On Jan 2, 2007, at 5:41 PM, M.-A. Lemburg wrote:
> 
>> Note that as side-effect of this it becomes a lot harder to manipulate
>> PYTHONPATH to trick Python into loading a standard module from a
>> non-standard location, improving security and robustness of the
>> Python installations.
> 
> Sometimes though you want to do this, as when you want your application
> to ensure it gets a particular version of a standard library module,
> regardless of the version of Python being used.  And now we're back to
> application-specific site-packages ;).

Well, I guess that's a rather particular use case and can probably
only be safely implemented by the maintainer of the module or package
in question ;-)

In such (rare) cases, it should be possible to use one of the harder
ways to achieve this:

 * monkey patching the package
 * using package.__path__ to redirect the in-package search
 * creating a private copy of the whole package which then has
   the modified modules and packages in place

Regarding application specific package setups:

In my experience it's better to have an application specific
sys.path setup function that manages this, rather than trying
to manipulate PYTHONPATH or trying to tweak Python's stdlib
site.py into using some particular way of setting up application
specific paths which then makes interop harder for all
applications using Python, rather than just the few that
require such setups.

The application can then call this path setup function early
on in the startup phase to make sure that the rest of the
startup and the application's main code then imports the
right modules and packages.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 03 2007)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From barry at python.org  Wed Jan  3 14:12:56 2007
From: barry at python.org (Barry Warsaw)
Date: Wed, 3 Jan 2007 08:12:56 -0500
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <459B8E84.3040000@egenix.com>
References: <20070102000242.331871E4007@bag.python.org>	<459ADF8B.1080506@egenix.com>
	<A92C8848-9DF2-4EF5-A51D-88F87D19B6F4@python.org>
	<459B8E84.3040000@egenix.com>
Message-ID: <B0FC8AF4-8730-468B-A743-2B6A9419BC02@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 3, 2007, at 6:07 AM, M.-A. Lemburg wrote:

> Regarding application specific package setups:
>
> In my experience it's better to have an application specific
> sys.path setup function that manages this, rather than trying
> to manipulate PYTHONPATH or trying to tweak Python's stdlib
> site.py into using some particular way of setting up application
> specific paths which then makes interop harder for all
> applications using Python, rather than just the few that
> require such setups.
>
> The application can then call this path setup function early
> on in the startup phase to make sure that the rest of the
> startup and the application's main code then imports the
> right modules and packages.

Oh, I totally agree MAL.  It's what I've done in Mailman for ages.   
What makes it more complicated is when you have dozens of entry  
points (read: command line scripts), but it's solvable.

I guess when I read "PYTHONPATH" I also read "sys.path", so now that  
I've slept a little, never mind!

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRZur2HEjvBPtnXfVAQIZjwP/Zbcz/aUooJtca/apUSkHwfhnTvMOLiiQ
uoWOltYJnwqy3S9EYpUoan0rXBVPd04ygWf9tgZiioTaVHAuXYTLL7SikpiQTxge
VxLQA6AegHlGMFgtuqTKNYDNeG2B9dlpHbT05ZSVqVaBeWi6E/ap/NNk7ufZ//pD
3F94t07yDaE=
=OHMX
-----END PGP SIGNATURE-----

From martin at v.loewis.de  Wed Jan  3 17:06:05 2007
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 03 Jan 2007 17:06:05 +0100
Subject: [Python-Dev] Renaming Include/object.h
Message-ID: <459BD46D.6090505@v.loewis.de>

In #1626545, Anton Tropashko requests that object.h should be
renamed, because it causes conflicts with other software.

I would like to comply with this requests for 2.6, assuming there
shouldn't be many problems with existing software as object.h
shouldn't be included directly, anyway.

What do you think?

Regards,
Martin

From fdrake at acm.org  Wed Jan  3 17:20:12 2007
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Wed, 3 Jan 2007 11:20:12 -0500
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <459BD46D.6090505@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de>
Message-ID: <200701031120.13083.fdrake@acm.org>

On Wednesday 03 January 2007 11:06, Martin v. L?wis wrote:
 > In #1626545, Anton Tropashko requests that object.h should be
 > renamed, because it causes conflicts with other software.
 >
 > I would like to comply with this requests for 2.6, assuming there
 > shouldn't be many problems with existing software as object.h
 > shouldn't be included directly, anyway.

+1


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From guido at python.org  Wed Jan  3 18:38:35 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Jan 2007 09:38:35 -0800
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <200701031120.13083.fdrake@acm.org>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
Message-ID: <ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>

On 1/3/07, Fred L. Drake, Jr. <fdrake at acm.org> wrote:
> On Wednesday 03 January 2007 11:06, Martin v. L?wis wrote:
>  > In #1626545, Anton Tropashko requests that object.h should be
>  > renamed, because it causes conflicts with other software.
>  >
>  > I would like to comply with this requests for 2.6, assuming there
>  > shouldn't be many problems with existing software as object.h
>  > shouldn't be included directly, anyway.
>
> +1

Maybe this should be done in a more systematic fashion? E.g. by giving
all "internal" header files a "py_" prefix?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From thomas at python.org  Wed Jan  3 19:11:08 2007
From: thomas at python.org (Thomas Wouters)
Date: Wed, 3 Jan 2007 19:11:08 +0100
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
Message-ID: <9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>

On 1/3/07, Guido van Rossum <guido at python.org> wrote:
>
> On 1/3/07, Fred L. Drake, Jr. <fdrake at acm.org> wrote:
> > On Wednesday 03 January 2007 11:06, Martin v. L?wis wrote:
> >  > In #1626545, Anton Tropashko requests that object.h should be
> >  > renamed, because it causes conflicts with other software.
> >  >
> >  > I would like to comply with this requests for 2.6, assuming there
> >  > shouldn't be many problems with existing software as object.h
> >  > shouldn't be included directly, anyway.
> >
> > +1
>
> Maybe this should be done in a more systematic fashion? E.g. by giving
> all "internal" header files a "py_" prefix?


I was thinking the same, and I'm sure Neal Norwitz is/was too (he suggested
this a few times in the past, at least outside of python-dev.) There are a
few headers that might be in 'legitimate' use right now (as in, there is no
way to do what they need to do without including those seemingly internal
headers) but personally I think breaking source compatibility and requiring
portable code that needs access to those to #if/#ifdef around it, to be a
reasonable price to pay. (Only for header files that should really be
internal, of course, not ones that are oft-used outside the core.)

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070103/172131af/attachment.html 

From fdrake at acm.org  Wed Jan  3 19:20:19 2007
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Wed, 3 Jan 2007 13:20:19 -0500
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
Message-ID: <200701031320.19579.fdrake@acm.org>

On Wednesday 03 January 2007 12:38, Guido van Rossum wrote:
 > Maybe this should be done in a more systematic fashion? E.g. by giving
 > all "internal" header files a "py_" prefix?

Even better.

+42


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From skip at pobox.com  Wed Jan  3 19:20:54 2007
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 3 Jan 2007 12:20:54 -0600
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
Message-ID: <17819.62470.160358.96829@montanaro.dyndns.org>

    >> > In #1626545, Anton Tropashko requests that object.h should be
    >> > renamed, because it causes conflicts with other software.
    ...
    Guido> Maybe this should be done in a more systematic fashion? E.g. by
    Guido> giving all "internal" header files a "py_" prefix?

Grand Renaming, part deux? ;-)

Skip

From martin at v.loewis.de  Wed Jan  3 20:29:11 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 03 Jan 2007 20:29:11 +0100
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
Message-ID: <459C0407.3040602@v.loewis.de>

Guido van Rossum schrieb:
> Maybe this should be done in a more systematic fashion? E.g. by giving
> all "internal" header files a "py_" prefix?

Yet another alternative would be to move all such header files into a
py/ directory, so you would refer to them as

#include "py/object.h"

Any preferences?

Regards,
Martin

From fdrake at acm.org  Wed Jan  3 20:42:11 2007
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Wed, 3 Jan 2007 14:42:11 -0500
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <459C0407.3040602@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<459C0407.3040602@v.loewis.de>
Message-ID: <200701031442.11825.fdrake@acm.org>

On Wednesday 03 January 2007 14:29, Martin v. L?wis wrote:
 > Yet another alternative would be to move all such header files into a
 > py/ directory, so you would refer to them as
 >
 > #include "py/object.h"
 >
 > Any preferences?

None here; the goal is the only part I care about.


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From brett at python.org  Wed Jan  3 20:50:05 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 3 Jan 2007 11:50:05 -0800
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <459C0407.3040602@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<459C0407.3040602@v.loewis.de>
Message-ID: <bbaeab100701031150r4ecfe8d2kf4c506b939b6b5fe@mail.gmail.com>

On 1/3/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>
> Guido van Rossum schrieb:
> > Maybe this should be done in a more systematic fashion? E.g. by giving
> > all "internal" header files a "py_" prefix?
>
> Yet another alternative would be to move all such header files into a
> py/ directory, so you would refer to them as
>
> #include "py/object.h"
>
> Any preferences?


Whatever is easier to make this happen.  +1 for either solution from me.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070103/586a7c93/attachment.html 

From martin at v.loewis.de  Wed Jan  3 20:53:56 2007
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 03 Jan 2007 20:53:56 +0100
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>
References: <459BD46D.6090505@v.loewis.de>
	<200701031120.13083.fdrake@acm.org>	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>
Message-ID: <459C09D4.5000105@v.loewis.de>

Thomas Wouters schrieb:
> (Only for header
> files that should really be internal, of course, not ones that are
> oft-used outside the core.)

Which are these?

Regards,
Martin

From barry at python.org  Wed Jan  3 20:54:34 2007
From: barry at python.org (Barry Warsaw)
Date: Wed, 3 Jan 2007 14:54:34 -0500
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <459C0407.3040602@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<459C0407.3040602@v.loewis.de>
Message-ID: <B0BBCF2C-CAB8-4B55-ADE5-6387C84834E4@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 3, 2007, at 2:29 PM, Martin v. L?wis wrote:

> Guido van Rossum schrieb:
>> Maybe this should be done in a more systematic fashion? E.g. by  
>> giving
>> all "internal" header files a "py_" prefix?
>
> Yet another alternative would be to move all such header files into a
> py/ directory, so you would refer to them as
>
> #include "py/object.h"
>
> Any preferences?

I think I prefer this, although I'd choose "python/object.h" just for  
explicitness.  But if you go with a header prefix, then the shorter  
"py_" is fine.

FWIW, I tried to do a quick grep around some of our code and I found  
that the only "internal" header we include is structmember.h.  Why is  
that not part of Python.h again?

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRZwJ+nEjvBPtnXfVAQL55wP/SxjN9/ncb86KnhRMUf24U6fp+u5JrpN3
irJfi/3tf9iXFtHXPkvkc4hEM9DkF8pa+jYDICG1pZ2J0YQD/AcSuB52WoWDwBtn
BIKt1QmJvxgWZLW+dAekWhgSD95laPw+72iCwBvFlIP3+IXtF/Fw9AtuxiGwQzE3
R71j5tZAde4=
=nA9B
-----END PGP SIGNATURE-----

From greg at electricrain.com  Wed Jan  3 23:17:48 2007
From: greg at electricrain.com (Gregory P. Smith)
Date: Wed, 3 Jan 2007 14:17:48 -0800
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <B0BBCF2C-CAB8-4B55-ADE5-6387C84834E4@python.org>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<459C0407.3040602@v.loewis.de>
	<B0BBCF2C-CAB8-4B55-ADE5-6387C84834E4@python.org>
Message-ID: <20070103221748.GV24758@electricrain.com>

On Wed, Jan 03, 2007 at 02:54:34PM -0500, Barry Warsaw wrote:
> On Jan 3, 2007, at 2:29 PM, Martin v. L?wis wrote:
> 
> > Guido van Rossum schrieb:
> >> Maybe this should be done in a more systematic fashion? E.g. by  
> >> giving all "internal" header files a "py_" prefix?
> >
> > Yet another alternative would be to move all such header files into a
> > py/ directory, so you would refer to them as
> >
> > #include "py/object.h"
> >
> > Any preferences?
> 
> I think I prefer this, although I'd choose "python/object.h" just for  
> explicitness.  But if you go with a header prefix, then the shorter  
> "py_" is fine.
> 
> FWIW, I tried to do a quick grep around some of our code and I found  
> that the only "internal" header we include is structmember.h.  Why is  
> that not part of Python.h again?
> 
> -Barry

+1   on using the python/*.h subdirectory.
+0.2 on renaming only the whined about .h file.
+0.1 on using a py_ prefix for all .h files.

I prefer the python/*.h subdirectory method.  py_ in the filename is
ugly and annoying even if it does solve the immediate issue.  Other
packages that install header files commonly put them all within a
named subdirectory.

-Greg

From Jack.Jansen at cwi.nl  Thu Jan  4 00:09:40 2007
From: Jack.Jansen at cwi.nl (Jack Jansen)
Date: Thu, 4 Jan 2007 00:09:40 +0100
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <20070103221748.GV24758@electricrain.com>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<459C0407.3040602@v.loewis.de>
	<B0BBCF2C-CAB8-4B55-ADE5-6387C84834E4@python.org>
	<20070103221748.GV24758@electricrain.com>
Message-ID: <F09C491D-1A92-4385-A6BA-7EE821EC00DF@cwi.nl>


On 3-Jan-2007, at 23:17 , Gregory P. Smith wrote:
> +1   on using the python/*.h subdirectory.

I'm a bit concerned about the "python/*.h": could it cause trouble in  
combination with Apple's framework naming convention (#include  
<QuickTime/Quicktime.h> magically gets the header out of the  
quicktime framework) and the case-insensitiveness of the Mac filesystem?

Will
	#include "python/blabla.h"
always find that file along the -I paths, and not somehow  
accidentally start looking for /Library/Framework/Python.framework/ 
Headers/blabla.h?
--
Jack Jansen, <Jack.Jansen at cwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma  
Goldman



From thomas at python.org  Thu Jan  4 00:31:45 2007
From: thomas at python.org (Thomas Wouters)
Date: Thu, 4 Jan 2007 00:31:45 +0100
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <459C09D4.5000105@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>
	<459C09D4.5000105@v.loewis.de>
Message-ID: <9e804ac0701031531n446c8ecax278793c18c0d491@mail.gmail.com>

On 1/3/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>
> Thomas Wouters schrieb:
> > (Only for header
> > files that should really be internal, of course, not ones that are
> > oft-used outside the core.)
>
> Which are these?


Mostly structmember.h and structseq.h, less often code.h, compile.h,
frameobject.h, marshal.h. Aside from the ones that are already prefixed with
'py' and not included by Python.h (pythread.h, pyexpat.h maybe?) I'm not
sure which should really be public. Both structmember.h and structseq.h are
generally useful to extension modules -- I've used both, although I don't
think I would have used structseq if I wasn't already quite aware of it. The
rest is useful only for extension modules that want to muck about with
internals (like profilers, debuggers, custom marshalmunching and nasty
extension modules that want to hook into Python internals that are not
easily hooked into) and they will have to deal with less backward
compatibility at the source and binary level anyway -- let them fix their
imports.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070104/04dfa7a2/attachment.htm 

From nnorwitz at gmail.com  Thu Jan  4 05:28:36 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 3 Jan 2007 20:28:36 -0800
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>
Message-ID: <ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>

On 1/3/07, Thomas Wouters <thomas at python.org> wrote:
>
>
> On 1/3/07, Guido van Rossum <guido at python.org> wrote:
> > On 1/3/07, Fred L. Drake, Jr. <fdrake at acm.org> wrote:
> > > On Wednesday 03 January 2007 11:06, Martin v. L?wis wrote:
> > >  > In #1626545, Anton Tropashko requests that object.h should be
> > >  > renamed, because it causes conflicts with other software.
> > >  >
> > >  > I would like to comply with this requests for 2.6, assuming there
> > >  > shouldn't be many problems with existing software as object.h
> > >  > shouldn't be included directly, anyway.
> > >
> > > +1
> >
> > Maybe this should be done in a more systematic fashion? E.g. by giving
> > all "internal" header files a "py_" prefix?
>
> I was thinking the same, and I'm sure Neal Norwitz is/was too (he suggested
> this a few times in the past, at least outside of python-dev.)

Wow, I didn't realize I was that much of a broken record. :-)
I don't even remember talking to Thomas about it, only Guido.  I
definitely would like to see all private header files clearly denoted
by their name or directory.

I saw Jack's comment about Apple's naming scheme, but I'm ignoring
that for the moment.
I have bad memories from the Motif days of including everything with
one file.  I prefer to see includes with the directory.  This provides
a sort of namespace:

#include "python/foo.h"

Though, if the user only has to include a single Python.h like
currently, this wouldn't make as much sense.  I don't recall the same
problems in Python that existed when using Motif.

Here are some options (python/ can be omitted too):

  #include "python/public.h"
  #include "python/internal/foo.h"
  #include "python/private/foo.h"
  #include "python/_private.h"

I don't really like prefixing with py_ because from a user's
perspective I interpret py_ to be a public header that gives me a
namespace.  I prefer a directory that indicates its intent because it
can't be misunderstood.  IIRC Guido didn't want to introduce a new
directory.  In that case my second choice is to prefix the filename
with an underscore as a leading underscore often is interpreted as
private.

Going along with this change I would like to see all identifiers in
public header files prefixed with [_]Py.  And public header files
shouldn't be able to include private header files (by convention).  Or
we should have some other way of denoting which files must prefix all
their identifiers and which can use anything because they are only
used in Python code.

For example, right now node (struct _node) is exported in node.h in
2.4.  I think it moved in 2.5, but it's still exported IIRC.

n

From nnorwitz at gmail.com  Thu Jan  4 07:59:15 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 3 Jan 2007 22:59:15 -0800
Subject: [Python-Dev] 2.5.1 plans
In-Reply-To: <ee2a432c0612252213h5444d981ub5bed522896302bd@mail.gmail.com>
References: <ee2a432c0612252213h5444d981ub5bed522896302bd@mail.gmail.com>
Message-ID: <ee2a432c0701032259s3d5c6ae7ned7cb3a22946e83b@mail.gmail.com>

The current schedule looks like it's shaping up to be:

Wed, Jan 24 for 2.5.1c1
Wed Jan 31 for 2.5.1

It would be great if you could comment on some of the bug reports
below.  I think several already have patches/suggested fixes.

These looks like they would be nice to fix::

    http://python.org/sf/1579370 Segfault provoked by generators and exceptions
    http://python.org/sf/1377858 segfaults when using __del__ and weakrefs

There is one outstanding issue I was notified of in private mail::

    http://python.org/sf/1568240 Tix is not included in 2.5 for Windows

It's not clear to me if this should be fixed, but it's got a high priority::

    http://python.org/sf/1467929 %-formatting and dicts

n
--
On 12/25/06, Neal Norwitz <nnorwitz at gmail.com> wrote:
> I don't have a schedule in mind for 2.5.1 yet, however we should start
> preparing for it.  The release will probably happen sometime in
> January if everyone is available.  The branch has been pretty quiet,
> so I'm not expecting too many problems.
>
> Once we figure out a date for release I'll follow up here.
>
> If you have any bugs or patches that should be addressed prior to
> release, please assign them to me and bump the priority to 9.  I'm not
> sure that there are any bugs which absolutely must be fixed before
> release, though there are a few that would be nice.
>
> Any other discussion for 2.5.1 necessary?
>
> n
>

From martin at v.loewis.de  Thu Jan  4 08:15:47 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 04 Jan 2007 08:15:47 +0100
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>	
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>	
	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>
	<ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>
Message-ID: <459CA9A3.8070601@v.loewis.de>

Neal Norwitz schrieb:
> Wow, I didn't realize I was that much of a broken record. :-)
> I don't even remember talking to Thomas about it, only Guido.  I
> definitely would like to see all private header files clearly denoted
> by their name or directory.

What is a private header file, and does Python have any?

I can see why Modules/sre.h is "private": it won't get installed at
all, so users can't include them. For everything in Include, I think
users can, and will, include them directly, unless they get them
through Python.h.

Regards,
Martin

From nnorwitz at gmail.com  Thu Jan  4 08:26:01 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 3 Jan 2007 23:26:01 -0800
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <459CA9A3.8070601@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>
	<ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>
	<459CA9A3.8070601@v.loewis.de>
Message-ID: <ee2a432c0701032326r334edb12v11e9e75def70b59@mail.gmail.com>

On 1/3/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Neal Norwitz schrieb:
> > Wow, I didn't realize I was that much of a broken record. :-)
> > I don't even remember talking to Thomas about it, only Guido.  I
> > definitely would like to see all private header files clearly denoted
> > by their name or directory.
>
> What is a private header file, and does Python have any?
>
> I can see why Modules/sre.h is "private": it won't get installed at
> all, so users can't include them. For everything in Include, I think
> users can, and will, include them directly, unless they get them
> through Python.h.

By private, I mean internal only to python and don't need to prefix
their identifiers with Py and are subject to change without backwards
compatibility.  Include/graminit.h is one example of what I mean.
Some others are:  bitset.h, grammar.h, opcode.h, metagrammar.h,
errcode.h

Others are kinda questionable (they have some things that are
definitely public, others I'm not so sure about):  code.h, parsetok.h,
pyarena.h, longintrepr.h, osdefs.h, pgen.h, node.h

These are just some examples of what I mean.  These may be in include
because they are used between two top level directories, but not meant
to be exported.  There could be other reasons too.

n

From martin at v.loewis.de  Thu Jan  4 08:47:28 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 04 Jan 2007 08:47:28 +0100
Subject: [Python-Dev] Private header files (Was: Renaming
	Include/object.h)
In-Reply-To: <ee2a432c0701032326r334edb12v11e9e75def70b59@mail.gmail.com>
References: <459BD46D.6090505@v.loewis.de>
	<200701031120.13083.fdrake@acm.org>	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>	<ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>	<459CA9A3.8070601@v.loewis.de>
	<ee2a432c0701032326r334edb12v11e9e75def70b59@mail.gmail.com>
Message-ID: <459CB110.10704@v.loewis.de>

Neal Norwitz schrieb:
> By private, I mean internal only to python and don't need to prefix
> their identifiers with Py and are subject to change without backwards
> compatibility.  Include/graminit.h is one example of what I mean.
> Some others are:  bitset.h, grammar.h, opcode.h, metagrammar.h,
> errcode.h

Ah. This seems to be a requirement completely different from the
one I'm talking about. By this definition, object.h is *not* an
internal header file, yet I want it to be renamed.

As for this issue: how about moving all such private header files
out of Include entirely? The parser-related ones should go into
Parser, for example (grammar.h, bitset.h, graminit.h, metagrammar.h,
errcode.h). This would leave us with opcode.h only.

> Others are kinda questionable (they have some things that are
> definitely public, others I'm not so sure about):  code.h, parsetok.h,
> pyarena.h, longintrepr.h, osdefs.h, pgen.h, node.h

Thomas said that at least code.h must stay where it is.

What is the reason that you want them to be renamed?

Regards,
Martin

From nnorwitz at gmail.com  Thu Jan  4 09:33:46 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 4 Jan 2007 00:33:46 -0800
Subject: [Python-Dev] Private header files (Was: Renaming
	Include/object.h)
In-Reply-To: <459CB110.10704@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>
	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>
	<ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>
	<459CA9A3.8070601@v.loewis.de>
	<ee2a432c0701032326r334edb12v11e9e75def70b59@mail.gmail.com>
	<459CB110.10704@v.loewis.de>
Message-ID: <ee2a432c0701040033r2bf6355er13a4927c637c5278@mail.gmail.com>

On 1/3/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Neal Norwitz schrieb:
> > By private, I mean internal only to python and don't need to prefix
> > their identifiers with Py and are subject to change without backwards
> > compatibility.  Include/graminit.h is one example of what I mean.
> > Some others are:  bitset.h, grammar.h, opcode.h, metagrammar.h,
> > errcode.h
>
> Ah. This seems to be a requirement completely different from the
> one I'm talking about. By this definition, object.h is *not* an
> internal header file, yet I want it to be renamed.

Agreed.  I was mixing two things that aren't necessarily related
because I see the same possible solution.  I'm also using this one
example as a good opportunity to clean up more things.  Let me try to
explain a bit more below.

> As for this issue: how about moving all such private header files
> out of Include entirely? The parser-related ones should go into
> Parser, for example (grammar.h, bitset.h, graminit.h, metagrammar.h,
> errcode.h). This would leave us with opcode.h only.
>
> > Others are kinda questionable (they have some things that are
> > definitely public, others I'm not so sure about):  code.h, parsetok.h,
> > pyarena.h, longintrepr.h, osdefs.h, pgen.h, node.h
>
> Thomas said that at least code.h must stay where it is.
>
> What is the reason that you want them to be renamed?

Sorry, I wasn't trying to imply that these should necessarily be
renamed, only that the internal portions be moved elsewhere.  I guess
I should explain my mental model first which might make things
clearer.  Then again, I'm tired, so who knows if it will explain
anything. :-)

I'm a Python embedder and I want to know what's available to me.  I
look in Include and see a ton of header files.  Do I need all these?
What do I *need* and what can I *use*?  I only want to see the public
stuff that is available to me.  Thus I want anything that has
internal/implementation details/etc out of my sight to reduce my
learning curve.  I don't ever want to learn about something I won't
need nor include anything I won't need.

That's one part.

Another part of my mental model is that I'm a Python developer and I'm
modifying a header file that is implementation specific.  I need to
share it among different subdiretories (say Python and Objects).  So I
really need to stick the header file in a common place, Include/ is
it.

I don't want to export anything, but I don't know if other third party
developers will use the header or not.  Or maybe I need to include
some implementation details in another public header.  I'll probably
be lazy and just make a single header which has some internal and some
public stuff.

I want clear rules on when identifiers need to be prefixed.  If it's
internal (e.g., in an internal directory or prefixed with _), it can
have any name and can't be included from any non-internal header.  I
can also change it in a point release.  If anyone uses anything from
here, they are on their own.  If I see any identifier in a
non-internal header, it must be public and therefore prefixed with Py
or _Py.

The Python headers are pretty good about prefixing most things.  But
they could be better.  I think it gets harder to maintain without the
rules though.

Finally, by putting everything in directories and always including a
directory in the header file, like:

  #include "python/python.h"
  #include "python/internal/foo.h"

There can never be an include file name collision as what started this
thread.  It also provides a simple way of demonstrating what's public
and what is not.  It addresses all my complaints.  There are only a
few rules and they are simple.  But I am addressing several points
that are only loosely related which I what I think generated some
confusion.

Adding the directory also makes clear were the header file comes from.
 If you see:

  #include "node.h"

you don't know if that's a python node.h, from some other part of the
code or a third party library.

Not to try to confuse things even more, but I will point out something
Google does that is only indirectly related.  Google requires
importing modules.  You aren't supposed to import classes.  You
wouldn't do:

  from foo.bar import Message
  # ...
  msg = Message(...)

You would do:

  from foo import bar
  # ...
  msg = bar.Message(...)

This makes it clear where Message comes from, just like adding a
python prefix to all header file names makes it clear where the header
file lives.  Both are good for traceability, though in different ways.
 This technique makes it easier to manage larger code bases,
particularly when there are multiple libraries used.

n

From martin at v.loewis.de  Thu Jan  4 10:17:45 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 04 Jan 2007 10:17:45 +0100
Subject: [Python-Dev] Private header files (Was: Renaming
	Include/object.h)
In-Reply-To: <ee2a432c0701040033r2bf6355er13a4927c637c5278@mail.gmail.com>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>	
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>	
	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>	
	<ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>	
	<459CA9A3.8070601@v.loewis.de>	
	<ee2a432c0701032326r334edb12v11e9e75def70b59@mail.gmail.com>	
	<459CB110.10704@v.loewis.de>
	<ee2a432c0701040033r2bf6355er13a4927c637c5278@mail.gmail.com>
Message-ID: <459CC639.5010302@v.loewis.de>

Neal Norwitz schrieb:
> I'm a Python embedder and I want to know what's available to me.  I
> look in Include and see a ton of header files.  Do I need all these?
> What do I *need* and what can I *use*?  I only want to see the public
> stuff that is available to me.  Thus I want anything that has
> internal/implementation details/etc out of my sight to reduce my
> learning curve.  I don't ever want to learn about something I won't
> need nor include anything I won't need.

Ideally, you should look at the documentation. It should have
everything you can use, and only that.

I know that this currently doesn't work, for two reasons:
- the documentation is incomplete, i.e. it omits things that people
  do use on a regular basis
- people read header files, even if the documentation was complete

> Another part of my mental model is that I'm a Python developer and I'm
> modifying a header file that is implementation specific.  I need to
> share it among different subdiretories (say Python and Objects).  So I
> really need to stick the header file in a common place, Include/ is
> it.

There could be another standard directory for header files also, like
Python.

> I don't want to export anything, but I don't know if other third party
> developers will use the header or not.

I find that debatable. I see no reason not to export everything, unless
a stable ABI is also an objective (which it may be). If people really
want to get to the internals, they can access structures even if they
aren't defined in a header file, by replicating the structure definition
in their own code.

There should be a "mostly stable" API, certainly, and this is the one
that is documented. Other things in the header files aren't "internal",
they are just undocumented and thus may change without notice.

> I want clear rules on when identifiers need to be prefixed.

Always, unless they are static functions. Even an "internal" function
must be prefixed, as you may otherwise get conflicts when embedding
Python. At the moment, only the AST exports symbols that aren't properly
mangled.

[internal symbols]
> I can also change it in a point release.  If anyone uses anything from
> here, they are on their own.

No. There shouldn't be any possible ABI breakage in a point release.

> Finally, by putting everything in directories and always including a
> directory in the header file, like:
> 
>  #include "python/python.h"
>  #include "python/internal/foo.h"
> 
> There can never be an include file name collision as what started this
> thread.  It also provides a simple way of demonstrating what's public
> and what is not.  It addresses all my complaints.  There are only a
> few rules and they are simple.

As specified, above, it is incompatible with the current API. I think

#include <Python.h>

should be preserved. I personally see no problem with a single header
file, and would prefer that include to indicate all API that we are
committed to document and keep "mostly stable".

Regards,
Martin



From mal at egenix.com  Thu Jan  4 12:04:15 2007
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 04 Jan 2007 12:04:15 +0100
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <bbaeab100701021642o4bce08ffy729c15507c4e2c55@mail.gmail.com>
References: <20070102000242.331871E4007@bag.python.org>	<459ADF8B.1080506@egenix.com>	<bbaeab100701021454i369192f4heb7bd63c246909d5@mail.gmail.com>	<459AE49A.5040704@egenix.com>
	<bbaeab100701021642o4bce08ffy729c15507c4e2c55@mail.gmail.com>
Message-ID: <459CDF2F.5020907@egenix.com>

On 2007-01-03 01:42, Brett Cannon wrote:
> On 1/2/07, M.-A. Lemburg <mal at egenix.com> wrote:
>> >> > +Open Issues
>> >> > +===========
>> >> > +
>> >> > +Consolidate dependent modules together into a single module or
>> >> package?
>> >> > ...
>> >> > +Consolidate certain modules with similar themes together in a
>> package?
>> >> >
>> +----------------------------------------------------------------------
>> >> > ...
>> >>
>> >> If you do follow this route, please take the chance to place
>> >> the whole Python stdlib under a single package. That way we'll
>> >> avoid name clashes with existing packages and modules now and
>> >> in the future.
>> >
>> >
>> > That has been suggested before (including by me) and Guido has always
>> shot
>> > it down.  That's why I left it out of this proposal.
>>
>> Even if it is shot down again, it still deserves to be documented
>> together with the reasons for being shot down.
>>
>> This is a one-in-a-lifetime chance, so it would be sad if it were
>> not taken into account.
>>
>> The extra effort would be minimal - the renaming would have to be
>> done using a script anyway and adding an extra 'from py import '
>> prefix to the modules wouldn't really make the renaming more
>> complicated ;-)
> 
> 
> I was about to start writing an open issue on this since the biggest
> objection from Guido I could find on this topic is
> http://mail.python.org/pipermail/python-dev/2002-July/026409.html , but
> then
> it started to feel like a separate PEP to me.  So I think I am going to
> pass
> on taking on this topic and let someone else tackle it in a PEP.  Sorry,
> MAL, but I need to worry about my sanity on this one.  =)

Oh well, it seemed like a perfect fit for the scope of PEP 3108.

Guido's reply seems to suggest that he's in favor of introducing
a multi-package stdlib structure:

"""
> > I'm rejecting the proposal of a single top-level package named "python".
>
> You've written that before, but you still haven't given any
> explanation of why a single package would be worse than a
> multi-level hierarchy of modules (e.g. grouped by application
> space).

Because a single package doesn't have any other benefits besides
getting out of the way from 3rd party developers.

At least a proper hierarchy would have the other benefits of grouping.
(But better make it a shallow hierarchy!  remember "Flat is better
than nested.")
"""

AFAICT, he was only objecting having a single package without any
extra restructuring.

Then again, the post is from 2002 - so things may have changed.

There have been a couple of attempts to reorg the stdlib into
packages, but AFAIR, I see, all of them were withdrawn
due to the problem of finding a suitable grouping (often enough,
a module would be suitable for more than just one functional
package, e.g. urllib would fit "io" as well as "net") or
lack of support from the developers.

Now that we're discussing moving the include files into
a subdirectory (for much the same reasons), I think it's
time to reboot the discussion of a Python package with or
without possible subpackages.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 04 2007)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From mal at egenix.com  Thu Jan  4 12:06:14 2007
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 04 Jan 2007 12:06:14 +0100
Subject: [Python-Dev] 2.5.1 plans
In-Reply-To: <ee2a432c0701032259s3d5c6ae7ned7cb3a22946e83b@mail.gmail.com>
References: <ee2a432c0612252213h5444d981ub5bed522896302bd@mail.gmail.com>
	<ee2a432c0701032259s3d5c6ae7ned7cb3a22946e83b@mail.gmail.com>
Message-ID: <459CDFA6.907@egenix.com>

On 2007-01-04 07:59, Neal Norwitz wrote:
> The current schedule looks like it's shaping up to be:
> 
> Wed, Jan 24 for 2.5.1c1
> Wed Jan 31 for 2.5.1
> 
> It would be great if you could comment on some of the bug reports
> below.  I think several already have patches/suggested fixes.
> 
> It's not clear to me if this should be fixed, but it's got a high priority::
> 
>     http://python.org/sf/1467929 %-formatting and dicts

+1

The patch is ready to be applied. The only reason it got delayed
was the 2.5 release timing.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 04 2007)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From agriff at tin.it  Thu Jan  4 13:24:27 2007
From: agriff at tin.it (Andrea Griffini)
Date: Thu, 04 Jan 2007 13:24:27 +0100
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <459CA9A3.8070601@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de>
	<200701031120.13083.fdrake@acm.org>		<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>		<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>	<ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>
	<459CA9A3.8070601@v.loewis.de>
Message-ID: <459CF1FB.9010602@tin.it>

Martin v. L?wis wrote:
> Neal Norwitz schrieb:
>   
>> Wow, I didn't realize I was that much of a broken record. :-)
>> I don't even remember talking to Thomas about it, only Guido.  I
>> definitely would like to see all private header files clearly denoted
>> by their name or directory.
>>     
>
> What is a private header file, and does Python have any?
>   
I've a partially related question... why isn't the module structure in 
an include file .h
and is instead in Objects/moduleobject.c ?
For the cached lookup optimization I copied the definition but that's surely
a bad way to do it.... I however wondered if there were good reasons for
module objects for not being published.

Andrea

From lgautier at gmail.com  Thu Jan  4 13:38:01 2007
From: lgautier at gmail.com (Laurent Gautier)
Date: Thu, 4 Jan 2007 20:38:01 +0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
Message-ID: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>

Ron,

I agree that pydoc could benefit a bit from some cleanup.
As you point it out, the ability to write quick viewers would be
very helpful. I came across that when wanting to develop script
on a remote web server for which I only had FTP access: I ended
up having to study pydoc more than I wanted in order to be able
to build a display-the-doc cgi.

However having two different modules might not be needed.
Introspection is probably already available in the separate module 'inspect',
and what a code pydoc would have to do is model the documentation
(as a tree) and offer convenience function to navigate the data. Beside that,
there would be sub-modules for the different viewers for the documentation
data - the interactive console being just one of the viewers.

Finally, I would suspect that an API-breaking modification of the module would
need time to be accepted. May be the original author of pydoc is considering
changes as well, and joining effort would be possible ?


L.


PS: I would also not go for a module name deliberately prefixed with "_"
(as some people might associate that with protected or private objects).

From rwgk at yahoo.com  Thu Jan  4 15:04:58 2007
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Thu, 4 Jan 2007 06:04:58 -0800 (PST)
Subject: [Python-Dev] 2.5.1 plans
Message-ID: <20070104140458.92703.qmail@web31103.mail.mud.yahoo.com>

It would be nice if this simple fix could be included (main branch and 2.5.1):

https://sourceforge.net/tracker/?func=detail&aid=1598181&group_id=5470&atid=105470

  [ 1598181 ] subprocess.py: O(N**2) bottleneck

I submitted the trivial fix almost two months ago, but apparently nobody feels responsible...

Ralf

----- Original Message ----
From: Neal Norwitz <nnorwitz at gmail.com>
To: Python Dev <python-dev at python.org>
Sent: Wednesday, January 3, 2007 10:59:15 PM
Subject: Re: [Python-Dev] 2.5.1 plans

The current schedule looks like it's shaping up to be:

Wed, Jan 24 for 2.5.1c1
Wed Jan 31 for 2.5.1

It would be great if you could comment on some of the bug reports
below.  I think several already have patches/suggested fixes.

These looks like they would be nice to fix::

    http://python.org/sf/1579370 Segfault provoked by generators and exceptions
    http://python.org/sf/1377858 segfaults when using __del__ and weakrefs

There is one outstanding issue I was notified of in private mail::

    http://python.org/sf/1568240 Tix is not included in 2.5 for Windows

It's not clear to me if this should be fixed, but it's got a high priority::

    http://python.org/sf/1467929 %-formatting and dicts

n





__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

From barry at python.org  Thu Jan  4 17:21:13 2007
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Jan 2007 11:21:13 -0500
Subject: [Python-Dev] Private header files (Was: Renaming
	Include/object.h)
In-Reply-To: <459CC639.5010302@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>	
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>	
	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>	
	<ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>	
	<459CA9A3.8070601@v.loewis.de>	
	<ee2a432c0701032326r334edb12v11e9e75def70b59@mail.gmail.com>	
	<459CB110.10704@v.loewis.de>
	<ee2a432c0701040033r2bf6355er13a4927c637c5278@mail.gmail.com>
	<459CC639.5010302@v.loewis.de>
Message-ID: <D1DDD2BB-8978-4E86-B865-2BB855D19C1E@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 4, 2007, at 4:17 AM, Martin v. L?wis wrote:

> As specified, above, it is incompatible with the current API. I think
>
> #include <Python.h>
>
> should be preserved. I personally see no problem with a single header
> file, and would prefer that include to indicate all API that we are
> committed to document and keep "mostly stable".

I agree.  I don't mind if Python.h is just a wrapper around #includes  
from python/*.h.  I think we should add structmember.h and  
structseq.h to Python.h and perhaps move everything else into a  
'python' subdirectory.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRZ0pfnEjvBPtnXfVAQKuyAP+J8Tm3z4am5BOfXCSJIeHsA1tEddeniuM
dqSUAPUQUag9WkvkAreQYXu3iRC26e52mJ0B8eceqiuBa16WPILb0CvRFCBoW2fc
/FAg4EROlMBhrE/MWVSfGSi76bL+4CwaogmHOnsvyCDEppA420C/8Zkz1VZYiGGd
T+Ppo1ZVsHA=
=b/Fm
-----END PGP SIGNATURE-----

From rrr at ronadam.com  Thu Jan  4 17:14:54 2007
From: rrr at ronadam.com (Ron Adam)
Date: Thu, 04 Jan 2007 10:14:54 -0600
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
Message-ID: <459D27FE.6070406@ronadam.com>

Laurent Gautier wrote:
> Ron,
> 
> I agree that pydoc could benefit a bit from some cleanup.
> As you point it out, the ability to write quick viewers would be
> very helpful. I came across that when wanting to develop script
> on a remote web server for which I only had FTP access: I ended
> up having to study pydoc more than I wanted in order to be able
> to build a display-the-doc cgi.
> 
> However having two different modules might not be needed.

I think weather it is two modules or more, or a package, is still an open issue. 
  Others have suggested it may be better for it to be package so I'm continuing 
in that direction.


> Introspection is probably already available in the separate module 
> 'inspect',
> and what a code pydoc would have to do is model the documentation
> (as a tree) and offer convenience function to navigate the data. Beside 
> that,
> there would be sub-modules for the different viewers for the documentation
> data - the interactive console being just one of the viewers.

Pydoc currently uses functions from the inspect module along with directly 
accessing attributes and other information where it is available.  It's not a 
replacement for the inspect module.

My first attempt used an xml tree to store the information, but to make that 
work requires also storing a fair amount of meta information in the tree as 
well.  I found parsing the tree and the meta information to be more complex than 
using an objective approach which is (to me) more readable and easier to extend. 
  But if you want to try it again, please do.  You may come up with something 
far better than I did.


> Finally, I would suspect that an API-breaking modification of the module 
> would
> need time to be accepted. 

I was considering this for python 3.0, but one of the developers suggested it 
would be nice to have in python 2.6 and to move the discussion here.

I think any API issues could be worked out.  Are there any programs you know of, 
(yours?), that import pydoc besides the python console?


May be the original author of pydoc is
> considering
> changes as well, and joining effort would be possible ?

AUTHOR
     Ka-Ping Yee <ping at lfw.org>

CREDITS
     Guido van Rossum, for an excellent programming language.
     Tommy Burnette, the original creator of manpy.
     Paul Prescod, for all his work on onlinehelp.
     Richard Chamberlain, for the first implementation of textdoc.


Ka-Ping reads python-dev but I'm cc'ing this to him just in case.  (I Used his 
python-dev email address since I don't know if the above one is current.)


Pydoc is a fairly complex program and it would definitely help if others took a 
look at various parts and made contributions and or suggestions to making it better.

I may have also gotten a bit over my head, but I'm willing to stick it out and 
try to get it finished with any suggestions (and help) that any one is willing 
to give me.  There are also too many important issues for me to be decided, so 
this isn't something that can be done in isolation.

The download link again is:

     http://ronadam.com/dl/_pydoc.zip

It's not fully functional yet, but does run. Some parts like the command line 
file output options still need to be reimplemented. Some output formatting still 
needs to be cleaned up, and the MRO tree parsing section still needs to be put 
back in.


One question that I have at the moment is:

* Would it be good to have the "KEYWORD" and "TOPICS" info as included data 
objects or files, and possibly use that to generate the python html (and other) 
documentation for these?  (Instead of the other way around like it is now.)

This would eliminate the requirement to install something extra in order for 
help on these items to work.

> L.
> 
> PS: I would also not go for a module name deliberately prefixed with "_"
> (as some people might associate that with protected or private objects).

The underscore was just a "temporary" convenience to avoid the name conflict 
with the existing module.  Weather to reuse the old name or use a new name is 
still one of the many open issues I think.

Ron


From martin at v.loewis.de  Thu Jan  4 17:33:12 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 04 Jan 2007 17:33:12 +0100
Subject: [Python-Dev] Private header files (Was: Renaming
	Include/object.h)
In-Reply-To: <D1DDD2BB-8978-4E86-B865-2BB855D19C1E@python.org>
References: <459BD46D.6090505@v.loewis.de> <200701031120.13083.fdrake@acm.org>	
	<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>	
	<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>	
	<ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>	
	<459CA9A3.8070601@v.loewis.de>	
	<ee2a432c0701032326r334edb12v11e9e75def70b59@mail.gmail.com>	
	<459CB110.10704@v.loewis.de>
	<ee2a432c0701040033r2bf6355er13a4927c637c5278@mail.gmail.com>
	<459CC639.5010302@v.loewis.de>
	<D1DDD2BB-8978-4E86-B865-2BB855D19C1E@python.org>
Message-ID: <459D2C48.2090803@v.loewis.de>

Barry Warsaw schrieb:
> I agree.  I don't mind if Python.h is just a wrapper around #includes
> from python/*.h.  I think we should add structmember.h and structseq.h
> to Python.h and perhaps move everything else into a 'python' subdirectory.

For the python subdirectory, there is the issue that the framework
includes in OSX magically look for python.framework when searching for
python/foo.h, which they find, so that may get us the wrong version.
Somebody would have to study the details here, first.

Regards,
Martin

From fdrake at acm.org  Thu Jan  4 17:56:47 2007
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Thu, 4 Jan 2007 11:56:47 -0500
Subject: [Python-Dev] Private header files (Was: Renaming
	Include/object.h)
In-Reply-To: <459D2C48.2090803@v.loewis.de>
References: <459BD46D.6090505@v.loewis.de>
	<D1DDD2BB-8978-4E86-B865-2BB855D19C1E@python.org>
	<459D2C48.2090803@v.loewis.de>
Message-ID: <200701041156.47988.fdrake@acm.org>

On Thursday 04 January 2007 11:33, Martin v. L?wis wrote:
 > For the python subdirectory, there is the issue that the framework
 > includes in OSX magically look for python.framework when searching for
 > python/foo.h, which they find, so that may get us the wrong version.
 > Somebody would have to study the details here, first.

If everything public gets included from Python.h, perhaps python/object.h and 
friends could become pythonX.Y/object.h; I'm not sure this will solve the Mac 
OS framework magic issue, though, not being a Mac OS developer.


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From mike.klaas at gmail.com  Thu Jan  4 19:22:54 2007
From: mike.klaas at gmail.com (Mike Klaas)
Date: Thu, 4 Jan 2007 10:22:54 -0800
Subject: [Python-Dev] 2.5.1 plans
In-Reply-To: <20070104140458.92703.qmail@web31103.mail.mud.yahoo.com>
References: <20070104140458.92703.qmail@web31103.mail.mud.yahoo.com>
Message-ID: <3d2ce8cb0701041022p572c9626o74b2088fa017c3b9@mail.gmail.com>

On 1/4/07, Ralf W. Grosse-Kunstleve <rwgk at yahoo.com> wrote:
> It would be nice if this simple fix could be included (main branch and 2.5.1):
>
> https://sourceforge.net/tracker/?func=detail&aid=1598181&group_id=5470&atid=105470
>
>   [ 1598181 ] subprocess.py: O(N**2) bottleneck
>
> I submitted the trivial fix almost two months ago, but apparently nobody feels responsible...

I just reviewed the patch, which should help it get accepted.

-Mike

From brett at python.org  Thu Jan  4 20:16:37 2007
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Jan 2007 11:16:37 -0800
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <459CDF2F.5020907@egenix.com>
References: <20070102000242.331871E4007@bag.python.org>
	<459ADF8B.1080506@egenix.com>
	<bbaeab100701021454i369192f4heb7bd63c246909d5@mail.gmail.com>
	<459AE49A.5040704@egenix.com>
	<bbaeab100701021642o4bce08ffy729c15507c4e2c55@mail.gmail.com>
	<459CDF2F.5020907@egenix.com>
Message-ID: <bbaeab100701041116m4577c578n7416e08602797179@mail.gmail.com>

On 1/4/07, M.-A. Lemburg <mal at egenix.com> wrote:
>
> On 2007-01-03 01:42, Brett Cannon wrote:
> > On 1/2/07, M.-A. Lemburg <mal at egenix.com> wrote:
> >> >> > +Open Issues
> >> >> > +===========
> >> >> > +
> >> >> > +Consolidate dependent modules together into a single module or
> >> >> package?
> >> >> > ...
> >> >> > +Consolidate certain modules with similar themes together in a
> >> package?
> >> >> >
> >> +----------------------------------------------------------------------
> >> >> > ...
> >> >>
> >> >> If you do follow this route, please take the chance to place
> >> >> the whole Python stdlib under a single package. That way we'll
> >> >> avoid name clashes with existing packages and modules now and
> >> >> in the future.
> >> >
> >> >
> >> > That has been suggested before (including by me) and Guido has always
> >> shot
> >> > it down.  That's why I left it out of this proposal.
> >>
> >> Even if it is shot down again, it still deserves to be documented
> >> together with the reasons for being shot down.
> >>
> >> This is a one-in-a-lifetime chance, so it would be sad if it were
> >> not taken into account.
> >>
> >> The extra effort would be minimal - the renaming would have to be
> >> done using a script anyway and adding an extra 'from py import '
> >> prefix to the modules wouldn't really make the renaming more
> >> complicated ;-)
> >
> >
> > I was about to start writing an open issue on this since the biggest
> > objection from Guido I could find on this topic is
> > http://mail.python.org/pipermail/python-dev/2002-July/026409.html , but
> > then
> > it started to feel like a separate PEP to me.  So I think I am going to
> > pass
> > on taking on this topic and let someone else tackle it in a PEP.  Sorry,
> > MAL, but I need to worry about my sanity on this one.  =)
>
> Oh well, it seemed like a perfect fit for the scope of PEP 3108.


I know, but I honestly just don't have the energy to deal with it.  If you
want to spear-head the discussion and help me add it to the PEP, then that's
great.

Guido's reply seems to suggest that he's in favor of introducing
> a multi-package stdlib structure:
>
> """
> > > I'm rejecting the proposal of a single top-level package named
> "python".
> >
> > You've written that before, but you still haven't given any
> > explanation of why a single package would be worse than a
> > multi-level hierarchy of modules (e.g. grouped by application
> > space).
>
> Because a single package doesn't have any other benefits besides
> getting out of the way from 3rd party developers.
>
> At least a proper hierarchy would have the other benefits of grouping.
> (But better make it a shallow hierarchy!  remember "Flat is better
> than nested.")
> """
>
> AFAICT, he was only objecting having a single package without any
> extra restructuring.


Yep.  PEP 3108 does have some basic package suggestions in the Open Issues
section and people seem to support them.  I will be making a separate push
for them on python-3000 once the whole discussion of what modules to remove
has settled down.

Then again, the post is from 2002 - so things may have changed.


Maybe.

There have been a couple of attempts to reorg the stdlib into
> packages, but AFAIR, I see, all of them were withdrawn
> due to the problem of finding a suitable grouping (often enough,
> a module would be suitable for more than just one functional
> package, e.g. urllib would fit "io" as well as "net") or
> lack of support from the developers.


Yep, that's what has happened.

Now that we're discussing moving the include files into
> a subdirectory (for much the same reasons), I think it's
> time to reboot the discussion of a Python package with or
> without possible subpackages.


Well, perhaps other people want to show support if they like the idea?  I am
personally split down the middle either way.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070104/158841bb/attachment.htm 

From ronaldoussoren at mac.com  Thu Jan  4 20:36:54 2007
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 4 Jan 2007 20:36:54 +0100
Subject: [Python-Dev] Private header files (Was:
	Renaming	Include/object.h)
In-Reply-To: <200701041156.47988.fdrake@acm.org>
References: <459BD46D.6090505@v.loewis.de>
	<D1DDD2BB-8978-4E86-B865-2BB855D19C1E@python.org>
	<459D2C48.2090803@v.loewis.de> <200701041156.47988.fdrake@acm.org>
Message-ID: <0B8DE54A-DA60-4544-AF23-2C7B1A2E1954@mac.com>


On 4 Jan, 2007, at 17:56, Fred L. Drake, Jr. wrote:

> On Thursday 04 January 2007 11:33, Martin v. L?wis wrote:
>> For the python subdirectory, there is the issue that the framework
>> includes in OSX magically look for python.framework when searching  
>> for
>> python/foo.h, which they find, so that may get us the wrong version.
>> Somebody would have to study the details here, first.
>
> If everything public gets included from Python.h, perhaps python/ 
> object.h and
> friends could become pythonX.Y/object.h; I'm not sure this will  
> solve the Mac
> OS framework magic issue, though, not being a Mac OS developer.

That would solve the problem, however I don't think there is one. An  
experiment seems to indicate that the include path is prefered over  
the magic inclusion of framework headers (see the trace below). I  
haven't checked yet if the behaviour shown below is intentional, but  
I'd be surprised if it isn't.

$ mkdir include
$ mkdir include/Python
$ cat > include/Python/object.h <<EOF
#error "my header included"
EOF
$ cat > demo.c <<-EOF
#include <Python/object.h>
EOF
$
$ cc -c demo.c
In file included from demo.c:1:
/System/Library/Frameworks/Python.framework/Headers/object.h:227:  
error: parse error before ?FILE?
/System/Library/Frameworks/Python.framework/Headers/object.h:353:  
error: parse error before ?PyType_IsSubtype?
... more errors removed, this clearly includes a header from the  
python framework
$
$ cc -I include -c t.c
In file included from demo.c:1:
include/Python/object.h:1:2: error: #error "my error"

Therefore moving all headers into a directory named Python would  
cause no problems users that use the normal way of linking with  
python and would even allow users to use the python framework as a  
normal framework.

Ronald
>
>
>   -Fred
>
> -- 
> Fred L. Drake, Jr.   <fdrake at acm.org>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ 
> ronaldoussoren%40mac.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3562 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20070104/e1d8808a/attachment.bin 

From barry at python.org  Thu Jan  4 20:49:58 2007
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Jan 2007 14:49:58 -0500
Subject: [Python-Dev] Private header files (Was:
	Renaming	Include/object.h)
In-Reply-To: <0B8DE54A-DA60-4544-AF23-2C7B1A2E1954@mac.com>
References: <459BD46D.6090505@v.loewis.de>
	<D1DDD2BB-8978-4E86-B865-2BB855D19C1E@python.org>
	<459D2C48.2090803@v.loewis.de> <200701041156.47988.fdrake@acm.org>
	<0B8DE54A-DA60-4544-AF23-2C7B1A2E1954@mac.com>
Message-ID: <B34E2676-FBDF-4FAE-A39B-5827F7656DB6@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 4, 2007, at 2:36 PM, Ronald Oussoren wrote:

> On 4 Jan, 2007, at 17:56, Fred L. Drake, Jr. wrote:
>
>> On Thursday 04 January 2007 11:33, Martin v. L?wis wrote:
>>> For the python subdirectory, there is the issue that the framework
>>> includes in OSX magically look for python.framework when  
>>> searching for
>>> python/foo.h, which they find, so that may get us the wrong version.
>>> Somebody would have to study the details here, first.
>>
>> If everything public gets included from Python.h, perhaps python/ 
>> object.h and
>> friends could become pythonX.Y/object.h; I'm not sure this will  
>> solve the Mac
>> OS framework magic issue, though, not being a Mac OS developer.
>
> That would solve the problem, however I don't think there is one.  
> An experiment seems to indicate that the include path is prefered  
> over the magic inclusion of framework headers (see the trace  
> below). I haven't checked yet if the behaviour shown below is  
> intentional, but I'd be surprised if it isn't.
>
> $ mkdir include
> $ mkdir include/Python
> $ cat > include/Python/object.h <<EOF
> #error "my header included"
> EOF
> $ cat > demo.c <<-EOF
> #include <Python/object.h>
> EOF
> $
> $ cc -c demo.c
> In file included from demo.c:1:
> /System/Library/Frameworks/Python.framework/Headers/object.h:227:  
> error: parse error before ?FILE?
> /System/Library/Frameworks/Python.framework/Headers/object.h:353:  
> error: parse error before ?PyType_IsSubtype?
> ... more errors removed, this clearly includes a header from the  
> python framework
> $
> $ cc -I include -c t.c
> In file included from demo.c:1:
> include/Python/object.h:1:2: error: #error "my error"
>
> Therefore moving all headers into a directory named Python would  
> cause no problems users that use the normal way of linking with  
> python and would even allow users to use the python framework as a  
> normal framework.

I think that's basically correct: framework includes are searched  
after normal includes.  You might be able to confuse things if you  
used -F/-framework options, because I believe those are interleaved  
with -I paths, but then if you add -F presumably you should know what  
you're doing.

Thanks for testing this.
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRZ1abHEjvBPtnXfVAQJQtAP/XgGgI2z7xUGJlxBGfZiggIEtxRYzJObn
TVl/2r7tJ58QCwTzc+eI/m18gcfi85q+hmS1hPc9tjq0ICiqZGjSI9hpSsq0Uqva
WXKFscmvnNyZLrhemy8AjHSbA7dKKBGKBmqycjEt26am4LetoCD/HCt44+AaoI3d
SIzFFiSKw/4=
=4FpY
-----END PGP SIGNATURE-----

From steve at holdenweb.com  Thu Jan  4 21:38:40 2007
From: steve at holdenweb.com (Steve Holden)
Date: Thu, 04 Jan 2007 20:38:40 +0000
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <bbaeab100701041116m4577c578n7416e08602797179@mail.gmail.com>
References: <20070102000242.331871E4007@bag.python.org>	<459ADF8B.1080506@egenix.com>	<bbaeab100701021454i369192f4heb7bd63c246909d5@mail.gmail.com>	<459AE49A.5040704@egenix.com>	<bbaeab100701021642o4bce08ffy729c15507c4e2c55@mail.gmail.com>	<459CDF2F.5020907@egenix.com>
	<bbaeab100701041116m4577c578n7416e08602797179@mail.gmail.com>
Message-ID: <459D65D0.2000503@holdenweb.com>

Brett Cannon wrote:
[ ... ]
> Yep.  PEP 3108 does have some basic package suggestions in the Open 
> Issues section and people seem to support them.  I will be making a 
> separate push for them on python-3000 once the whole discussion of what 
> modules to remove has settled down.
> 
>     Then again, the post is from 2002 - so things may have changed.
> 
> 
> Maybe.
> 
>     There have been a couple of attempts to reorg the stdlib into
>     packages, but AFAIR, I see, all of them were withdrawn
>     due to the problem of finding a suitable grouping (often enough,
>     a module would be suitable for more than just one functional
>     package, e.g. urllib would fit "io" as well as "net") or
>     lack of support from the developers. 
> 
> 
> Yep, that's what has happened.

I can't believe that we need to be flummoxed by the necessity of having 
the same package appear at two (or more!) different places in the 
package naming hierarchy. I suspect lack of support is more due to 
developers feeling there are more profitable ways to spend their time.
> 
>     Now that we're discussing moving the include files into
>     a subdirectory (for much the same reasons), I think it's
>     time to reboot the discussion of a Python package with or
>     without possible subpackages.
> 
> 
> Well, perhaps other people want to show support if they like the idea?  
> I am personally split down the middle either way.
> 
It would be an excellent idea to clean up the standard library space. It 
should be possible in most cases to provide backwards-compatible 
implementations of the current modules (at least the pure Python ones) 
by doing an import * from the appropriate new-style package.

Some such compatibility mechanism will be essential if the re-org is to 
happen in an acceptable way before Py3k.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com

From steve at holdenweb.com  Thu Jan  4 21:38:40 2007
From: steve at holdenweb.com (Steve Holden)
Date: Thu, 04 Jan 2007 20:38:40 +0000
Subject: [Python-Dev] pep-3108.txt
In-Reply-To: <bbaeab100701041116m4577c578n7416e08602797179@mail.gmail.com>
References: <20070102000242.331871E4007@bag.python.org>	<459ADF8B.1080506@egenix.com>	<bbaeab100701021454i369192f4heb7bd63c246909d5@mail.gmail.com>	<459AE49A.5040704@egenix.com>	<bbaeab100701021642o4bce08ffy729c15507c4e2c55@mail.gmail.com>	<459CDF2F.5020907@egenix.com>
	<bbaeab100701041116m4577c578n7416e08602797179@mail.gmail.com>
Message-ID: <459D65D0.2000503@holdenweb.com>

Brett Cannon wrote:
[ ... ]
> Yep.  PEP 3108 does have some basic package suggestions in the Open 
> Issues section and people seem to support them.  I will be making a 
> separate push for them on python-3000 once the whole discussion of what 
> modules to remove has settled down.
> 
>     Then again, the post is from 2002 - so things may have changed.
> 
> 
> Maybe.
> 
>     There have been a couple of attempts to reorg the stdlib into
>     packages, but AFAIR, I see, all of them were withdrawn
>     due to the problem of finding a suitable grouping (often enough,
>     a module would be suitable for more than just one functional
>     package, e.g. urllib would fit "io" as well as "net") or
>     lack of support from the developers. 
> 
> 
> Yep, that's what has happened.

I can't believe that we need to be flummoxed by the necessity of having 
the same package appear at two (or more!) different places in the 
package naming hierarchy. I suspect lack of support is more due to 
developers feeling there are more profitable ways to spend their time.
> 
>     Now that we're discussing moving the include files into
>     a subdirectory (for much the same reasons), I think it's
>     time to reboot the discussion of a Python package with or
>     without possible subpackages.
> 
> 
> Well, perhaps other people want to show support if they like the idea?  
> I am personally split down the middle either way.
> 
It would be an excellent idea to clean up the standard library space. It 
should be possible in most cases to provide backwards-compatible 
implementations of the current modules (at least the pure Python ones) 
by doing an import * from the appropriate new-style package.

Some such compatibility mechanism will be essential if the re-org is to 
happen in an acceptable way before Py3k.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From amk at amk.ca  Thu Jan  4 22:00:15 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 4 Jan 2007 16:00:15 -0500
Subject: [Python-Dev] 2.5.1 plans
In-Reply-To: <3d2ce8cb0701041022p572c9626o74b2088fa017c3b9@mail.gmail.com>
References: <20070104140458.92703.qmail@web31103.mail.mud.yahoo.com>
	<3d2ce8cb0701041022p572c9626o74b2088fa017c3b9@mail.gmail.com>
Message-ID: <20070104210015.GA15252@localhost.localdomain>

On Thu, Jan 04, 2007 at 10:22:54AM -0800, Mike Klaas wrote:
> >  [ 1598181 ] subprocess.py: O(N**2) bottleneck
> >
> >I submitted the trivial fix almost two months ago, but apparently nobody 
> >feels responsible...

Is Peter Astrand still actively maintaining the module?  I've been
assigning subprocess bugs to him on the theory that he'll fix them.
If he's not around, we can apply patches to subprocess.  (The problem
with subprocess in particular would be the difficulty of testing
changes that affect both Windows and POSIX code; I think relatively
few developers use both Windows and Unix.)

We should also try to make an effort to go through the tracker and
look for bug fixes and patches suitable for 2.5.1.

--amk

From nnorwitz at gmail.com  Thu Jan  4 22:47:13 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 4 Jan 2007 13:47:13 -0800
Subject: [Python-Dev] 2.5.1 plans
In-Reply-To: <20070104210015.GA15252@localhost.localdomain>
References: <20070104140458.92703.qmail@web31103.mail.mud.yahoo.com>
	<3d2ce8cb0701041022p572c9626o74b2088fa017c3b9@mail.gmail.com>
	<20070104210015.GA15252@localhost.localdomain>
Message-ID: <ee2a432c0701041347p6c7734a6r7932dc721f75ce57@mail.gmail.com>

On 1/4/07, A.M. Kuchling <amk at amk.ca> wrote:
> On Thu, Jan 04, 2007 at 10:22:54AM -0800, Mike Klaas wrote:
> > >  [ 1598181 ] subprocess.py: O(N**2) bottleneck
> > >
> > >I submitted the trivial fix almost two months ago, but apparently nobody
> > >feels responsible...
>
> Is Peter Astrand still actively maintaining the module?  I've been
> assigning subprocess bugs to him on the theory that he'll fix them.
> If he's not around, we can apply patches to subprocess.  (The problem
> with subprocess in particular would be the difficulty of testing
> changes that affect both Windows and POSIX code; I think relatively
> few developers use both Windows and Unix.)

We have the buildbots to help with this.

> We should also try to make an effort to go through the tracker and
> look for bug fixes and patches suitable for 2.5.1.

Definitely!  I only did a really quick review.  If you want someone to
think about any of these, assign them to me with a priority of 9.
I'll discuss with others and see if the fixes should go in to 2.5.1 or
not.

n

From python-dev at zesty.ca  Thu Jan  4 23:49:44 2007
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Thu, 4 Jan 2007 16:49:44 -0600 (CST)
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459D27FE.6070406@ronadam.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
Message-ID: <Pine.LNX.4.58.0701041609240.23077@server1.LFW.org>

Hi Ron and Laurent,

I welcome attempts to improve pydoc (especially since I don't have
much time to work on improving it myself).  I definitely agree that
moving to CSS is long overdue, though I would like some input on
the style of the produced pages.

It's probably a good idea to explain how pydoc got to be the way
that it is.  The module boundary between inspect and pydoc is a
pretty clear one, intended to isolate pydoc from future changes
to Python's introspection features (such as attributes on internal
types like frames and functions).

On the other hand, I've often seen the question of why pydoc does
both text and HTML generation instead of generating some intermediate
data structure from which both kinds of output are produced.  The
answer is: I tried it.  The result turned out to be longer than
I expected and needlessly more complicated than what we have now.
It may be that a better job could have been done, but I think there
is a rational basis for why it turned out that way.

The Python objects themselves already are a data structure containing
all of the information we need.  I discovered that translating this
data structure into another data structure and then producing text
or HTML was more work than simply producing text or HTML.  With CSS,
the last step gets even easier and so the intermediate stage becomes
even less necessary.  Also, the intermediate step required me to
essentially invent an API, and I decided that I trusted the stability
of Python's API more than that of some API I invented just for this.

This is not to say that text and HTML generation can't be separated;
it's just a caution against attempting to overgeneralize by creating
an intermediate format.  I'm glad you backed away from XML (or I'd
have warned you that processing the XML would be a lot of extra work).

The inspect module was intended to pull out as much as possible of
the extraction functionality that's shared by the text and HTML
documentation generators.  But pydoc is still big.  At the time I was
proposing pydoc for addition to the standard library, I didn't want
to pollute the top-level module namespace with too many names, so I
tried hard to minimize the number of modules.  And of course it has
grown since then with bits of new functionality and support for new
language features in Python.

But now if a package is being considered, it makes sense to split
out some of the pieces (as you have done), such as the web server,
the search function, and the interactive interpreter help prompt.
It may even enable pydoc to provide search from the interactive help
prompt, which would be a nice feature!  The package could contain
several modules for ease of maintenance, while still providing a
single, convenient command for running pydoc from the Unix prompt.


-- ?!ng

From kbk at shore.net  Fri Jan  5 04:40:49 2007
From: kbk at shore.net (Kurt B. Kaiser)
Date: Thu, 4 Jan 2007 22:40:49 -0500 (EST)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200701050340.l053en59025586@bayview.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  418 open ( +5) /  3522 closed ( +1) /  3940 total ( +6)
Bugs    :  959 open (+13) /  6405 closed ( +5) /  7364 total (+18)
RFE     :  250 open ( +2) /   245 closed ( -1) /   495 total ( +1)

New / Reopened Patches
______________________

update to PEP 344 - exception attributes  (2007-01-02)
CLOSED http://python.org/sf/1626538  opened by  Jim Jewett

backticks will not be used at all  (2007-01-03)
       http://python.org/sf/1627052  opened by  Jim Jewett

Fix for #1601399 (urllib2 does not close sockets properly)  (2007-01-03)
       http://python.org/sf/1627441  opened by  John J Lee

Win32: Fix build when you have TortoiseSVN but no .svn/*  (2007-01-04)
       http://python.org/sf/1628061  opened by  Larry Hastings

Win32: Add bytesobject.c to pythoncore.vcproj  (2007-01-04)
       http://python.org/sf/1628062  opened by  Larry Hastings

socket.readline() interface doesn't handle EINTR properly  (2007-01-04)
       http://python.org/sf/1628205  opened by  Maxim Sobolev

Patches Closed
______________

update to PEP 344 - exception attributes  (2007-01-02)
       http://python.org/sf/1626538  closed by  ping

New / Reopened Bugs
___________________

sqlite3 documentation omits: close(), commit(), autocommit  (2006-12-30)
       http://python.org/sf/1625205  opened by  kitbyaydemir

re module documentation on search/match is unclear  (2006-12-31)
       http://python.org/sf/1625381  opened by  Richard Boulton

'imp' documentation does not mention that lock is re-entrant  (2006-12-31)
       http://python.org/sf/1625509  opened by  Dustin J. Mitchell

add ability to specify name to os.fdopen  (2007-01-01)
       http://python.org/sf/1625576  opened by  Mark Diekhans

'Installing Python Modules' does not work for Windows  (2007-01-02)
       http://python.org/sf/1626300  opened by  Christopher Lambacher

Would you mind renaming object.h to pyobject.h?  (2007-01-02)
       http://python.org/sf/1626545  opened by  Anton Tropashko

posixmodule.c leaks crypto context on Windows  (2007-01-03)
       http://python.org/sf/1626801  opened by  Yitz Gale

website issue reporter down  (2007-01-03)
       http://python.org/sf/1627036  opened by  Jim Jewett

mention side-lists from python-dev description  (2007-01-03)
       http://python.org/sf/1627039  opened by  Jim Jewett

xml.dom.minidom parse bug  (2007-01-03)
CLOSED http://python.org/sf/1627096  opened by  Pierre Imbaud

xml.dom.minidom parse bug  (2007-01-03)
CLOSED http://python.org/sf/1627244  opened by  Pierre Imbaud

an extra comma in condition command crashes pdb  (2007-01-03)
       http://python.org/sf/1627316  opened by  Ilya Sandler

Typo in module index for Carbon.CarbonEvt  (2007-01-03)
CLOSED http://python.org/sf/1627373  opened by  Brett Cannon

Status bar on OSX garbled  (2007-01-03)
       http://python.org/sf/1627543  opened by  sigzero

RotatingFileHandler cannot recover from failed doRollover()  (2007-01-03)
       http://python.org/sf/1627575  opened by  Forest Wilkinson

documentation error for "startswith" string method  (2007-01-04)
CLOSED http://python.org/sf/1627690  opened by  Keith Briggs

plat-mac videoreader.py auido format info  (2007-01-04)
       http://python.org/sf/1627952  opened by  Ryan Owen

documentation error for "startswith" string method  (2007-01-04)
       http://python.org/sf/1627956  opened by  Keith Briggs

Bugs Closed
___________

test_logging hangs on cygwin  (2006-04-06)
       http://python.org/sf/1465643  closed by  sf-robot

xml.dom.minidom parse bug  (2007-01-03)
       http://python.org/sf/1627096  closed by  loewis

xml.dom.minidom parse bug  (2007-01-03)
       http://python.org/sf/1627244  closed by  nnorwitz

Typo in module index for Carbon.CarbonEvt  (2007-01-03)
       http://python.org/sf/1627373  closed by  nnorwitz

documentation error for "startswith" string method  (2007-01-04)
       http://python.org/sf/1627690  closed by  loewis

Logging problem on Windows XP  (2006-09-27)
       http://python.org/sf/1566280  closed by  loewis

New / Reopened RFE
__________________

smarter temporary file object  (2001-04-12)
       http://python.org/sf/415692  reopened by  gvanrossum

optparse "store" action should not gobble up next  option  (2007-01-03)
       http://python.org/sf/1627266  opened by  Raghuram Devarakonda


From rrr at ronadam.com  Fri Jan  5 04:54:23 2007
From: rrr at ronadam.com (Ron Adam)
Date: Thu, 04 Jan 2007 21:54:23 -0600
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <Pine.LNX.4.58.0701041609240.23077@server1.LFW.org>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<Pine.LNX.4.58.0701041609240.23077@server1.LFW.org>
Message-ID: <459DCBEF.7000209@ronadam.com>

Ka-Ping Yee wrote:
> Hi Ron and Laurent,
> 
> I welcome attempts to improve pydoc (especially since I don't have
> much time to work on improving it myself).  I definitely agree that
> moving to CSS is long overdue, though I would like some input on
> the style of the produced pages.

Additional input would be good.  The html output I used is nearly pure nested 
definition lists with CSS styling to set the fonts, borders, and indents.  It 
was A bit tricky in some places to keep it looking like the current pydoc pages. 
   My mental target was something that would both look good printed and also fit 
in with Pythons current web site design while not changing it too much.

Changing the CSS file to produce other output styled pages should not be that 
difficult.  A little experimenting would be good in order to find where 
additional style tags in the html code may be needed.


> It's probably a good idea to explain how pydoc got to be the way
> that it is.  The module boundary between inspect and pydoc is a
> pretty clear one, intended to isolate pydoc from future changes
> to Python's introspection features (such as attributes on internal
> types like frames and functions).
> 
> On the other hand, I've often seen the question of why pydoc does
> both text and HTML generation instead of generating some intermediate
> data structure from which both kinds of output are produced.  The
> answer is: I tried it.  The result turned out to be longer than
> I expected and needlessly more complicated than what we have now.
> It may be that a better job could have been done, but I think there
> is a rational basis for why it turned out that way.

Yes, I found it was a trade off from one type of complexity to another.  And I 
didn't like importing something that will probably go through more changes like 
xmltree.


> The Python objects themselves already are a data structure containing
> all of the information we need.  I discovered that translating this
> data structure into another data structure and then producing text
> or HTML was more work than simply producing text or HTML.  With CSS,
> the last step gets even easier and so the intermediate stage becomes
> even less necessary.  Also, the intermediate step required me to
> essentially invent an API, and I decided that I trusted the stability
> of Python's API more than that of some API I invented just for this.
 >
> This is not to say that text and HTML generation can't be separated;
> it's just a caution against attempting to overgeneralize by creating
> an intermediate format.  I'm glad you backed away from XML (or I'd
> have warned you that processing the XML would be a lot of extra work).
 >
> The inspect module was intended to pull out as much as possible of
> the extraction functionality that's shared by the text and HTML
> documentation generators.  But pydoc is still big.  At the time I was
> proposing pydoc for addition to the standard library, I didn't want
> to pollute the top-level module namespace with too many names, so I
> tried hard to minimize the number of modules.  And of course it has
> grown since then with bits of new functionality and support for new
> language features in Python.

And it will continue to grow as python does.  Hopefully we can make the process 
of supporting new language features easier.


> But now if a package is being considered, it makes sense to split
> out some of the pieces (as you have done), such as the web server,
> the search function, and the interactive interpreter help prompt.
> It may even enable pydoc to provide search from the interactive help
> prompt, which would be a nice feature!  

I think that could be done without too much trouble.  It only takes adding a new 
allcaps word "FIND <something>" or "SEARCH <something>", in addition to KEYWORDS 
and TOPICS.


> The package could contain
> several modules for ease of maintenance, while still providing a
> single, convenient command for running pydoc from the Unix prompt.

I was thinking of two convenient entry points. One for text and the interactive 
console and one for html, and the web browser interface.  pyhelp and pydoc 
respectfully.

There is also the possibility of splitting it into two much smaller packages, 
one for the command line and interactive help console.  No html stuff or server 
stuff here. This could be better controlled and maintained as it's used in 
pythons console. Another plus is it will be easier to maintain as well.  The 
other package (or module) would be an example of how to extend or build an 
application, an html formatter and help browser in this case, from the console 
help package.

Cheers,
    Ron


From larry at hastings.org  Fri Jan  5 05:14:37 2007
From: larry at hastings.org (Larry Hastings)
Date: Thu, 04 Jan 2007 20:14:37 -0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <458C719F.3000904@ronadam.com>
References: <458C719F.3000904@ronadam.com>
Message-ID: <459DD0AD.6000707@hastings.org>

Ron Adam wrote:
> Improving pydoc has been suggested before by me and others.  I've been working
> on a version that is probably 80% done and would like to get feed back at this
> point to determine if I'm approaching this in the best way.
Just asking--are you going in a PEP-287-ly way as you work?  If not, 
would your work make PEP 287 easier to implement?

For those of us without eidetic memories, PEP 287 is "use 
reStructuredText for docstrings":
    http://www.python.org/dev/peps/pep-0287/

Cheers,


/larry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070104/dd8b50f0/attachment.html 

From nnorwitz at gmail.com  Fri Jan  5 06:35:16 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 4 Jan 2007 21:35:16 -0800
Subject: [Python-Dev] crashing on shutdown due to warning at shutdown
Message-ID: <ee2a432c0701042135s58f47907h8b8d78f0beb3eedb@mail.gmail.com>

I fixed the crash that was due to raising a warning on shutdown.  I
have heard about crashes at shutdown and wonder if this was the cause.
 There might be similar bugs lurking that assume PyModule_GetDict()
always returns a valid pointer.  It doesn't, it can return NULL.

I'm not sure if the original problem existed or not, but after this
fix on shutdown, I see:

warning: DBTxn aborted in destructor.  No prior commit() or abort().

I don't know if this is a real problem, a test problem, or what.

n

From rrr at ronadam.com  Fri Jan  5 06:38:56 2007
From: rrr at ronadam.com (Ron Adam)
Date: Thu, 04 Jan 2007 23:38:56 -0600
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459DD0AD.6000707@hastings.org>
References: <458C719F.3000904@ronadam.com> <459DD0AD.6000707@hastings.org>
Message-ID: <459DE470.4030000@ronadam.com>

Larry Hastings wrote:

> Just asking--are you going in a PEP-287-ly way as you work?  If not, 
> would your work make PEP 287 easier to implement?

Pydoc does no reformatting or changes to doc strings.  They are displayed "as 
is" in plain text.  About the only formatting that is done is to wrap long lines 
a bit better, such as 100 character length lines on a 80 character (or less) 
console window. In those cases, it tries to maintain the indent and break lines 
on white space.

The html pages produced also makes html, rfc, and pep referfences into links.

One of the goals is to make it easer to use it as a base for generating other 
types of formats.  So it should also make it easier for someone (else) to 
implement a PEP-287 extended version for their own needs.


> For those of us without eidetic memories, PEP 287 is "use 
> reStructuredText for docstrings":
>     http://www.python.org/dev/peps/pep-0287/

Thanks for the link. PEP 287 looks to be fairly general in that it expresses a 
general desire rather than a specification.

Ron


From talin at acm.org  Fri Jan  5 07:06:55 2007
From: talin at acm.org (Talin)
Date: Thu, 04 Jan 2007 22:06:55 -0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459DE470.4030000@ronadam.com>
References: <458C719F.3000904@ronadam.com> <459DD0AD.6000707@hastings.org>
	<459DE470.4030000@ronadam.com>
Message-ID: <459DEAFF.3080007@acm.org>

Ron Adam wrote:
> Larry Hastings wrote:
>> For those of us without eidetic memories, PEP 287 is "use 
>> reStructuredText for docstrings":
>>     http://www.python.org/dev/peps/pep-0287/
> 
> Thanks for the link. PEP 287 looks to be fairly general in that it expresses a 
> general desire rather than a specification.

Apologies for the digression, but I have a comment on this.

Rather than fixing on a standard markup, I would like to see support for 
a __markup__ module variable which specifies the specific markup 
language that is used in that module. Doc processors could inspect that 
variable and then load the appropriate markup translator.

Why? Because its hard to get everyone to agree on which markup language 
is best for documentation. I personally think that reStructuredText is 
not a good choice, because I want to add markup that adds semantic 
information, whereas reStructuredText deals solely with presentation and 
visual appearance. (In other words, I'd like to be able to define 
machine-readable metadata that identifies parameters, return values, and 
exceptions -- not just hyperlinks and text styles.) Having used a lot of 
different documentation markup languages, and written a few of them, I 
prefer "non-invasive" semantic markup as seen in markup processors such 
as Doc-o-matic and NaturalDocs. (By non-invasive, I mean that the markup 
doesn't detract in any way from the readability of the marked-up text. 
Doc-o-matic's markup language is very powerful, and yet unless you know 
what you are looking for you'd think its just regular prose.) I have a 
prototype (called "DocLobster") which does similar types of processing 
on Python docstrings, but I haven't publicized it because I didn't feel 
like competing against ReST.

However, I realize that I'm in the minority with this opinion; I don't 
want to force anyone to conform to my idea of markup, but at the same 
time I'd prefer not to have other people dictate my choice either.

Instead, what I'd like to see is a way for multiple markup languages to 
coexist and compete with each other on a level playing field, instead of 
one being chosen as the winner.

-- Talin

From greg at electricrain.com  Fri Jan  5 07:28:24 2007
From: greg at electricrain.com (Gregory P. Smith)
Date: Thu, 4 Jan 2007 22:28:24 -0800
Subject: [Python-Dev] crashing on shutdown due to warning at shutdown
In-Reply-To: <ee2a432c0701042135s58f47907h8b8d78f0beb3eedb@mail.gmail.com>
References: <ee2a432c0701042135s58f47907h8b8d78f0beb3eedb@mail.gmail.com>
Message-ID: <20070105062824.GZ24758@electricrain.com>

On Thu, Jan 04, 2007 at 09:35:16PM -0800, Neal Norwitz wrote:
> I fixed the crash that was due to raising a warning on shutdown.  I
> have heard about crashes at shutdown and wonder if this was the cause.
> There might be similar bugs lurking that assume PyModule_GetDict()
> always returns a valid pointer.  It doesn't, it can return NULL.
> 
> I'm not sure if the original problem existed or not, but after this
> fix on shutdown, I see:
> 
> warning: DBTxn aborted in destructor.  No prior commit() or abort().
> 
> I don't know if this is a real problem, a test problem, or what.
> 
> n

Its a courtesy warning, it should not cause harm.

could it possibly be from Lib/bsddb/test/test_1413192.py which makes
no attempt at cleaning up on purpose?

-greg

From larry at hastings.org  Fri Jan  5 07:44:16 2007
From: larry at hastings.org (Larry Hastings)
Date: Thu, 04 Jan 2007 22:44:16 -0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459DEAFF.3080007@acm.org>
References: <458C719F.3000904@ronadam.com>
	<459DD0AD.6000707@hastings.org>	<459DE470.4030000@ronadam.com>
	<459DEAFF.3080007@acm.org>
Message-ID: <459DF3C0.5020005@hastings.org>

Ron Adam wrote:
> Thanks for the link. PEP 287 looks to be fairly general in that it 
> expresses a general desire rather than a specification.
I thought it was pretty specific.  I'd summarize PEP 287 by quoting 
entry #1 from its "goals of this PEP" section:

    * To establish reStructuredText as a standard structured plaintext
      format for docstrings (inline documentation of Python modules and
      packages), PEPs, README-type files and other standalone documents.


Talin wrote:
> Rather than fixing on a standard markup, I would like to see support 
> for a __markup__ module variable which specifies the specific markup 
> language that is used in that module. Doc processors could inspect 
> that variable and then load the appropriate markup translator.
I guess I'll go for the whole-hog +1.0 here.  I was going to say +0.8, 
citing "There should be one---and preferably only one---obvious way to 
do it.".  But I can see organizations desiring something besides ReST, 
like if they already had already invested in their own internal 
standardized markup language and wanted to use that.

This makes the future clear; the default __markup__ in 2.6 would be 
"plain", so that all the existing docstrings work unmodified. At which 
point PEP 287 becomes "write a ReST driver for the new pydoc".  
Continuing my dreaming here, Python 3000 flips the switch so that the 
default __markup__ is "ReST", and the docstrings that ship with Python 
are touched up to match---or set explicitly to "plain" if some strange 
necessity required it.

(And when do you unveil DocLobster?)

Cheers,


/larry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070104/c8e158d3/attachment.html 

From greg at electricrain.com  Fri Jan  5 07:40:52 2007
From: greg at electricrain.com (Gregory P. Smith)
Date: Thu, 4 Jan 2007 22:40:52 -0800
Subject: [Python-Dev] kill the cbsoutdoor.co.uk autoresponder
Message-ID: <20070105064052.GA24758@electricrain.com>

Whoever is subscribed to python-dev with a broken corporate
autoresponder that sends everyone who posts to the list this useless
response multiple times please unsubscribe yourself.  Its highly
annoying and entirely useless since its not even identifying the list
subscriber(s) deserving the blame.

-g

On Fri, Jan 05, 2007 at 06:28:35AM +0000, noreply at cbsoutdoor.co.uk wrote:
> Dear Recipient,
> We have now changed our company name to CBS Outdoor Limited. My new email address is therefore firstname.surname at cbsoutdoor.co.uk  Your email has been redirected but please amend your address book accordingly.
>  
> Thank you.
> 
> 
> 
> 
> CBS Outdoor Limited
> 
> 
> 
> noreply at cbsoutdoor.co.uk
> www.cbsoutdoor.co.uk
> 
> The contents of this e-mail are confidential to the ordinary user of the e-mail address to which it was addressed, and may also be privileged. If you are not the addressee of this e-mail you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you have received this e-mail in error, please e-mail the sender by replying to this message. CBS Outdoor Ltd reserves the right to monitor e-mail communications from external/internal sources for the purposes of ensuring correct and appropriate use of CBS Outdoor facilities.



From talin at acm.org  Fri Jan  5 08:49:05 2007
From: talin at acm.org (Talin)
Date: Thu, 04 Jan 2007 23:49:05 -0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459DF3C0.5020005@hastings.org>
References: <458C719F.3000904@ronadam.com>	<459DD0AD.6000707@hastings.org>	<459DE470.4030000@ronadam.com>	<459DEAFF.3080007@acm.org>
	<459DF3C0.5020005@hastings.org>
Message-ID: <459E02F1.3000001@acm.org>

Larry Hastings wrote:
> Ron Adam wrote:
>> Thanks for the link. PEP 287 looks to be fairly general in that it 
>> expresses a general desire rather than a specification.
> I thought it was pretty specific.  I'd summarize PEP 287 by quoting 
> entry #1 from its "goals of this PEP" section:
> 
>    * To establish reStructuredText as a standard structured plaintext
>      format for docstrings (inline documentation of Python modules and
>      packages), PEPs, README-type files and other standalone documents.
> 
> 
> Talin wrote:
>> Rather than fixing on a standard markup, I would like to see support 
>> for a __markup__ module variable which specifies the specific markup 
>> language that is used in that module. Doc processors could inspect 
>> that variable and then load the appropriate markup translator.
> I guess I'll go for the whole-hog +1.0 here.  I was going to say +0.8, 
> citing "There should be one---and preferably only one---obvious way to 
> do it.".  But I can see organizations desiring something besides ReST, 
> like if they already had already invested in their own internal 
> standardized markup language and wanted to use that.
> 
> This makes the future clear; the default __markup__ in 2.6 would be 
> "plain", so that all the existing docstrings work unmodified. At which 
> point PEP 287 becomes "write a ReST driver for the new pydoc".  
> Continuing my dreaming here, Python 3000 flips the switch so that the 
> default __markup__ is "ReST", and the docstrings that ship with Python 
> are touched up to match---or set explicitly to "plain" if some strange 
> necessity required it.
> 
> (And when do you unveil DocLobster?)

Well, I'd be more interested in working on it once there's something to 
plug it into - I didn't really want to write a whole pydoc replacement, 
just a markup transformer.

One issue that needs to be worked out, however, is the division of 
responsibility between markup processor and output formatter. Does a 
__markup__ plugin do both jobs, or does it just do parsing, and leave 
the formatting of output to the appropriate HTML / text output module? 
How does the HTML output module know how to handle non-standard metadata?

Let me give an example: Suppose you have a simple markup language that 
has various section tags, such as "Author", "See Also", etc.:

    """
    Description:
       A long description of this thing whatever it is.

    Parameters:
       fparam - the first parameter
       sparam - the second parameter

    Raises:
       ArgumentError - when invalid arguments are passed.

    Author: Someone

    See Also:
       PyDoc
       ReST
    """

So the parser understands these various section headings - how does it 
tell the HTML output module that 'Author' is a section heading? 
Moreover, in the case of "Parameters" and "Exceptions", the content of 
the section is parsed as a table (parameter, description) which is 
stored as a list of tuples, whereas the content of the "Description" 
section is just a long string.

I guess the markup processor has to deliver some kind of DOM tree, which 
can be rendered either into text or into HTML. CSS can take over from 
that point on.

-- Talin

From fdrake at acm.org  Fri Jan  5 09:09:26 2007
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Fri, 5 Jan 2007 03:09:26 -0500
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459E02F1.3000001@acm.org>
References: <458C719F.3000904@ronadam.com> <459DF3C0.5020005@hastings.org>
	<459E02F1.3000001@acm.org>
Message-ID: <200701050309.27050.fdrake@acm.org>

On Friday 05 January 2007 02:49, Talin wrote:
 > One issue that needs to be worked out, however, is the division of
 > responsibility between markup processor and output formatter. Does a
 > __markup__ plugin do both jobs, or does it just do parsing, and leave
 > the formatting of output to the appropriate HTML / text output module?
 > How does the HTML output module know how to handle non-standard metadata?

There's already __docformat__; see:

  http://www.python.org/dev/peps/pep-0258/#choice-of-docstring-format


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From lgautier at gmail.com  Fri Jan  5 11:01:22 2007
From: lgautier at gmail.com (Laurent Gautier)
Date: Fri, 5 Jan 2007 18:01:22 +0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459D27FE.6070406@ronadam.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
Message-ID: <27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>

Ron,

Thanks for your detailed answer.
I inserted comments below.

2007/1/5, Ron Adam <rrr at ronadam.com>:
> Laurent Gautier wrote:
[cut]
>
> > Introspection is probably already available in the separate module
> > 'inspect',
> > and what a code pydoc would have to do is model the documentation
> > (as a tree) and offer convenience function to navigate the data. Beside
> > that,
> > there would be sub-modules for the different viewers for the documentation
> > data - the interactive console being just one of the viewers.
>
> Pydoc currently uses functions from the inspect module along with directly
> accessing attributes and other information where it is available.  It's not a
> replacement for the inspect module.
>
> My first attempt used an xml tree to store the information, but to make that
> work requires also storing a fair amount of meta information in the tree as
> well.  I found parsing the tree and the meta information to be more complex than
> using an objective approach which is (to me) more readable and easier to extend.
>   But if you want to try it again, please do.  You may come up with something
> far better than I did.

Well, I was coining the idea from the understanding that the main split was
console viewer vs other viewer. I was thinking of something a design
along the lines of the Model-View-Presenter pattern.... and I guess
that you will have to
read your code if I want to debate on that ;-).

> > Finally, I would suspect that an API-breaking modification of the module
> > would
> > need time to be accepted.
>
> I was considering this for python 3.0, but one of the developers suggested it
> would be nice to have in python 2.6 and to move the discussion here.
>
> I think any API issues could be worked out.  Are there any programs you know of,
> (yours?), that import pydoc besides the python console?

What I did barely qualifies as a hack for my own usage -it won't count-.

>From the top of my head, there might be  "ipython" (the excellent
interactive console) is possibly using pydoc
(in any case, I would say that the authors would be interested in
developments with pydoc)

Otherwise a quick search lead to:
- "cgitb" (!? - it seems that the HTML formatting functions of pydoc
are only in use - wouldn't these functions belong more naturally to "cgi" ?)
- "DocXMLRPCServer" (hey ! it looks like kind-of what I was needing !!!).
- "happydoc" (reportedly having problems with python 2.4 - I am not
sure that it is maintained)

"cgitb" and "DocXMLRPCServer" are both distributed bundled with Python.

"cgitb" seems to be mostly using HTML formatting helpers (and that would
suggest the need for an HTML-rendering module - may be for a future
improvement,  a first step would be separate the rendering/viewing
from extraction and modeling of documentation data).

"DocXMLRPCServer" looks (at first sight), like a viewer that would be bundled
with pydoc as a sub-module (i.e., module in a package).


>
[cut:  Ka-Ping Yee <ping at lfw.org> is now in the loop]
>

> Pydoc is a fairly complex program and it would definitely help if others took a
> look at various parts and made contributions and or suggestions to making it better.

Well, I stumbled upon your recent posts in python-ideas (that I tracked up the
one in python-devel) because I looked into it I thought that it would be
a *lot* of work for one person.
(more on that in the next inlined comment)

> I may have also gotten a bit over my head, but I'm willing to stick it out and
> try to get it finished with any suggestions (and help) that any one is willing
> to give me.  There are also too many important issues for me to be decided, so
> this isn't something that can be done in isolation.
>
> The download link again is:
>
>      http://ronadam.com/dl/_pydoc.zip

I would be willing to help out, as probably others will as well (I
found blogs and
posts of people discussing pydoc, it might be worthwhile dropping a line to
the people - we can discuss that off-list if you wish), but may be at
one condition.

I do not think it will work as a zip file shuttled around (in my experience).
A versioning system would be extremely helpful (SVN, or CVS. would
come to my mind).
Well, if you are ok with having the source tree hosted in a
SVN/CVS/alike I am on
(opening an account on sourceforge or savannah -for example- would be
the next step then, as it can take few days for a project to be
approved)

> It's not fully functional yet, but does run. Some parts like the command line
> file output options still need to be reimplemented. Some output formatting still
> needs to be cleaned up, and the MRO tree parsing section still needs to be put
> back in.

I will have a look then.

>
> One question that I have at the moment is:
>
> * Would it be good to have the "KEYWORD" and "TOPICS" info as included data
> objects or files, and possibly use that to generate the python html (and other)
> documentation for these?  (Instead of the other way around like it is now.)
>
> This would eliminate the requirement to install something extra in order for
> help on these items to work.

I see that I was slow to write this email. I will read the following
before commenting  further.



L.


> > L.
> >
> > PS: I would also not go for a module name deliberately prefixed with "_"
> > (as some people might associate that with protected or private objects).
>
> The underscore was just a "temporary" convenience to avoid the name conflict
> with the existing module.  Weather to reuse the old name or use a new name is
> still one of the many open issues I think.
>
> Ron
>
>

From python-dev at zesty.ca  Fri Jan  5 11:08:47 2007
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Fri, 5 Jan 2007 04:08:47 -0600 (CST)
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459E02F1.3000001@acm.org>
References: <458C719F.3000904@ronadam.com> <459DD0AD.6000707@hastings.org>
	<459DE470.4030000@ronadam.com> <459DEAFF.3080007@acm.org>
	<459DF3C0.5020005@hastings.org> <459E02F1.3000001@acm.org>
Message-ID: <Pine.LNX.4.58.0701050401421.23077@server1.LFW.org>

On Thu, 4 Jan 2007, Talin wrote:
> One issue that needs to be worked out, however, is the division of
> responsibility between markup processor and output formatter. Does a
> __markup__ plugin do both jobs, or does it just do parsing, and leave
> the formatting of output to the appropriate HTML / text output module?
> How does the HTML output module know how to handle non-standard metadata?
[...]
> I guess the markup processor has to deliver some kind of DOM tree, which
> can be rendered either into text or into HTML. CSS can take over from
> that point on.

If the markup processor is going to deliver a tree, let me just point
out that it would be a pretty major project to define the format of
that tree -- about as large as inventing ReST or any other markup
language, except that the design of such an intermediate format has
to foresee future changes to the input and be flexible enough to target
multiple output formats.  The design would also have to tackle the
question of whether the intermediate format should contain semantic
information (what about cross-references?) and what types of such
information should be allowed (e.g. names of modules, arguments,
exceptions, Python expressions, etc.)


-- ?!ng

From anthony at interlink.com.au  Fri Jan  5 12:06:59 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 5 Jan 2007 22:06:59 +1100
Subject: [Python-Dev] kill the cbsoutdoor.co.uk autoresponder
In-Reply-To: <20070105064052.GA24758@electricrain.com>
References: <20070105064052.GA24758@electricrain.com>
Message-ID: <200701052207.00508.anthony@interlink.com.au>

On Friday 05 January 2007 17:40, Gregory P. Smith wrote:
> Whoever is subscribed to python-dev with a broken corporate
> autoresponder that sends everyone who posts to the list this
> useless response multiple times please unsubscribe yourself.  Its
> highly annoying and entirely useless since its not even
> identifying the list subscriber(s) deserving the blame.

Can you determine the original email domain from the Received 
headers, perhaps?

From lgautier at gmail.com  Fri Jan  5 12:20:10 2007
From: lgautier at gmail.com (Laurent Gautier)
Date: Fri, 5 Jan 2007 19:20:10 +0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <Pine.LNX.4.58.0701041609240.23077@server1.LFW.org>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<Pine.LNX.4.58.0701041609240.23077@server1.LFW.org>
Message-ID: <27d1e6020701050320s670cfff4ne52cc8768ed26ef8@mail.gmail.com>

2007/1/5, Ka-Ping Yee <python-dev at zesty.ca>:
[cut]
> On the other hand, I've often seen the question of why pydoc does
> both text and HTML generation instead of generating some intermediate
> data structure from which both kinds of output are produced.  The
> answer is: I tried it.  The result turned out to be longer than
> I expected and needlessly more complicated than what we have now.
> It may be that a better job could have been done, but I think there
> is a rational basis for why it turned out that way.
>
> The Python objects themselves already are a data structure containing
> all of the information we need.  I discovered that translating this
> data structure into another data structure and then producing text
> or HTML was more work than simply producing text or HTML.  With CSS,
> the last step gets even easier and so the intermediate stage becomes
> even less necessary.  Also, the intermediate step required me to
> essentially invent an API, and I decided that I trusted the stability
> of Python's API more than that of some API I invented just for this.

Point well taken. This is very sensible.

I would still try to keep common-and-presenter-independent component.
Rather than a sole distinction console/HTML, I would think of a distinction
between stateless and interactive presenters, and still have the console and
static HTML as specific presenters. The search functions Ron suggested
would be part of that presenter-independent part (and for the refinement,
stateless vs. interactive would make sense for performances).

The distinction may look like an unnecessary complication... but I would
think that it does not have to be complicated, and that the number of practical
things it would allow would make it almost necessary (ah ! delusions ;-) ).
There a number of python editors/consoles/IDE around, some of which
are implemented in python, and having the necessary infrastructute to
let implement
easily documentation presenters would be very nice.

> This is not to say that text and HTML generation can't be separated;
> it's just a caution against attempting to overgeneralize by creating
> an intermediate format.  I'm glad you backed away from XML (or I'd
> have warned you that processing the XML would be a lot of extra work).

Your warning regarding the creation of  a n-th data structure is completely
agreed upon. I also understand your point about the dangers of
overgeneralizing.

> The inspect module was intended to pull out as much as possible of
> the extraction functionality that's shared by the text and HTML
> documentation generators.  But pydoc is still big.  At the time I was
> proposing pydoc for addition to the standard library, I didn't want
> to pollute the top-level module namespace with too many names, so I
> tried hard to minimize the number of modules.  And of course it has
> grown since then with bits of new functionality and support for new
> language features in Python.
>
> But now if a package is being considered, it makes sense to split
> out some of the pieces (as you have done), such as the web server,
> the search function, and the interactive interpreter help prompt.
> It may even enable pydoc to provide search from the interactive help
> prompt, which would be a nice feature!  The package could contain
> several modules for ease of maintenance, while still providing a
> single, convenient command for running pydoc from the Unix prompt.

Having things already split by Ron is probably a good starting base
(and generalization introduced progressively, if agreed upon).

I see that there is debating on the format for documentation strings,
may there is as well room for flexibility regarding how the strings are
utilized.

The search would be not only useful to the python console, but also to
other editors, as well as to editor (as well as python programs),
as well as to stateless presenters (the case I had was to work on a
server (web-hosting) on which I only had FTP access and on which I did
not know the python version or the package installed -hey, what about
an Ajax-capable HTML viewer for the documentation ?) -



Laurent


>
> -- ?!ng
>

From lgautier at gmail.com  Fri Jan  5 12:23:26 2007
From: lgautier at gmail.com (Laurent Gautier)
Date: Fri, 5 Jan 2007 19:23:26 +0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <Pine.LNX.4.58.0701050401421.23077@server1.LFW.org>
References: <458C719F.3000904@ronadam.com> <459DD0AD.6000707@hastings.org>
	<459DE470.4030000@ronadam.com> <459DEAFF.3080007@acm.org>
	<459DF3C0.5020005@hastings.org> <459E02F1.3000001@acm.org>
	<Pine.LNX.4.58.0701050401421.23077@server1.LFW.org>
Message-ID: <27d1e6020701050323q413aac3bxc2dad9e63694d200@mail.gmail.com>

2007/1/5, Ka-Ping Yee <python-dev at zesty.ca>:
> On Thu, 4 Jan 2007, Talin wrote:
> > One issue that needs to be worked out, however, is the division of
> > responsibility between markup processor and output formatter. Does a
> > __markup__ plugin do both jobs, or does it just do parsing, and leave
> > the formatting of output to the appropriate HTML / text output module?
> > How does the HTML output module know how to handle non-standard metadata?
> [...]
> > I guess the markup processor has to deliver some kind of DOM tree, which
> > can be rendered either into text or into HTML. CSS can take over from
> > that point on.
>
> If the markup processor is going to deliver a tree, let me just point
> out that it would be a pretty major project to define the format of
> that tree -- about as large as inventing ReST or any other markup
> language, except that the design of such an intermediate format has
> to foresee future changes to the input and be flexible enough to target
> multiple output formats.  The design would also have to tackle the
> question of whether the intermediate format should contain semantic
> information (what about cross-references?) and what types of such
> information should be allowed (e.g. names of modules, arguments,
> exceptions, Python expressions, etc.)
>

Wouldn't it be conceivable to have the processing of the markup
performed by a separate function, that could eventually be overridden/passed
as a parameter  when specific needs regarding the markup are needed  ?


L.

> -- ?!ng
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com
>

From amk at amk.ca  Fri Jan  5 13:30:26 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 5 Jan 2007 07:30:26 -0500
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
Message-ID: <20070105123026.GA12988@andrew-kuchlings-computer.local>

On Fri, Jan 05, 2007 at 06:01:22PM +0800, Laurent Gautier wrote:
> Well, if you are ok with having the source tree hosted in a
> SVN/CVS/alike I am on
> (opening an account on sourceforge or savannah -for example- would be
> the next step then, as it can take few days for a project to be
> approved)

The Python SVN repository has a sandbox/ directory that's intended for
storing code in development; you could certainly use that.

--amk


From lgautier at gmail.com  Fri Jan  5 14:12:28 2007
From: lgautier at gmail.com (Laurent Gautier)
Date: Fri, 5 Jan 2007 21:12:28 +0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <20070105123026.GA12988@andrew-kuchlings-computer.local>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
	<20070105123026.GA12988@andrew-kuchlings-computer.local>
Message-ID: <27d1e6020701050512m433a1db6k2e5c8597963cb349@mail.gmail.com>

2007/1/5, A.M. Kuchling <amk at amk.ca>:
> On Fri, Jan 05, 2007 at 06:01:22PM +0800, Laurent Gautier wrote:
> > Well, if you are ok with having the source tree hosted in a
> > SVN/CVS/alike I am on
> > (opening an account on sourceforge or savannah -for example- would be
> > the next step then, as it can take few days for a project to be
> > approved)
>
> The Python SVN repository has a sandbox/ directory that's intended for
> storing code in development; you could certainly use that.

I suspect that this is aside from the rest of the python source tree.
(or I would anticipate peppered emails if the module is broken during
its early days
-and it will- ).

I also suspect that write access is not granted easily (as I know a
number of python modules being on sourceforge before being ultimately
included in
the standard modules).

> --amk
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com
>

From ndbecker2 at gmail.com  Fri Jan  5 14:13:39 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 05 Jan 2007 08:13:39 -0500
Subject: [Python-Dev] Pydoc Improvements / Rewrite
References: <458C719F.3000904@ronadam.com>
Message-ID: <enliu3$kg6$1@sea.gmane.org>

No time to review this now, but I'd just like to say that the 1 thing I'd
like to see is support for decent mathematical markup.  I think at this
point that support for latex markup is the way to achieve this.


From fredrik at pythonware.com  Fri Jan  5 14:29:11 2007
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Fri, 5 Jan 2007 14:29:11 +0100
Subject: [Python-Dev] Pydoc Improvements / Rewrite
References: <458C719F.3000904@ronadam.com>
	<459DD0AD.6000707@hastings.org><459DE470.4030000@ronadam.com>
	<459DEAFF.3080007@acm.org>
Message-ID: <enljr7$o6s$1@sea.gmane.org>

Talin wrote:

> Rather than fixing on a standard markup, I would like to see support for
> a __markup__ module variable which specifies the specific markup
> language that is used in that module. Doc processors could inspect that
> variable and then load the appropriate markup translator.

Ideally, a module should be able to specify what *documentation provider*
to use.  Not everyone wants to stuff everything into docstrings, and, especially
if you're building larger components, automatic introspection simply doesn't
work very well.

fwiw, I have hacks for PythonDoc that monkey-patches "inspect" to provide
"virtual docstrings", but it would be nice to have an official API for this.  It
doesn't have to be much more complicated than:

    def __inspect__(path, format_hint=None):
        ...
        return format, data, subpaths

where path is a dotted path to the target object, and format_hint is a preferred
format.

> Why? Because its hard to get everyone to agree on which markup language
> is best for documentation. I personally think that reStructuredText is
> not a good choice, because I want to add markup that adds semantic
> information, whereas reStructuredText deals solely with presentation and
> visual appearance.

And does a rather bad job at that too (the "squint if you don't want to see the
markup" approach is fundamentally flawed), but that's another story for another
forum.

</F> 




From ronaldoussoren at mac.com  Fri Jan  5 14:37:02 2007
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 05 Jan 2007 05:37:02 -0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <enljr7$o6s$1@sea.gmane.org>
References: <458C719F.3000904@ronadam.com> <459DD0AD.6000707@hastings.org>
	<459DE470.4030000@ronadam.com> <459DEAFF.3080007@acm.org>
	<enljr7$o6s$1@sea.gmane.org>
Message-ID: <8A92FBEE-010F-1000-E5F6-80FE2E49B06F-Webmail-10018@mac.com>

 
On Friday, January 05, 2007, at 02:30PM, "Fredrik Lundh" <fredrik at pythonware.com> wrote:
>Talin wrote:
>
>> Rather than fixing on a standard markup, I would like to see support for
>> a __markup__ module variable which specifies the specific markup
>> language that is used in that module. Doc processors could inspect that
>> variable and then load the appropriate markup translator.
>
>Ideally, a module should be able to specify what *documentation provider*
>to use.  Not everyone wants to stuff everything into docstrings, and, especially
>if you're building larger components, automatic introspection simply doesn't
>work very well.

+lots on that. This is not only true for larger components but projects like wxpython, pyqt and pyobjc could also use these hooks to add links to the C version of those libraries (I don't know about pyqt but wxpython and pyobjc both have choosen to document the differences with the C library instead of trying to duplicate their work).

Ronald

From barry at python.org  Fri Jan  5 15:29:34 2007
From: barry at python.org (Barry Warsaw)
Date: Fri, 5 Jan 2007 09:29:34 -0500
Subject: [Python-Dev] kill the cbsoutdoor.co.uk autoresponder
In-Reply-To: <200701052207.00508.anthony@interlink.com.au>
References: <20070105064052.GA24758@electricrain.com>
	<200701052207.00508.anthony@interlink.com.au>
Message-ID: <E40372AB-4FF2-46F8-8764-DE17BC5BA9B9@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 5, 2007, at 6:06 AM, Anthony Baxter wrote:

> On Friday 05 January 2007 17:40, Gregory P. Smith wrote:
>> Whoever is subscribed to python-dev with a broken corporate
>> autoresponder that sends everyone who posts to the list this
>> useless response multiple times please unsubscribe yourself.  Its
>> highly annoying and entirely useless since its not even
>> identifying the list subscriber(s) deserving the blame.
>
> Can you determine the original email domain from the Received
> headers, perhaps?

I think I found the offending address, which I won't name on this  
mailing list.  I have disabled the address so please let me know if  
you continue to see the autoresponses so that we can either rejoice  
or I can re-enable the poor guy and we can all continue to despair.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)

iQCVAwUBRZ5gznEjvBPtnXfVAQImwwP8DDnOUXWDbwN0ZnHsAHj18qxAUDEL7sH/
JfXUH74RYIp2h/YGKUqY1WCEtcqRaTeKpqh2TZLChZkAsXcSEU9r8cwJY9ReUOkE
13JZRy4xh52wlnF5G6gChUBSAueks74/OWsk8XYO46bsc1OGSsMQB3ee/9LI57DF
cpLJCPW4jKw=
=aPTq
-----END PGP SIGNATURE-----

From amk at amk.ca  Fri Jan  5 15:29:49 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 5 Jan 2007 09:29:49 -0500
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <27d1e6020701050512m433a1db6k2e5c8597963cb349@mail.gmail.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
	<20070105123026.GA12988@andrew-kuchlings-computer.local>
	<27d1e6020701050512m433a1db6k2e5c8597963cb349@mail.gmail.com>
Message-ID: <20070105142949.GD4930@localhost.localdomain>

On Fri, Jan 05, 2007 at 09:12:28PM +0800, Laurent Gautier wrote:
> I suspect that this is aside from the rest of the python source tree.
> (or I would anticipate peppered emails if the module is broken during
> its early days
> -and it will- ).

Correct; it can be browsed at <http://svn.python.org/view/sandbox/trunk/>

> I also suspect that write access is not granted easily (as I know a
> number of python modules being on sourceforge before being ultimately
> included in
> the standard modules).

I think most of those projects weren't initially started with the goal
of stdlib inclusion.  The problems with a separate Sourceforge project are:

* fewer people will see the commit messages.
* if the project isn't completed, everyone will forget about the SF 
  project in a few year.  In sandbox/, at least the code will still be
  somewhat visible; maybe someone would finish it.

--amk

From barry at python.org  Fri Jan  5 15:36:30 2007
From: barry at python.org (Barry Warsaw)
Date: Fri, 5 Jan 2007 09:36:30 -0500
Subject: [Python-Dev] kill the cbsoutdoor.co.uk autoresponder
In-Reply-To: <E40372AB-4FF2-46F8-8764-DE17BC5BA9B9@python.org>
References: <20070105064052.GA24758@electricrain.com>
	<200701052207.00508.anthony@interlink.com.au>
	<E40372AB-4FF2-46F8-8764-DE17BC5BA9B9@python.org>
Message-ID: <181F1925-59C8-474E-8CE9-957DD22A34A8@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 5, 2007, at 9:29 AM, Barry Warsaw wrote:

> On Jan 5, 2007, at 6:06 AM, Anthony Baxter wrote:
>
>> On Friday 05 January 2007 17:40, Gregory P. Smith wrote:
>>> Whoever is subscribed to python-dev with a broken corporate
>>> autoresponder that sends everyone who posts to the list this
>>> useless response multiple times please unsubscribe yourself.  Its
>>> highly annoying and entirely useless since its not even
>>> identifying the list subscriber(s) deserving the blame.
>>
>> Can you determine the original email domain from the Received
>> headers, perhaps?
>
> I think I found the offending address, which I won't name on this
> mailing list.  I have disabled the address so please let me know if
> you continue to see the autoresponses so that we can either rejoice
> or I can re-enable the poor guy and we can all continue to despair.

Ah, I didn't get a response spam for that message, and I /did/ get  
one when I sent a test email to the suspect address.  So I think I  
nailed it.  I've also sent a polite <wink> email to their postmaster  
in complaint.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)

iQCVAwUBRZ5ib3EjvBPtnXfVAQKrSgP/ceak+5odGhVjgippP4Gy+ya9WbY98UJ8
71iICp5eGgvVJeD6paoMZhlHvFDezc7muWycMS43ec51TRn979xJnmSbMLNUeVZa
4HPCmOgMUuNkTz843maQaXyA/xtDcETo2EWCpQa6NuWS0e+TEiEeUeaeiosAPWnI
UubWVqtBaMk=
=P5FM
-----END PGP SIGNATURE-----

From amk at amk.ca  Fri Jan  5 17:17:40 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 5 Jan 2007 11:17:40 -0500
Subject: [Python-Dev] 2.5.1 plans
In-Reply-To: <ee2a432c0701041347p6c7734a6r7932dc721f75ce57@mail.gmail.com>
References: <20070104140458.92703.qmail@web31103.mail.mud.yahoo.com>
	<3d2ce8cb0701041022p572c9626o74b2088fa017c3b9@mail.gmail.com>
	<20070104210015.GA15252@localhost.localdomain>
	<ee2a432c0701041347p6c7734a6r7932dc721f75ce57@mail.gmail.com>
Message-ID: <20070105161740.GA8740@localhost.localdomain>

On Thu, Jan 04, 2007 at 01:47:13PM -0800, Neal Norwitz wrote:
> We have the buildbots to help with this.

According to http://www.python.org/dev/buildbot/trunk/ we do not have
a single working XP or Cygwin buildbot right now.

> Definitely!  I only did a really quick review.  If you want someone to
> think about any of these, assign them to me with a priority of 9.
> I'll discuss with others and see if the fixes should go in to 2.5.1 or
> not.

Bug #1599254 is the most critical for me (possible data loss with
mailbox.py); I'll try to fix it for 2.5.1.  I've changed its priority
but not reassigned it.

Should the fix for bug #1552726 (interactive Python polls every 1/10
sec) be applied to 2.5.1?  It's already been applied to trunk.  The
bug was noted by an OLPC person, but I don't think this bug is really
very critical for them.  Maybe we should ask them.  I've bumped the
priority as a reminder; if we decide to not apply it to 2.5.1, the bug
can be closed.

--amk

From rrr at ronadam.com  Fri Jan  5 22:44:24 2007
From: rrr at ronadam.com (Ron Adam)
Date: Fri, 05 Jan 2007 15:44:24 -0600
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>	
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
Message-ID: <459EC6B8.7000600@ronadam.com>

Laurent Gautier wrote:
 > Ron,
 >
 > Thanks for your detailed answer.
 > I inserted comments below.

You welcome.

 >> I think any API issues could be worked out.  Are there any programs
 >> you know of,
 >> (yours?), that import pydoc besides the python console?
 >
 > What I did barely qualifies as a hack for my own usage -it won't count-.

It could be these changes will give you a way to do the same thing in a less 
haskish way.


 > From the top of my head, there might be  "ipython" (the excellent
 > interactive console) is possibly using pydoc
 > (in any case, I would say that the authors would be interested in
 > developments with pydoc)

According to the web site, ipython is based on twisted, and is currently still 
limited to python 2.3 and 2.4.  Also, the output of the help() function will not 
change much so I doubt it would be a problem for them.

 > Otherwise a quick search lead to:
 > - "cgitb" (!? - it seems that the HTML formatting functions of pydoc
 > are only in use - wouldn't these functions belong more naturally to
 > "cgi" ?)

Thanks!, These html formatting functions still exist or are small enough to move 
into cgitb, so it will be just a matter of making sure they can be reached.  I 
don't think they will be a problem.


 > - "DocXMLRPCServer" (hey ! it looks like kind-of what I was needing !!!).

Thanks again.  This might be better to move into the pydoc package.  Any opinions?


 > - "happydoc" (reportedly having problems with python 2.4 - I am not
 > sure that it is maintained)

Happydoc does not import pydoc as far as I could tell, so this won't effect them 
in any direct way I think.  They've pretty much implemented everything from 
scratch.  At worse they would just need to copy the parts from the older version 
into their distribution.

I think you got a false positive on this because pydoc is a substring of happydoc.


 > "cgitb" and "DocXMLRPCServer" are both distributed bundled with Python.
 >
 > "cgitb" seems to be mostly using HTML formatting helpers (and that would
 > suggest the need for an HTML-rendering module - may be for a future
 > improvement,  a first step would be separate the rendering/viewing
 > from extraction and modeling of documentation data).

Making sure these still work would be a good sub project for someone a little 
later.  (I'll do it if no one else has time or wants to.)  I'm trying not to 
change thing to drastically.  If the changes are too big, ie... introducing and 
altering a lot of other modules.  Then this will need to move back to the 
python-3000 list.


 > "DocXMLRPCServer" looks (at first sight), like a viewer that would be
 > bundled
 > with pydoc as a sub-module (i.e., module in a package).

Yes, that was my thought too. :-)

But moving it may need to wait until python-3000.  (?)


 >> Pydoc is a fairly complex program and it would definitely help if
 >> others took a
 >> look at various parts and made contributions and or suggestions to
 >> making it better.
 >
 > Well, I stumbled upon your recent posts in python-ideas (that I tracked
 > up the
 > one in python-devel) because I looked into it I thought that it would be
 > a *lot* of work for one person.
 > (more on that in the next inlined comment)

It was tedious going though the module, but now that it's split up into smaller 
parts it shouldn't be too difficult.


 >> I may have also gotten a bit over my head, but I'm willing to stick it
 >> out and
 >> try to get it finished with any suggestions (and help) that any one is
 >> willing
 >> to give me.  There are also too many important issues for me to be
 >> decided, so
 >> this isn't something that can be done in isolation.
 >>
 >> The download link again is:
 >>
 >>      http://ronadam.com/dl/_pydoc.zip
 >
 > I would be willing to help out, as probably others will as well (I
 > found blogs and
 > posts of people discussing pydoc, it might be worthwhile dropping a line to
 > the people - we can discuss that off-list if you wish), but may be at
 > one condition.
 >
 > I do not think it will work as a zip file shuttled around (in my
 > experience).
 > A versioning system would be extremely helpful (SVN, or CVS. would
 > come to my mind).
 > Well, if you are ok with having the source tree hosted in a
 > SVN/CVS/alike I am on
 > (opening an account on sourceforge or savannah -for example- would be
 > the next step then, as it can take few days for a project to be
 > approved)

I don't have any experience with SVN, but it could be an opportunity to learn 
something new.  I have a couple of other projects that could benefit by moving 
them to SVN or CVS like system.

What I intended was to get enough feed back that I could get it to a point where 
it's 90% done and then upload it as a patch where it could be further polished 
up.  To do that, I first need to verify my design goals are the correct 
approach. (see my reply to Ka-Ping.)  If they are, then it's more a matter of 
cleaning up what I've already started.  If not, then it's a bit (or bunch) more 
work and would benefit from having others work on it in a more formal way.

If someone who has more experience with group projects would like to manage it, 
that would be good too.  That may speed things up considerably.


 > I will have a look then.

Great.  :-)


Cheers,
   Ron

From rrr at ronadam.com  Sat Jan  6 00:11:05 2007
From: rrr at ronadam.com (Ron Adam)
Date: Fri, 05 Jan 2007 17:11:05 -0600
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459DEAFF.3080007@acm.org>
References: <458C719F.3000904@ronadam.com>
	<459DD0AD.6000707@hastings.org>	<459DE470.4030000@ronadam.com>
	<459DEAFF.3080007@acm.org>
Message-ID: <459EDB09.20604@ronadam.com>

Talin wrote:
> Ron Adam wrote:
>> Larry Hastings wrote:
>>> For those of us without eidetic memories, PEP 287 is "use 
>>> reStructuredText for docstrings":
>>>     http://www.python.org/dev/peps/pep-0287/
>> Thanks for the link. PEP 287 looks to be fairly general in that it expresses a 
>> general desire rather than a specification.
> 
> Apologies for the digression, but I have a comment on this.
> 
> Rather than fixing on a standard markup, I would like to see support for 
> a __markup__ module variable which specifies the specific markup 
> language that is used in that module. Doc processors could inspect that 
> variable and then load the appropriate markup translator.
> 
> Why? Because its hard to get everyone to agree on which markup language 
> is best for documentation. I personally think that reStructuredText is 
> not a good choice, because I want to add markup that adds semantic 
> information, whereas reStructuredText deals solely with presentation and 
> visual appearance. (In other words, I'd like to be able to define 
> machine-readable metadata that identifies parameters, return values, and 
> exceptions -- not just hyperlinks and text styles.) Having used a lot of 
> different documentation markup languages, and written a few of them, I 
> prefer "non-invasive" semantic markup as seen in markup processors such 
> as Doc-o-matic and NaturalDocs. (By non-invasive, I mean that the markup 
> doesn't detract in any way from the readability of the marked-up text. 
> Doc-o-matic's markup language is very powerful, and yet unless you know 
> what you are looking for you'd think its just regular prose.) I have a 
> prototype (called "DocLobster") which does similar types of processing 
> on Python docstrings, but I haven't publicized it because I didn't feel 
> like competing against ReST.
> 
> However, I realize that I'm in the minority with this opinion; I don't 
> want to force anyone to conform to my idea of markup, but at the same 
> time I'd prefer not to have other people dictate my choice either.
> 
> Instead, what I'd like to see is a way for multiple markup languages to 
> coexist and compete with each other on a level playing field, instead of 
> one being chosen as the winner.

How about if plain text be the default, with the ability to over ride it to 
generate another type of output?  This is pretty much the design I'm following. 
  It doesn't choose any markup style or have a preference other than plain text.

Basically you import the gettext module, then add methods to the class's for 
each section to produce the marked up output you want. Then add a few functions 
to assemble it into a page.  I'm sure there's room for improvements, but this 
seems like the most direct way to do it.

Pydoc doesn't just process doc strings.  If that was all there was to it, then 
it would be much easier to automate in a generalized way.  Each section, may get 
it's data from different sources other than doc strings, and those may need to 
be handled in special ways, especially in the cases where it is nested within 
other sections.  This is what makes it difficult to generalize into a sequential 
producer to consumer pattern.  Not weather or not the doc string has additional 
markup in it.

I think the main goal for 2.6 should be a cleaned up package with some modest 
user interface and API improvements.  But also to keep things open ended for 
later enhancements.


[Maybe for python 3.0]

While reading some of the other discussions on multi-methods, generic functions, 
  and ABC's with mixins, it occurred to me that pydoc may be a good place to 
test some of those ideas in.  It's complex enough of a problem that it may 
benefit from those more advanced python features.

But I think that we shouldn't wait for those to do a basic face lift now.  Think 
of it a preliminary clean up if you will in this case.

Ron


From brett at python.org  Sat Jan  6 01:34:18 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 5 Jan 2007 16:34:18 -0800
Subject: [Python-Dev] Rewrite of import in Python source (sans docs) is
	complete
Message-ID: <bbaeab100701051634g74a63c77lcc495f2ffc73bbd0@mail.gmail.com>

Finally, after a few months worth of work, I have finally gotten far enough
in my import rewrite that I am willing to stick my neck out and say it is
semantically complete!  You can find it in the sandbox under import_in_py.

So, details of this implementation.  I implemented PEP 302 importers/loaders
for built-in, frozen, extension, .py, and .pyc files along with rewriting
the steps __import__ goes through to do an import.  I also developed an API
for .py/.pyc file handling so that there is a generic filesystem
importer/loader and a separate handler for .py/.pyc files.  This should
allow for (relatively) easy selective overriding of just how .py/.pyc files
are stored (e.g., introducing a database backend) or how variants on
.py/.pyc files are handled (e.g., Quixote's .ptl format).

This code has extensive tests and so I am fairly confident that it does what
is expected of an import rewrite.  There are actually more lines in the test
file than the implementation.  There is also a mock implementation used for
testing.  Was interesting doing this in such a test-driven, XP style of only
coding what I needed.

I have run this code through the entire regression test suite and that is
where you find out subtle differences between this implementation and the
built-in import (you can see for yourself with the regrtest.sh shell
script).  First test_pkg will fail because currently the new import adds a
__loader__ attribute on all modules (that might change for security reasons)
and test_pkg is an old, stdout comparing test.  Second, test_runpy fails
because I have not implemented get_code on the filesystem loader which is
required by runpy.  Both are shallow issues that can be dealt with.

Third, and the hardest difference to deal with, is that you will get some
warnings that print out that you normally don't see.  This is because
warnings.warn and its stack_level argument don't have the effect people are
used to when importing a deprecated module.  Before you could set
stack_level to 2 and it would look like it came from the import statement.
But now, with import written in Python and thus on the call stack compared
to being in C and thus not showing up, two levels back is still in the
import code.  I really don't know how this should be dealt with short of
saying that the rule of thumb with 2 stack levels back for a warning does
not work when done at the import level.

It is not blazing fast at the moment.  Some things, like the built-in and
frozen importers/loaders could be rewritten in C without huge issue.  I am
also sure I have made some stupid design decisions at various points in the
code.  But there is benchmarking code in the sandbox called importbench and
it showed up a  10x speed slowdown on a Linux box I was using in mid to late
December when doing a fresh import of certain types (I don't remember
exactly which kind off the top of my head).

Because of this current slowness I don't know if people want to rush into
trying to make this the default import implementation quite yet or if this
is not too big of a thing since the common case of pulling out of
sys.modules is not that much slower.  I know I am currently not planning on
devoting the time to bootstrap it in as I have my security work to finish
first along with other Python stuff that seems more pressing to me.  And
since (I think) I don't need to bootstrap it in order to finish my security
work I can't justify spending work time on it.  But I can rearrange
priorities if people really want to pursue this (especially if I can get
some help with it).

As with the module's name, it is currently named 'importer', but that is bad
since it conflicts with the idea of importers from PEP 302.  I was thinking
importlib, but I wanted to wait and see what other people thought.

Don't know if you guys are okay with me checking this in without having it
vetted by the community first like we prefer all new modules to do.  I have
not done the LaTeX docs yet.

I think that is all of the details that I can think of.  I am still working
towards implementing the security needed so that an application that embeds
Python can execute arbitrary code securely.  Giving a talk at PyCon on the
topic for anyone interested.

Special thanks needs to go to Paul Moore who I talked to through most of the
design of the code.  Nick Coghlan also provided some handy feedback.  And
Neal Norwitz for bugging about wanting something like this done.  Plus
thanks to everyone who has shown support.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070105/c4f7bebe/attachment.htm 

From martin at v.loewis.de  Sat Jan  6 01:41:41 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 06 Jan 2007 01:41:41 +0100
Subject: [Python-Dev] Renaming Include/object.h
In-Reply-To: <459CF1FB.9010602@tin.it>
References: <459BD46D.6090505@v.loewis.de>	<200701031120.13083.fdrake@acm.org>		<ca471dc20701030938r62aae4c3x23ed25e9f91e8a58@mail.gmail.com>		<9e804ac0701031011i2b89b9e6p3dd3caa340160f6a@mail.gmail.com>	<ee2a432c0701032028s6421d8b2k149ed9ccd5424814@mail.gmail.com>	<459CA9A3.8070601@v.loewis.de>
	<459CF1FB.9010602@tin.it>
Message-ID: <459EF045.30608@v.loewis.de>

Andrea Griffini schrieb:
> I've a partially related question... why isn't the module structure in 
> an include file .h
> and is instead in Objects/moduleobject.c ?
> For the cached lookup optimization I copied the definition but that's surely
> a bad way to do it.... I however wondered if there were good reasons for
> module objects for not being published.

I guess it's included in the C file because that's the easiest way to
implement it. AFAICT, it has been that way from the beginning.

Regards,
Martin

From fperez.net at gmail.com  Sat Jan  6 02:35:25 2007
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 05 Jan 2007 18:35:25 -0700
Subject: [Python-Dev] Pydoc Improvements / Rewrite
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
	<459EC6B8.7000600@ronadam.com>
Message-ID: <enmucu$nb9$1@sea.gmane.org>

Ron Adam wrote:

> Laurent Gautier wrote:

>  > From the top of my head, there might be  "ipython" (the excellent
>  > interactive console) is possibly using pydoc
>  > (in any case, I would say that the authors would be interested in
>  > developments with pydoc)

Certainly :)  I'd like to ask whether this discussion considers any kind of
merging of pydoc with epydoc.  Many projects (ipython included) are moving
towards epydoc as a more capable system than pydoc, though it would be nice
to see its API-doc-generation capabilities be part of the stdlib.  I don't
know if that's considered either too large or too orthogonal to the current
direction.

> According to the web site, ipython is based on twisted, and is currently
> still
> limited to python 2.3 and 2.4.  Also, the output of the help() function
> will not change much so I doubt it would be a problem for them.

A few corrections:

- IPython's trunk is NOT based on twisted at all, it's a self-contained
Python package which depends only on the Python standard library (plus
readline support under windows, which we also provide but requires ctypes).

- The ipython development branch does use twisted, but only for its
distributed and parallel computing capabilities.  Eventually when that
branch becomes trunk, there will /always/ be a non-twisted component for
local, terminal-based work.

- The last release (0.7.3) fully supports Python 2.5.  In fact, one nasty
bug in 2.5 with extremely slow traceback generation was kindly fixed by
python-dev in the nick of time after my pestering (an ipython user found it
first and brought it to my attention).

>  > Otherwise a quick search lead to:
>  > - "cgitb" (!? - it seems that the HTML formatting functions of pydoc
>  > are only in use - wouldn't these functions belong more naturally to
>  > "cgi" ?)
> 
> Thanks!, These html formatting functions still exist or are small enough
> to move
> into cgitb, so it will be just a matter of making sure they can be
> reached.  I don't think they will be a problem.

If anyone is interested in updating cgitb, you might want to look at
ipython's ultratb (which was born as a cgitb port to ANSI terminals):

http://projects.scipy.org/ipython/ipython/browser/ipython/trunk/IPython/ultraTB.py

It contains functionality for generating /extremely/ detailed tracebacks
with gobs of local detail.  These verbose tracebacks have let me fix many
ipython bugs from crash dumps triggered by remote code and libraries I
don't have access to, in cases where a normal traceback would have been
insufficient.  Here's a link to a slightly outdated simple example (the
formatting has improved a bit):

http://ipython.scipy.org/screenshots/snapshot1.png

Obviously the right thing to do would be to separate the ANSI coloring from
the structural formatting, so that the traceback could be formatted as
HTML, ANSI colors or anything else.  That is very much on my todo list,
since the network-enabled ipython will have browser-based interfaces in the
future.

All of ipython is BSD licensed, so any code in there is for the taking.

Best,

f


From lgautier at gmail.com  Sat Jan  6 03:21:41 2007
From: lgautier at gmail.com (Laurent Gautier)
Date: Sat, 6 Jan 2007 10:21:41 +0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <459EC6B8.7000600@ronadam.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
	<459EC6B8.7000600@ronadam.com>
Message-ID: <27d1e6020701051821r749b8f4dya5b28107b8f410ba@mail.gmail.com>

2007/1/6, Ron Adam <rrr at ronadam.com>:
> Laurent Gautier wrote:
[cut]
>  >> I think any API issues could be worked out.  Are there any programs
>  >> you know of,
>  >> (yours?), that import pydoc besides the python console?
>  >
>  > What I did barely qualifies as a hack for my own usage -it won't count-.
>
> It could be these changes will give you a way to do the same thing in a less
> haskish way.
>

This precisely why I get myself into the present trouble ;-)

>  > From the top of my head, there might be  "ipython" (the excellent
>  > interactive console) is possibly using pydoc
>  > (in any case, I would say that the authors would be interested in
>  > developments with pydoc)
>
> According to the web site, ipython is based on twisted, and is currently still
> limited to python 2.3 and 2.4.  Also, the output of the help() function will not
> change much so I doubt it would be a problem for them.

Sorry for answering a bit off-the-question. My meaning was that they would be
interested in knowning that "pydoc" is changing (and would surely have ideas).

>  > Otherwise a quick search lead to:
>  > - "cgitb" (!? - it seems that the HTML formatting functions of pydoc
>  > are only in use - wouldn't these functions belong more naturally to
>  > "cgi" ?)
>
> Thanks!, These html formatting functions still exist or are small enough to move
> into cgitb, so it will be just a matter of making sure they can be reached.  I
> don't think they will be a problem.
>

I read your comment about having not too many things changed for 2.6.
(or that will be bumped to 3000).

A suggestion I would have would be to create an html/htmlrender module
in the pydoc-package-to-be and start putting all the html formatting function
(as they are completely independent of pydoc, as far as I can see, over there).
You can then create wrappers to the original functions including a deprecation
warning. You can refer to Michael Chermside's recipe for an example of
implementation with a deprecation decorator -
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/391367
)
The suggestion above would actually apply to *anything* that is changed
in pydoc, providing the benefit of allowing the necessary changes while having
a temporary API to provide back-compatibility.


>
>  > - "DocXMLRPCServer" (hey ! it looks like kind-of what I was needing !!!).
>
> Thanks again.  This might be better to move into the pydoc package.  Any opinions?
>

We both pretty much agree, but now we've got to find the modules depending
on DocXMLRPCServer (and hope the recursion won't go on for too long). We
would also have to contact the author of DocXMLRPCServer (moving it would
go with potential changes, and he would certainly have suggestions about that).

>  > - "happydoc" (reportedly having problems with python 2.4 - I am not
>  > sure that it is maintained)
>
> Happydoc does not import pydoc as far as I could tell, so this won't effect them
> in any direct way I think.  They've pretty much implemented everything from
> scratch.  At worse they would just need to copy the parts from the older version
> into their distribution.
>
> I think you got a false positive on this because pydoc is a substring of happydoc.
>

Yes this was a false positive.
I kept it in the list thinking there might be interesting ideas there as well
(but I forget to label it as such - sorry for the confusion).

>  > "cgitb" and "DocXMLRPCServer" are both distributed bundled with Python.
>  >
>  > "cgitb" seems to be mostly using HTML formatting helpers (and that would
>  > suggest the need for an HTML-rendering module - may be for a future
>  > improvement,  a first step would be separate the rendering/viewing
>  > from extraction and modeling of documentation data).
>
> Making sure these still work would be a good sub project for someone a little
> later.  (I'll do it if no one else has time or wants to.)  I'm trying not to
> change thing to drastically.  If the changes are too big, ie... introducing and
> altering a lot of other modules.  Then this will need to move back to the
> python-3000 list.
>

I agree that altering a number of other modules would be a little
tricky to manage
for version 2.6, but I would think that the makeover of pydoc would
benefit from being made early (back-compatibility being ensured by the
deprecation decorator
mentioned above, for example).

>  > "DocXMLRPCServer" looks (at first sight), like a viewer that would be
>  > bundled
>  > with pydoc as a sub-module (i.e., module in a package).
>
> Yes, that was my thought too. :-)
>
> But moving it may need to wait until python-3000.  (?)

An other way is to move it, while keeping a dummy module that imports the
module from its new location (and issue deprecation warnings).
What I fear is that the "let's wait for python-3000"  correspond to postponing
changes to "some other time".

[cut]
>  > I do not think it will work as a zip file shuttled around (in my
>  > experience).
>  > A versioning system would be extremely helpful (SVN, or CVS. would
>  > come to my mind).
>  > Well, if you are ok with having the source tree hosted in a
>  > SVN/CVS/alike I am on
>  > (opening an account on sourceforge or savannah -for example- would be
>  > the next step then, as it can take few days for a project to be
>  > approved)
>
> I don't have any experience with SVN, but it could be an opportunity to learn
> something new.  I have a couple of other projects that could benefit by moving
> them to SVN or CVS like system.

I have seen A.M.Kuching's suggestion about python's sandbox,
but I do not know if it would work in the short term. Anyway,
and at least for the long(er) term, it will make sense to favor SVN
over CVS (since this is what the python project is using). That rules
out savannah and leaves us with sourceforge.

> What I intended was to get enough feed back that I could get it to a point where
> it's 90% done and then upload it as a patch where it could be further polished
> up.  To do that, I first need to verify my design goals are the correct
> approach. (see my reply to Ka-Ping.)  If they are, then it's more a matter of
> cleaning up what I've already started.  If not, then it's a bit (or bunch) more
> work and would benefit from having others work on it in a more formal way.

No matter the number of developpers, it would give people interested in
following the developpements a convenient way to do it.

> If someone who has more experience with group projects would like to manage it,
> that would be good too.  That may speed things up considerably.

I have some experience in it (in companies, and in an open source project)
I can always file a application for a sourceforge project,
and can help you with managing it until you feel like taking it on your own
(or it is merged with the python trunk)
I am willing to contribute to the implementation (I suspect that
things like unit tests be needed).


>
>  > I will have a look then.
>
> Great.  :-)
>

I should have the time during the week-end. I will get back to you off-list.


Cheers,



L.


> Cheers,
>    Ron
>

From pje at telecommunity.com  Sat Jan  6 03:41:02 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 05 Jan 2007 21:41:02 -0500
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <enmucu$nb9$1@sea.gmane.org>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
	<459EC6B8.7000600@ronadam.com>
Message-ID: <5.1.1.6.0.20070105213901.029fd058@sparrow.telecommunity.com>

At 06:35 PM 1/5/2007 -0700, Fernando Perez wrote:
>Ron Adam wrote:
>
> > Laurent Gautier wrote:
>
> >  > From the top of my head, there might be  "ipython" (the excellent
> >  > interactive console) is possibly using pydoc
> >  > (in any case, I would say that the authors would be interested in
> >  > developments with pydoc)
>
>Certainly :)  I'd like to ask whether this discussion considers any kind of
>merging of pydoc with epydoc.

Unless there's been a complete rewrite of epydoc since the last time I 
looked at it, I'd have to give a very strong -1 against epydoc; it has all 
the problems of pydoc, plus new ones.


From barry at python.org  Sat Jan  6 06:16:51 2007
From: barry at python.org (Barry Warsaw)
Date: Sat, 6 Jan 2007 00:16:51 -0500
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <5.1.1.6.0.20070105213901.029fd058@sparrow.telecommunity.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
	<459EC6B8.7000600@ronadam.com>
	<5.1.1.6.0.20070105213901.029fd058@sparrow.telecommunity.com>
Message-ID: <BEC51643-4623-4630-B976-7FF78DAC5D1F@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 5, 2007, at 9:41 PM, Phillip J. Eby wrote:

> Unless there's been a complete rewrite of epydoc since the last time I
> looked at it, I'd have to give a very strong -1 against epydoc; it  
> has all
> the problems of pydoc, plus new ones.

I haven't read this entire thread, so I'll just chime in to say that  
I've /used/ epydoc and like it quite a bit.  I've even hacked on it a  
little to fix a few things and it didn't seem that bad, though I  
didn't do any major work on its internals.  I've used both the 2.x  
version and the 3.x version but I haven't used anything in the last,  
I dunno, 4 or 5 months.

Note that I was using it on a heavily embedded/extended application  
and it did a pretty good job of pulling docs out of C coded  
docstrings.  I had to patch Python a bit here and there (I think most  
of those fixes are in Python 2.5) and I know that the epydoc guys  
fixed a few things related to C types (e.g. such as that the tp_doc  
has to document both the type and the constructor).  Probably the  
biggest issue that I remember was needing to invoke it  
programmatically, which was an absolute requirement for us, since  
none of the extension modules were importable unless epydoc was run  
inside the embedded environment.  I got it to work, but it was a bit  
of a pain.

If you've already explained it, that's fine, but if not, could you  
outline what you have against epydoc?

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)

iQCVAwUBRZ8ww3EjvBPtnXfVAQJmnwQAnTn7W7Nri5Q+pSPmTLaIvqnmRJWDegpF
HSIDY8nBcMwsST76gzpwt02GikWtyy4gujgHiAEyr4/eQyJsMcnXMptkgXixtuPz
wA2pJkeo87eorPBMtOMoB9XpoyUkQTh5W/lGnR3rOinZPeiqJFEzc//DIJV+H/p7
Iqrie+FVnis=
=sNBX
-----END PGP SIGNATURE-----

From pje at telecommunity.com  Sat Jan  6 09:09:13 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 06 Jan 2007 03:09:13 -0500
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <BEC51643-4623-4630-B976-7FF78DAC5D1F@python.org>
References: <5.1.1.6.0.20070105213901.029fd058@sparrow.telecommunity.com>
	<27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
	<459EC6B8.7000600@ronadam.com>
	<5.1.1.6.0.20070105213901.029fd058@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20070106030117.028e3e20@sparrow.telecommunity.com>

At 12:16 AM 1/6/2007 -0500, Barry Warsaw wrote:
>If you've already explained it, that's fine, but if not, could you
>outline what you have against epydoc?

The last I tried to work with it, it had even more hardcoded typechecking 
than pydoc does, spread out over more of the code base.  Also, over at 
OSAF, I've been repeatedly called upon to sort out some peculiarity in how 
it discovers things or how it handles types it doesn't recognize.

My net impression has been that it's very brittle, even more so than pydoc.

On the plus side, there are some very good ideas and a *lot* of good 
execution in there, but its extensibility has historically struck me as 
non-existent.

To be fair, the last time I had to deal with any problems with it at OSAF 
was almost a year ago, if memory serves.  I don't know if anything has 
improved in it since then.

The last time I seriously analyzed its internal architecture was several 
years ago (maybe 5?) when I was investigating it as an alternative to 
HappyDoc for doing PEAK's API documentation.  I could never get it to work 
on anything but a small subset of PEAK without crashing in any of several 
ways, including segfaulting its GUI!  It had built into it a variety of 
restrictive assumptions about how programs are structured that were not 
compatible with what I was doing.  pydoc at least only crashed when dealing 
with metaclass instances, but I believe that was fixed in 2.3 or a late 
2.2.x release.

Anyway, I like the *idea* of epydoc and a lot of its execution, but IMO it 
needs just as much work as pydoc, if not more.


From robert.kern at gmail.com  Sat Jan  6 09:41:04 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 06 Jan 2007 02:41:04 -0600
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <5.1.1.6.0.20070106030117.028e3e20@sparrow.telecommunity.com>
References: <5.1.1.6.0.20070105213901.029fd058@sparrow.telecommunity.com>	<27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>	<459D27FE.6070406@ronadam.com>	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>	<459EC6B8.7000600@ronadam.com>	<5.1.1.6.0.20070105213901.029fd058@sparrow.telecommunity.com>
	<BEC51643-4623-4630-B976-7FF78DAC5D1F@python.org>
	<5.1.1.6.0.20070106030117.028e3e20@sparrow.telecommunity.com>
Message-ID: <ennnb4$620$1@sea.gmane.org>

Phillip J. Eby wrote:
> At 12:16 AM 1/6/2007 -0500, Barry Warsaw wrote:
>> If you've already explained it, that's fine, but if not, could you
>> outline what you have against epydoc?
> 
> The last I tried to work with it, it had even more hardcoded typechecking 
> than pydoc does, spread out over more of the code base.  Also, over at 
> OSAF, I've been repeatedly called upon to sort out some peculiarity in how 
> it discovers things or how it handles types it doesn't recognize.
> 
> My net impression has been that it's very brittle, even more so than pydoc.
> 
> On the plus side, there are some very good ideas and a *lot* of good 
> execution in there, but its extensibility has historically struck me as 
> non-existent.
> 
> To be fair, the last time I had to deal with any problems with it at OSAF 
> was almost a year ago, if memory serves.  I don't know if anything has 
> improved in it since then.

FWIW, a 3.0a3 was released in August 2006, and according to the History,
"Significant portions of epydoc were written for version 3.0." It seems a lot of
that was to add parsing as a complementary means to extract documentation. I'm
not particularly familiar with the introspection code of either 2.1 or 3.0a3,
but a cursory examination shows that 3.0a3 has an introspecter registry that 2.1
doesn't:

# In epydoc.docintrospecter:

def register_introspecter(applicability_test, introspecter, priority=10):
    """
    Register an introspecter function.  Introspecter functions take
    two arguments, a python value and a C{ValueDoc} object, and should
    add information about the given value to the the C{ValueDoc}.
    Usually, the first line of an inspecter function will specialize
    it to a sublass of C{ValueDoc}, using L{ValueDoc.specialize_to()}:

        >>> def typical_introspecter(value, value_doc):
        ...     value_doc.specialize_to(SomeSubclassOfValueDoc)
        ...     <add info to value_doc>

    @param priority: The priority of this introspecter, which determines
    the order in which introspecters are tried -- introspecters with lower
    numbers are tried first.  The standard introspecters have priorities
    ranging from 20 to 30.  The default priority (10) will place new
    introspecters before standard introspecters.
    """

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robey at lag.net  Sat Jan  6 10:00:04 2007
From: robey at lag.net (Robey Pointer)
Date: Sat, 6 Jan 2007 01:00:04 -0800
Subject: [Python-Dev] features i'd like [Python 3000] ... #3: fix super()
In-Reply-To: <457402E1.6060705@gmail.com>
References: <4573909F.5050405@666.com> <457402E1.6060705@gmail.com>
Message-ID: <A353062D-A55D-4EF3-A6D0-B3DE1E7FB346@lag.net>


On 4 Dec 2006, at 3:13, Nick Coghlan wrote:

> Ben Wing wrote:
>> i don't like the current super() at all.  having to type super(Foo,
>> self).__init__(...) is messy, hard to remember, and error-prone.
>
> Yup.
>
>>  it
>> also introduces an unfortunate dependency in that the name of the  
>> class
>> (Foo) has to be hard-coded in the call, and if you change the  
>> class name
>> you also have to go and change all super() calls -- more chances for
>> errors.  as a result, i imagine there's a strong urge to just  
>> hardcode
>> the name of the parent -- a bad idea, and the reason why super() was
>> introduced in the first place.
>
> Also yup.
>
> The fact that these drawbacks are fairly obvious, yet nothing has  
> been done
> about them should suggest something to you about the difficulty of  
> actually
> solving this problem in a reliable fashion ;)
>
> The Python Cookbook has a few interesting approaches to making  
> cooperative
> super calls easier to write, but they all tend to have some kind of  
> drawback
> that makes them inappropriate for making part of the language core.
>
>> instead, `super' should work like in other languages.  a couple of  
>> ideas:
>>
>> -- super.meth(args) calls the superclass method `meth', in the  
>> same way 
>> that super(Foo, self).meth(args) currently works. (this is the same
>> syntax as in java.  this probably requires making `super' be a  
>> keyword,
>> although it might be possible to hack this by examining the  
>> current call
>> stack.)
>
> It's the implementation that's the real sticking point here, so  
> without code
> this idea isn't likely to go anywhere.

I was bitten by the urge to play with this today, and modified my  
previous "self" hack to handle "super" also, so that the following  
code works:

     class D (C):
         @method
         def sum(n):
             return super.sum(n * 2) - self.base

Posted as "evil2.py" here:

     http://www.lag.net/robey/code/surf/

Because hacking "super" requires having the class object handy, this  
one needs a metaclass to do its magic, which is a shame.  I guess if  
it was implemented inside the cpython compiler, it would be less of a  
problem.

robey


From eopadoan at altavix.com  Sat Jan  6 18:16:19 2007
From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan)
Date: Sat, 6 Jan 2007 15:16:19 -0200
Subject: [Python-Dev] features i'd like [Python 3000] ... #3: fix super()
In-Reply-To: <A353062D-A55D-4EF3-A6D0-B3DE1E7FB346@lag.net>
References: <4573909F.5050405@666.com> <457402E1.6060705@gmail.com>
	<A353062D-A55D-4EF3-A6D0-B3DE1E7FB346@lag.net>
Message-ID: <dea92f560701060916r1dc04e96x86e13020397d4922@mail.gmail.com>

> I was bitten by the urge to play with this today, and modified my
> previous "self" hack to handle "super" also, so that the following
> code works:
>
>      class D (C):
>          @method
>          def sum(n):
>              return super.sum(n * 2) - self.base
>
> Posted as "evil2.py" here:
>
>      http://www.lag.net/robey/code/surf/
>
> Because hacking "super" requires having the class object handy, this
> one needs a metaclass to do its magic, which is a shame.  I guess if
> it was implemented inside the cpython compiler, it would be less of a
> problem.

BTW, a "super-only" version of this decortor (that I think could be
called "implement") has  some more chances in Python. But think
belongs more to Python-Ideas list, ok?

-- 
EduardoOPadoan (eopadoan->altavix::com)
Bookmarks: http://del.icio.us/edcrypt
Blog: http://edcrypt.blogspot.com
Jabber: edcrypt at jabber dot org
ICQ: 161480283
GTalk: eduardo dot padoan at gmail dot com
MSN: eopadoan at altavix dot com

From rrr at ronadam.com  Sat Jan  6 23:15:30 2007
From: rrr at ronadam.com (Ron Adam)
Date: Sat, 06 Jan 2007 16:15:30 -0600
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <27d1e6020701051821r749b8f4dya5b28107b8f410ba@mail.gmail.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>	
	<459D27FE.6070406@ronadam.com>	
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>	
	<459EC6B8.7000600@ronadam.com>
	<27d1e6020701051821r749b8f4dya5b28107b8f410ba@mail.gmail.com>
Message-ID: <45A01F82.9070802@ronadam.com>

Laurent Gautier wrote:
> 2007/1/6, Ron Adam <rrr at ronadam.com>:
>> Laurent Gautier wrote:

> [...]

> I read your comment about having not too many things changed for 2.6.
> (or that will be bumped to 3000).
> 
> A suggestion I would have would be to create an html/htmlrender module
> in the pydoc-package-to-be and start putting all the html formatting 
> function
> (as they are completely independent of pydoc, as far as I can see, over 
> there).

There really isn't as many html specific formatting functions as you might think 
since I used a very consistent format of css tagged definition lists.

In the case of cgitb, it's probably better to copy those functions to it and 
just get rid of the dependency.  If someone does put together a library of 
useful html functions and class's then pydoc, cgitb and other programs can use it.

BTW, all the html specific functions and formatting have already been collected 
in the gethtml.py file.  I've also added more comments to both gettext.py and 
gethtml.py last night, so it should be easier to see how it works.  First look 
though gettext.py to get a general idea of how the info is collected and 
formatted in text, then look at gethtml.py.

      http://ronadam.com/dl/_pydoc.zip


> You can then create wrappers to the original functions including a 
> deprecation
> warning. You can refer to Michael Chermside's recipe for an example of
> implementation with a deprecation decorator -
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/391367
> )
> The suggestion above would actually apply to *anything* that is changed
> in pydoc, providing the benefit of allowing the necessary changes while 
> having
> a temporary API to provide back-compatibility.

Yes, that would be good and may be needed, but it's still a ways off.  Lets get 
it to work first, then if it gets approved for inclusion, then how to move it 
into the python distribution (with any needed changes needed for that) can be 
worked out.  First things first, if you know what I mean.


> [...]

>>  > I do not think it will work as a zip file shuttled around (in my
>>  > experience).
>>  > A versioning system would be extremely helpful (SVN, or CVS. would
>>  > come to my mind).
>>  > Well, if you are ok with having the source tree hosted in a
>>  > SVN/CVS/alike I am on
>>  > (opening an account on sourceforge or savannah -for example- would be
>>  > the next step then, as it can take few days for a project to be
>>  > approved)
>>
>> I don't have any experience with SVN, but it could be an opportunity 
>> to learn
>> something new.  I have a couple of other projects that could benefit 
>> by moving
>> them to SVN or CVS like system.
> 
> I have seen A.M.Kuching's suggestion about python's sandbox,
> but I do not know if it would work in the short term. Anyway,
> and at least for the long(er) term, it will make sense to favor SVN
> over CVS (since this is what the python project is using). That rules
> out savannah and leaves us with sourceforge.

I'd like to know more about using the sandbox, I know it would be easy for 
people to read the source there, but who all can have write access to it without 
having write access to other python areas?  I would not mind giving that a try 
if someone who already knows how could point me to the correct how-to 
documentation with some advice on what not to do.

I'm actually more concerned about the what not to do stuff.  I really would not 
like to clobber someone else's work or create problems because of my 
inexperience with CVS.


>[...]

>> If someone who has more experience with group projects would like to 
>> manage it,
>> that would be good too.  That may speed things up considerably.
> 
> I have some experience in it (in companies, and in an open source project)
> I can always file a application for a sourceforge project,
> and can help you with managing it until you feel like taking it on your own
> (or it is merged with the python trunk)

I don't see this as taking a long time if we keep it to cleaning up with some 
API and user interface improvements.

I know there are some here who want a smart parsing engine, which probably would 
take a long term commitment to maintain and fix bugs, etc.  But lets look at the 
actual use's that pydoc serves first.

Use's for pydoc in order of importance and frequency of use:

    1. Console (builtin) help.  ie.. the help() function.
    2. HTML browsing and quick reference.
    3. Document generation in text.
    4. Document generation in html.
    5. Document generation in other formats.  (not currently possible)

I'm concentrating on 1 and 2.  Use cases 3 and 4 are just an easy to do 
byproduct of doing 1 and 2.  I think the cleaning up may make doing 5 possible.

Lets turn the question around.  How well would other document generators supply 
pydoc with the equivalent text of the help() function, interactive help session 
output, and the equivalent html needed for dynamic html browsing?

Also keep in mind the help function is always by default imported into python, 
so keeping that small and relatively simple with a minimum of external 
dependencies is good.


 > I am willing to contribute to the implementation (I suspect that
 > things like unit tests be needed).

Yes, there will need to be some unit tests.  It may even help for those be 
written now.  That would help us identify things that still need to be done.


[...]

> I should have the time during the week-end. I will get back to you 
> off-list.

Cool. :-)

Cheers,
    Ron




From tzot at mediconsa.com  Sun Jan  7 11:38:04 2007
From: tzot at mediconsa.com (Christos Georgiou)
Date: Sun, 7 Jan 2007 12:38:04 +0200
Subject: [Python-Dev] Bug or not? Different behaviour iterating list and
	collections.deque
Message-ID: <enqiif$ko4$1@sea.gmane.org>

Hello, people. I am not sure whether this is a bug or intentional, so I 
thought checking it with you before opening a bug. I will explain this 
issue, but please understand this is not a question for help to change the 
algorithm (this has been done already), so it's not a question of c.l.py. 
It's a matter of discrepancy.

A list that changes while iterating produces no errors, while a deque fails.

Given the following example code:

#code start
import itertools, collections

def item_is_special(item):
 "Just a dummy check in this example"
 return item % 3 == 0

def item_products(item):
 "Also a dummy function for the example"
 return [item*20+1, item*30+1]

def process_list(items, type_of_que, special_case):
 # we want to process some items, but some of them
 # produce other items to be processed by the
 # same algorithm
 products= type_of_que()
 if special_case: products.append(-1)
 for item in itertools.chain(items, products):
  if item_is_special(item):
   for product in item_products(item):
    products.append(product)
  else:
   print "got", item
#code end

we have the following cases:

>>> process_list(numbers, list, False)
got 1
got 2
got 61
got 91

List works as expected.

>>> process_list(numbers, collections.deque, False)
got 1
got 2

deque does not work, most probably because deque.__iter__ of an empty deque 
ignores later changes. For this reason the `special_case' parameter was 
inserted in the code above, so that there is at least an item when 
itertools.chain calls iter on the deque:

>>> process_list(numbers, collections.deque, True)
got 1
got 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "testdequeiter.py", line 17, in process_list
    for item in itertools.chain(items, products):
RuntimeError: deque mutated during iteration

Is that intentional? If not, let me know so that I open a bug. 



From tzot at mediconsa.com  Sun Jan  7 11:50:40 2007
From: tzot at mediconsa.com (Christos Georgiou)
Date: Sun, 7 Jan 2007 12:50:40 +0200
Subject: [Python-Dev] Bug or not? Different behaviour iterating list
	andcollections.deque
References: <enqiif$ko4$1@sea.gmane.org>
Message-ID: <enql7l$io5$1@sea.gmane.org>

Forgive my piggy backing, but I forgot to include the only related post I 
found, which did not clear things up for me:

http://groups.google.com/group/comp.lang.python/msg/e2dcb2362649a601




From jcarlson at uci.edu  Sun Jan  7 18:58:17 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 07 Jan 2007 09:58:17 -0800
Subject: [Python-Dev] Bug or not? Different behaviour iterating list and
	collections.deque
In-Reply-To: <enqiif$ko4$1@sea.gmane.org>
References: <enqiif$ko4$1@sea.gmane.org>
Message-ID: <20070107095022.8D37.JCARLSON@uci.edu>


"Christos Georgiou" <tzot at mediconsa.com> wrote:
> 
> Hello, people. I am not sure whether this is a bug or intentional, so I 
> thought checking it with you before opening a bug. I will explain this 
> issue, but please understand this is not a question for help to change the 
> algorithm (this has been done already), so it's not a question of c.l.py. 
> It's a matter of discrepancy.
[snip]
> Forgive my piggy backing, but I forgot to include the only related post I 
> found, which did not clear things up for me:
> 
> http://groups.google.com/group/comp.lang.python/msg/e2dcb2362649a601


Lists are not deques, deques are not lists.  As stated in the post you
link to states, the behavior of lists and deques is different in the
same situation, so while one may "work" for your task, the other may not.

I would, generally speaking, concur with Rayomond's recommendation in
the post and say that mutation of the item you are iterating over is
confusing, and it certainly is likely far more tricky than you should be
when writing software so that people can use it.

In this case, you are making it even worse by attempting to merge a list
and deque, under the presumption that they should do the same thing. 
They don't.  They are documented not to do the same thing.  And you read
a post explaining that they don't do the same thing and cited it.


My suggestion: don't do what you are doing.  Input list, output list. 
Make your life simple.  Appending to a list is fast.


If you *need* to merge a list and deque, do to some insane requirements
by an organization, write your own wrapper that does what you think is
the right thing to do in this case.

 - Josiah


From tzot at mediconsa.com  Sun Jan  7 19:23:44 2007
From: tzot at mediconsa.com (Christos Georgiou)
Date: Sun, 7 Jan 2007 20:23:44 +0200
Subject: [Python-Dev] Bug or not? Different behaviour iterating list
	andcollections.deque
References: <enqiif$ko4$1@sea.gmane.org> <20070107095022.8D37.JCARLSON@uci.edu>
Message-ID: <enrdrh$m1e$1@sea.gmane.org>


"Josiah Carlson" <jcarlson at uci.edu> wrote in message 
news:20070107095022.8D37.JCARLSON at uci.edu...
>
> "Christos Georgiou" <tzot at mediconsa.com> wrote:

>> [snip]
>> issue, but please understand this is not a question for help to change 
>> the
>> algorithm (this has been done already), so it's not a question of c.l.py.
>> It's a matter of discrepancy.
> [snip]
>
> [snip Josiah's wordy "it's intentional and not a bug" in the form of a 
> suggestion
> for a change of algorithm]

Like I said, this wasn't a c.l.py question, even if you thought it deserved 
a c.l.py answer. In any case, you answered my question, and thank you. It 
not being a bug suits me just fine.

Allow me to make sure we talk about the same thing here, though: both the 
example code I provided and the original one do modify the iterable *only* 
between the following A and B points in time:

Point A: itertools.chain calls iter() on the iterable.

(Appending to the iterable (list, deque) occur here, and only here.)

Point B: itertools.chain starts calling iterable.next().

This is a different case from the one mentioned in the post by Raymond, and 
that is why I asked. For example, if itertools.chain called iter() on its 
arguments when actually needing to iterate over them instead of at the 
beginning, the code would work. But I really, really don't mind whatever the 
function, as long as it's by design, and that's the reason I didn't submit a 
bug in the tracker. That's all. I won't (and never intended) to defend any 
algorithms. 



From dustin at v.igoro.us  Sun Jan  7 21:45:20 2007
From: dustin at v.igoro.us (dustin at v.igoro.us)
Date: Sun, 7 Jan 2007 14:45:20 -0600
Subject: [Python-Dev] smarter temporary file object (SF #415692,
	#1630118)
In-Reply-To: <ee2a432c0701022207s7b361842jdc6acd11f8bef4a0@mail.gmail.com>
References: <20070103052739.GF19957@v.igoro.us>
	<ee2a432c0701022207s7b361842jdc6acd11f8bef4a0@mail.gmail.com>
Message-ID: <20070107204520.GB5986@v.igoro.us>

On Tue, Jan 02, 2007 at 10:07:58PM -0800, Neal Norwitz wrote:
> Thanks for your patch!

With some advice from Jim Jewett, the addition of some test cases, and a
paragraph of documentation, I've uploaded the corresponding patch at
  http://python.org/sf/1630118
if there are any other modifications or improvements anyone has in mind,
please let me know.

Dustin

From martin at v.loewis.de  Mon Jan  8 01:37:23 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 08 Jan 2007 01:37:23 +0100
Subject: [Python-Dev] Bug or not? Different behaviour iterating list and
 collections.deque
In-Reply-To: <enqiif$ko4$1@sea.gmane.org>
References: <enqiif$ko4$1@sea.gmane.org>
Message-ID: <45A19243.2000109@v.loewis.de>

Christos Georgiou schrieb:
> Is that intentional?

It would have helped if you had said what "that" is you are referring
to, it would also have helped if you had stated an opinion on whether
you believe that to be a bug. For example, I think I would have phrased
your post like this:

"""
If I apply .next() to an iter of a deque that has been modified, I
get a RuntimeError:

py> d=collections.deque()
py> d.append(10)
py> i=iter(d)
py> d.append(10)
py> i.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
RuntimeError: deque mutated during iteration

Yet when I apply .next() to an iter of an initially-empty deque,
I get StopIteration:

py> d=collections.deque()
py> i=iter(d)
py> d.append(10)
py> i.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
StopIteration

Is this a bug? Shouldn't the second example also raise the RuntimeError
as the deque has been modified?
(also, is appending an element a modification or a mutation?)
"""

To this question (.next on an iterator of a modified deque), my answer
would be: "yes, that is a bug".

However, I feel you are referring to a different issue, unfortunately,
I cannot tell from your post what that issue is.

Regards,
Martin

From rrr at ronadam.com  Mon Jan  8 06:07:48 2007
From: rrr at ronadam.com (Ron Adam)
Date: Sun, 07 Jan 2007 23:07:48 -0600
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <enliu3$kg6$1@sea.gmane.org>
References: <458C719F.3000904@ronadam.com> <enliu3$kg6$1@sea.gmane.org>
Message-ID: <45A1D1A4.9070808@ronadam.com>

Neal Becker wrote:
> No time to review this now, but I'd just like to say that the 1 thing I'd
> like to see is support for decent mathematical markup.  I think at this
> point that support for latex markup is the way to achieve this.

There are two separate issues related to this I'd like to point out because some 
of the other suggestions have indicated both of these without spelling out which 
they are addressing.


(1.) Processing existing text markup from additional text hints that are 
inserted into doc strings.

I think this can easily be handled with a single text output point.  General 
*post* formatters of this type are very doable I think.  We just need to 
document the function or method to get the *raw* plain text for that purpose.


(2.) Parsing and inserting additional markup where there is none, based on what 
and where the information came from.

This is the more difficult problem.  I've tried to handle this case by creating 
an object for each "thing", that can be extended by adding a formatting method 
to it.  This type of markup can be very specific and may depend on context as 
well as what or where the source data came from.


I don't know latex markup, but it seems like mathematical latex markup might be 
done either way.

Cheers,
    Ron


From ironfroggy at gmail.com  Mon Jan  8 09:30:42 2007
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Mon, 8 Jan 2007 03:30:42 -0500
Subject: [Python-Dev] DRAFT: pydev summary for 2006-12-01 to 2006-12-15
Message-ID: <76fd5acf0701080030t1cfc6d32n922efa64a29e4e43@mail.gmail.com>

=============
Announcements
=============

Some of you may know that Steven Bethard, having taken over the summaries from
Brett Cannon some time ago, is no longer able to keep up with them. After much
holiday stress and busy days, I've come to take over this honor and archive all
the goings on here. I hope I can do a good job, and allow the summaries to
continue being a useful and transparent fixture.

Also, after catching up with my backlog of summary work, I plan to begin the
first summaries of python-3000 (and, possibly, but not likely, the python-
ideas list), as more and more actual work, rather than debating and theory-talk,
is going on there.

Sincerly,
Your new summary writer,
Calvin Spealman

=========
Summaries
=========

-----------------------------------------------------------------------------
[NPERS] Re: a feature i'd like to see in python #2: indexing of match objects
-----------------------------------------------------------------------------

Subscripting re match objects directly was proposed and the debate the waged
mostly on how to deal with slicing and m.group(0) to be the entire match. The
consensus seemed to be that returning a match object would break, largely
because m[1:2][0] != m[1], which breaks intuition with slices. Returning a list
as a slice also breaks the intuitions about the type of a slice.

Contributing thread:

- `a feature i'd like to see in python #2: indexing of match objects
<http://mail.python.org/pipermail/python-dev/2006-December/070138.html>`__

----------------------------------------
Python and the Linux Standard Base (LSB)
----------------------------------------

Discussions about how to properly document the expected Python environment for
all LSB compliant distributions continued shortly. Focus was on how distros have
a need to split the standard library up, even if the entire thing is eventually
installed, and issues such as Tkinter requiring X leading to standard modules
being ommitted for flexable packages.

Contributing thread:

- `Python and the Linux Standard Base (LSB)
<http://mail.python.org/pipermail/python-dev/2006-December/070112.html>`__

----------------------------------------------------------------------
[Python-checkins] MSI being downloaded 10x more than all other files?!
----------------------------------------------------------------------

Guido was impressed with out download numbers on Windows, but most people just
attributed it to other platforms including and mirroring Python. Even taking
that to consideration, the ratio of Windows to source downloads has shifted in
Windows favor a good bit. The use of Python in introductory courses at MIT was
mentioned as something to push these numbers further in the near future.

Contributing thread:

- `[Python-checkins] MSI being downloaded 10x more than all other
files?! <http://mail.python.org/pipermail/python-dev/2006-December/070294.html>`__

----------------------------------------------------------------
a feature i'd like to see in python #1: better iteration control
----------------------------------------------------------------

The request was made to allow direct deletion of items from iterables through
the iterator, but the idea was deemed unscalable to iterables in general and
broken with dictionaries. The solutions to work around the problem were given
and generally agreed to be sufficient (such as copying all the items you want
to keep, instead).

Contributing thread:

- `a feature i'd like to see in python #1: better iteration control
<http://mail.python.org/pipermail/python-dev/2006-December/070137.html>`__

-------------------------
LSB: Binary compatibility
-------------------------

The other side of the LSB issue was discussed, related to the CPython ABI, the
.pyc bytecode and the marshall formats, and exactly which versions of Python
may or may not gain a standard ABI, as well as how to carry this forward to
future (already existing and evolving) version of Python. Some mention was made
of the difficulties and conflicts being brought on by the idea of LSB standards
for Python, while the benefits of this were challenged.

Contributing thread:

- `LSB: Binary compatibility
<http://mail.python.org/pipermail/python-dev/2006-December/070191.html>`__

----------------------------------------
Adding resume (206) support to urllib(2)
----------------------------------------

Daniel Watkins asked about adapting his HTTP resuming support into urllib, but
suggestions directly moved this to urllib2 and then to httplib. The conversation
quickly drifted into talking about how abstract and minimal the urllib(2) should
be, while the httplib module may grow increasing complex and feature supporting.
The original poster made no final comment on his intentions to follow through
with the patch.

Contributing thread:

- `Adding resume (206) support to urllib(2)
<http://mail.python.org/pipermail/python-dev/2006-December/070320.html>`__

-------------------
__str__ and unicode
-------------------

The details of what an object's __str__() method must return were inquired. Must
the string returned by a str or any basestring? It was announced to allow
unicode objects, with no restriction on content. However, the caution that the
return is likely to be encoded into the default encoding was given.

Contributing thread:

- `__str__ and unicode
<http://mail.python.org/pipermail/python-dev/2006-December/070237.html>`__

----------------------------------
LSB: Selection of a Python version
----------------------------------

Martin v. Lowis announced that LSB will likely move forward with Python 2.4 as
the first LSB standard Python version. How to handle future bug patches, as well
as the comfort of the distributions for being required to support an outdated
package (2.4 is already "retired") was considered. The consensus settled on a
documented support for 2.4 moving forward with security patches, and likely
even 2.3, as security patches are rare enough to not cause too many cycles to
be spent on them.

Contributing thread:

- `LSB: Selection of a Python version
<http://mail.python.org/pipermail/python-dev/2006-December/070189.html>`__

------------------------------
Threading, atexit, and logging
------------------------------

A bug (#1566280) was previously reported, related to the ordering of the atexit
handlers for the logging and threading modules. Participants in the thread had
a general agreement that the threading module using such a user visiable
mechanism for cleanup and lifetime was a bad design in the first place to be
remedied.

Contributing thread:

- `Threading, atexit, and logging
<http://mail.python.org/pipermail/python-dev/2006-December/070233.html>`__

-----------------------------------
Creating dicts from dict subclasses
-----------------------------------

A common problem of subclassing built-in types (dict, in this case) arose trying
to move weakref.WeakValueDictionary to a subclass of dict, rather than UserDict.

[NOTE: Could someone give a better description of this? I don't know enough of
the issue here to write any content full enough. Thanks.]

Contributing thread:

- `Creating dicts from dict subclasses
<http://mail.python.org/pipermail/python-dev/2006-December/070334.html>`__

-----------------------------------------------------------------------------
OT: World's oldest ritual discovered. Worshipped the python 70, 000 years ago
-----------------------------------------------------------------------------

This was a quick and off-topic post about other instances of python symbolism in
ancient humanity, including a definition of our word "pythonic" to mean
"prophetic", which seems fitting enough.

Contributing thread:

- `OT: World's oldest ritual discovered. Worshipped the python 70, 000
years ago <http://mail.python.org/pipermail/python-dev/2006-December/070123.html>`__

------------------
LSB: pyc stability
------------------

The LSB meetings brought up the issue of standardized bytecode formats, again.
This has always been something discouraged from Python. Keeping the bytecode
format free and the implementation open to ignoring old versions, has always
been helpful for maintanence and development. The common reasoning of protecting
IP was used, although most in-the-know consider this to be a false sense of
security. Reguardless, it might be useful in legal ways, as trade secrets do
require a reasonable attempt at hidding them in order to be recognized as such.

The possible solution of importing encypted zip packages with a custom import
hook was proposed and seems logical.

Contributing thread:

- `LSB: pyc stability
<http://mail.python.org/pipermail/python-dev/2006-December/070187.html>`__

----------------------------------------------------------------------
features i'd like [Python 3000?] ... #4: interpolated strings ala perl
----------------------------------------------------------------------

Perl/PHP style string interpolation was brought up, but is found unreasonable
for various reasons. Cons include the use of format strings in scopes outside
where they are defined, the dangers of eval, and possibly the general resisting
of any idea coming from Perl. This seems a highly unlikely proposal.

Contributing thread:

- `features i'd like [Python 3000?] ... #4: interpolated strings ala
perl <http://mail.python.org/pipermail/python-dev/2006-December/070173.html>`__

---------------------------
Polling with Pending Calls?
---------------------------

Issues with the RPM library blocking interupts for long periods of time were
brought up with proposed fixes by patching the rpmlib bindings to
"automagically" poll for any pending calls. An existing patch was suggested but
did not seem to fit the case at hand.

Contributing thread:

- `Polling with Pending Calls?
<http://mail.python.org/pipermail/python-dev/2006-December/070181.html>`__

===============
Skipped Threads
===============

- `Weekly Python Patch/Bug Summary
<http://mail.python.org/pipermail/python-dev/2006-December/070119.html>`__
- `cross-compile patches
<http://mail.python.org/pipermail/python-dev/2006-December/070216.html>`__
- `Windows SDK <http://mail.python.org/pipermail/python-dev/2006-December/070308.html>`__
- `help for a noob - version for a sharp ARM
<http://mail.python.org/pipermail/python-dev/2006-December/070286.html>`__
- `fileinput module tests
<http://mail.python.org/pipermail/python-dev/2006-December/070307.html>`__
- `Open CPython VM for type information
<http://mail.python.org/pipermail/python-dev/2006-December/070328.html>`__
- `fpectl: does a better implementation make sense?
<http://mail.python.org/pipermail/python-dev/2006-December/070118.html>`__
- `Small tweak to tokenize.py?
<http://mail.python.org/pipermail/python-dev/2006-December/070127.html>`__
- `features i'd like [Python 3000] ... #3: fix super()
<http://mail.python.org/pipermail/python-dev/2006-December/070168.html>`__
- `Makefile.pre.in Patch
<http://mail.python.org/pipermail/python-dev/2006-December/070172.html>`__
- `Python and LSB: Meeting Report
<http://mail.python.org/pipermail/python-dev/2006-December/070186.html>`__
- `Virtual Python (was Re: Python and the Linux Standard Base (LSB))
<http://mail.python.org/pipermail/python-dev/2006-December/070194.html>`__
- `DRAFT: python-dev summary for 2006-11-16 through 2006-11-30
<http://mail.python.org/pipermail/python-dev/2006-December/070229.html>`__
- `Looking for new Python-Dev Summarizers
<http://mail.python.org/pipermail/python-dev/2006-December/070232.html>`__
- `Adding C99 bool to struct module
<http://mail.python.org/pipermail/python-dev/2006-December/070276.html>`__
- `Spam and semi-spam and neo-spam
<http://mail.python.org/pipermail/python-dev/2006-December/070284.html>`__
- `[NPERS] Re: a feature i'd like to see in python #1: betteriteration
control <http://mail.python.org/pipermail/python-dev/2006-December/070171.html>`__
- `Next PyPy sprint: Leysin 8-14th January 2007
<http://mail.python.org/pipermail/python-dev/2006-December/070217.html>`__
- `Distribution tools: What I would like to see
<http://mail.python.org/pipermail/python-dev/2006-December/070234.html>`__
- `RealClearPolitics - Articles - Exit Rumsfeld, Smiling
<http://mail.python.org/pipermail/python-dev/2006-December/070283.html>`__

From tzot at mediconsa.com  Mon Jan  8 10:29:48 2007
From: tzot at mediconsa.com (Christos Georgiou)
Date: Mon, 8 Jan 2007 11:29:48 +0200
Subject: [Python-Dev] Bug or not? Different behaviour iterating list and
	collections.deque
References: <enqiif$ko4$1@sea.gmane.org> <45A19243.2000109@v.loewis.de>
Message-ID: <ent2uf$5je$1@sea.gmane.org>

""Martin v. L?wis"" <martin at v.loewis.de> wrote in message 
news:45A19243.2000109 at v.loewis.de...
> Christos Georgiou schrieb:
>> Is that intentional?
>
> It would have helped if you had said what "that" is you are referring
> to, it would also have helped if you had stated an opinion on whether
> you believe that to be a bug. For example, I think I would have phrased
> your post like this:

Yes, you are correct: I see now that I was too cryptic, but being cryptic 
was not my intention; instead, I wanted to avoid a discussion on the issue 
of the algorithm, which I didn't manage to avoid, so obviously my terseness 
was mistaken. I should be clear from the start.

In retrospection, the example code I chose, although it showed the two 
issues I thought important ('list / deque iteration discrepancy' and 'empty 
/ non-empty deque iteration discrepancy') was not as plain as needed, since 
it gave to Josiah, at least, the impression that the matter was about list / 
deque merging.

> """
> If I apply .next() to an iter of a deque that has been modified, I
> get a RuntimeError:
>
> py> d=collections.deque()
> py> d.append(10)
> py> i=iter(d)
> py> d.append(10)
> py> i.next()
> Traceback (most recent call last):
>  File "<stdin>", line 1, in ?
> RuntimeError: deque mutated during iteration
>
> Yet when I apply .next() to an iter of an initially-empty deque,
> I get StopIteration:
>
> py> d=collections.deque()
> py> i=iter(d)
> py> d.append(10)
> py> i.next()
> Traceback (most recent call last):
>  File "<stdin>", line 1, in ?
> StopIteration
>
> Is this a bug? Shouldn't the second example also raise the RuntimeError
> as the deque has been modified?
> (also, is appending an element a modification or a mutation?)
> """
>
> To this question (.next on an iterator of a modified deque), my answer
> would be: "yes, that is a bug".

Yes. This behaviour was meant to be shown with the 'special_case' check. I 
will open a bug for this.

> However, I feel you are referring to a different issue, unfortunately,
> I cannot tell from your post what that issue is.

No, you managed to capture large part of the essence of what I was saying, 
but again, this was not your responsibility, but mine. I should be more 
explicit.

Thank you. 



From lgautier at gmail.com  Mon Jan  8 12:21:30 2007
From: lgautier at gmail.com (Laurent Gautier)
Date: Mon, 8 Jan 2007 19:21:30 +0800
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <45A01F82.9070802@ronadam.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>
	<459D27FE.6070406@ronadam.com>
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>
	<459EC6B8.7000600@ronadam.com>
	<27d1e6020701051821r749b8f4dya5b28107b8f410ba@mail.gmail.com>
	<45A01F82.9070802@ronadam.com>
Message-ID: <27d1e6020701080321h77a4b04cia221cf7d6ca6c853@mail.gmail.com>

2007/1/7, Ron Adam <rrr at ronadam.com>:
> Laurent Gautier wrote:
> > 2007/1/6, Ron Adam <rrr at ronadam.com>:
[...]
> I'd like to know more about using the sandbox, I know it would be easy for
> people to read the source there, but who all can have write access to it without
> having write access to other python areas?  I would not mind giving that a try
> if someone who already knows how could point me to the correct how-to
> documentation with some advice on what not to do.

Limiting where different people can commit code changes is possible... it's
just that I am not certain whether sourceforge allows it or not.
I asked A.M. Kuchling about that.

> I'm actually more concerned about the what not to do stuff.  I really would not
> like to clobber someone else's work or create problems because of my
> inexperience with CVS.

I see that you are under Microsoft windows, so you may want to check
TortoiseSVN.
(The python project is stored on a SVN server, so it would make sense
to favor this  one over CVS - in the case the project administrators
have directory-level control -).

Regarding the possibility of jeopardizing something in the repository,
the directory-level sandbox should only allow you trash your own work
;-)
(but even then, you should always be able to recover from mistakes).

>
> >[...]
>
> >> If someone who has more experience with group projects would like to
> >> manage it,
> >> that would be good too.  That may speed things up considerably.
> >
> > I have some experience in it (in companies, and in an open source project)
> > I can always file a application for a sourceforge project,
> > and can help you with managing it until you feel like taking it on your own
> > (or it is merged with the python trunk)
>
> I don't see this as taking a long time if we keep it to cleaning up with some
> API and user interface improvements.

That's all in the meaning of "some" I guess... ;-)

> I know there are some here who want a smart parsing engine, which probably would
> take a long term commitment to maintain and fix bugs, etc.  But lets look at the
> actual use's that pydoc serves first.
>
> Use's for pydoc in order of importance and frequency of use:
>
>     1. Console (builtin) help.  ie.. the help() function.
>     2. HTML browsing and quick reference.
>     3. Document generation in text.
>     4. Document generation in html.
>     5. Document generation in other formats.  (not currently possible)
>
> I'm concentrating on 1 and 2.  Use cases 3 and 4 are just an easy to do
> byproduct of doing 1 and 2.  I think the cleaning up may make doing 5 possible.

I am fully on that line, with the remark that thinking about point 5
early is that
could make the cut. The exercise will be in avoiding over-complication
in a design that is not used in the end.

Reactions on this thread brought a lot of good ideas and pointers to
existing work.
It loo


> Lets turn the question around.  How well would other document generators supply
> pydoc with the equivalent text of the help() function, interactive help session
> output, and the equivalent html needed for dynamic html browsing?
>
> Also keep in mind the help function is always by default imported into python,
> so keeping that small and relatively simple with a minimum of external
> dependencies is good.
>
>
>  > I am willing to contribute to the implementation (I suspect that
>  > things like unit tests be needed).
>
> Yes, there will need to be some unit tests.  It may even help for those be
> written now.  That would help us identify things that still need to be done.

Regarding tests, it is never early enough to think about them (that
let one write
code that is actually "test-able").

>
> [...]

From rrr at ronadam.com  Mon Jan  8 17:05:45 2007
From: rrr at ronadam.com (Ron Adam)
Date: Mon, 08 Jan 2007 10:05:45 -0600
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <27d1e6020701080321h77a4b04cia221cf7d6ca6c853@mail.gmail.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>	
	<459D27FE.6070406@ronadam.com>	
	<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>	
	<459EC6B8.7000600@ronadam.com>	
	<27d1e6020701051821r749b8f4dya5b28107b8f410ba@mail.gmail.com>	
	<45A01F82.9070802@ronadam.com>
	<27d1e6020701080321h77a4b04cia221cf7d6ca6c853@mail.gmail.com>
Message-ID: <45A26BD9.8080404@ronadam.com>

Laurent Gautier wrote:
> 2007/1/7, Ron Adam <rrr at ronadam.com>:
>> Laurent Gautier wrote:
>> > 2007/1/6, Ron Adam <rrr at ronadam.com>:
> [...]
>> I'd like to know more about using the sandbox, I know it would be easy 
>> for
>> people to read the source there, but who all can have write access to 
>> it without
>> having write access to other python areas?  I would not mind giving 
>> that a try
>> if someone who already knows how could point me to the correct how-to
>> documentation with some advice on what not to do.
> 
> Limiting where different people can commit code changes is possible... it's
> just that I am not certain whether sourceforge allows it or not.
> I asked A.M. Kuchling about that.

I'm not concerned about limiting changes to this project.  I want others to work 
on it.  Can write access be *easily* granted to just one cvs sandbox directory, 
for this project, without granting access to other directories in the sandbox or 
trunk?

>> I'm actually more concerned about the what not to do stuff.  I really 
>> would not
>> like to clobber someone else's work or create problems because of my
>> inexperience with CVS.
> 
> I see that you are under Microsoft windows, so you may want to check
> TortoiseSVN.
> (The python project is stored on a SVN server, so it would make sense
> to favor this  one over CVS - in the case the project administrators
> have directory-level control -).

Thank's I'll try TortoiseSVN out. :-)


> Regarding the possibility of jeopardizing something in the repository,
> the directory-level sandbox should only allow you trash your own work
> ;-)
> (but even then, you should always be able to recover from mistakes).

That would be fine then.  But I'll let you decide since you are offering to 
manage getting it set up.


>> >[...]
>>
>> >> If someone who has more experience with group projects would like to
>> >> manage it,
>> >> that would be good too.  That may speed things up considerably.
>> >
>> > I have some experience in it (in companies, and in an open source 
>> project)
>> > I can always file a application for a sourceforge project,
>> > and can help you with managing it until you feel like taking it on 
>> your own
>> > (or it is merged with the python trunk)
>>
>> I don't see this as taking a long time if we keep it to cleaning up 
>> with some
>> API and user interface improvements.
> 
> That's all in the meaning of "some" I guess... ;-)

Yep.

>> I know there are some here who want a smart parsing engine, which 
>> probably would
>> take a long term commitment to maintain and fix bugs, etc.  But lets 
>> look at the
>> actual use's that pydoc serves first.
>>
>> Use's for pydoc in order of importance and frequency of use:
>>
>>     1. Console (builtin) help.  ie.. the help() function.
>>     2. HTML browsing and quick reference.
>>     3. Document generation in text.
>>     4. Document generation in html.
>>     5. Document generation in other formats.  (not currently possible)
>>
>> I'm concentrating on 1 and 2.  Use cases 3 and 4 are just an easy to do
>> byproduct of doing 1 and 2.  I think the cleaning up may make doing 5 
>> possible.
> 
> I am fully on that line, with the remark that thinking about point 5
> early is that
> could make the cut. The exercise will be in avoiding over-complication
> in a design that is not used in the end.
> 
> Reactions on this thread brought a lot of good ideas and pointers to
> existing work.
> It loo

Yes, It does help to have additional view points and references.


>> Lets turn the question around.  How well would other document 
>> generators supply
>> pydoc with the equivalent text of the help() function, interactive 
>> help session
>> output, and the equivalent html needed for dynamic html browsing?
>>
>> Also keep in mind the help function is always by default imported into 
>> python,
>> so keeping that small and relatively simple with a minimum of external
>> dependencies is good.
>>
>>
>>  > I am willing to contribute to the implementation (I suspect that
>>  > things like unit tests be needed).
>>
>> Yes, there will need to be some unit tests.  It may even help for 
>> those be
>> written now.  That would help us identify things that still need to be 
>> done.
> 
> Regarding tests, it is never early enough to think about them (that
> let one write
> code that is actually "test-able").

Right, and thanks for taking more than a casual interest in this. :-)

Cheers,
    Ron



From steve at holdenweb.com  Mon Jan  8 17:26:55 2007
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 08 Jan 2007 16:26:55 +0000
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <45A26BD9.8080404@ronadam.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>		<459D27FE.6070406@ronadam.com>		<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>		<459EC6B8.7000600@ronadam.com>		<27d1e6020701051821r749b8f4dya5b28107b8f410ba@mail.gmail.com>		<45A01F82.9070802@ronadam.com>	<27d1e6020701080321h77a4b04cia221cf7d6ca6c853@mail.gmail.com>
	<45A26BD9.8080404@ronadam.com>
Message-ID: <45A270CF.7070301@holdenweb.com>

Ron Adam wrote:
> Laurent Gautier wrote:
>> 2007/1/7, Ron Adam <rrr at ronadam.com>:
>>> Laurent Gautier wrote:
>>>> 2007/1/6, Ron Adam <rrr at ronadam.com>:
>> [...]
>>> I'd like to know more about using the sandbox, I know it would be easy 
>>> for
>>> people to read the source there, but who all can have write access to 
>>> it without
>>> having write access to other python areas?  I would not mind giving 
>>> that a try
>>> if someone who already knows how could point me to the correct how-to
>>> documentation with some advice on what not to do.
>> Limiting where different people can commit code changes is possible... it's
>> just that I am not certain whether sourceforge allows it or not.
>> I asked A.M. Kuchling about that.
> 
> I'm not concerned about limiting changes to this project.  I want others to work 
> on it.  Can write access be *easily* granted to just one cvs sandbox directory, 
> for this project, without granting access to other directories in the sandbox or 
> trunk?
> 
An alternative is just to trust people only to scribble in the sandbox. 
I really don't think we have a bog security issue here as long as the 
individuals are "known the the community" (or should that be "known to 
the PSU"?).

After all, it *is* a configuration control system, which by definition 
can revert to previous content in the event of some unwanted change.

>>> I'm actually more concerned about the what not to do stuff.  I really 
>>> would not
>>> like to clobber someone else's work or create problems because of my
>>> inexperience with CVS.
>> I see that you are under Microsoft windows, so you may want to check
>> TortoiseSVN.
>> (The python project is stored on a SVN server, so it would make sense
>> to favor this  one over CVS - in the case the project administrators
>> have directory-level control -).
> 
> Thank's I'll try TortoiseSVN out. :-)
> 
I can heartily recommend it.
> 
>> Regarding the possibility of jeopardizing something in the repository,
>> the directory-level sandbox should only allow you trash your own work
>> ;-)
>> (but even then, you should always be able to recover from mistakes).
> 
> That would be fine then.  But I'll let you decide since you are offering to 
> manage getting it set up.
>
[ ... ]

Let's not spend too much time on paranoid administration, since we are 
supposed to be an open source community :)

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com

From steve at holdenweb.com  Mon Jan  8 17:26:55 2007
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 08 Jan 2007 16:26:55 +0000
Subject: [Python-Dev] Pydoc Improvements / Rewrite
In-Reply-To: <45A26BD9.8080404@ronadam.com>
References: <27d1e6020701040438t4a456209iff118bcf2931cfa4@mail.gmail.com>		<459D27FE.6070406@ronadam.com>		<27d1e6020701050201v12097967p8a6ff7aa6d581fd@mail.gmail.com>		<459EC6B8.7000600@ronadam.com>		<27d1e6020701051821r749b8f4dya5b28107b8f410ba@mail.gmail.com>		<45A01F82.9070802@ronadam.com>	<27d1e6020701080321h77a4b04cia221cf7d6ca6c853@mail.gmail.com>
	<45A26BD9.8080404@ronadam.com>
Message-ID: <45A270CF.7070301@holdenweb.com>

Ron Adam wrote:
> Laurent Gautier wrote:
>> 2007/1/7, Ron Adam <rrr at ronadam.com>:
>>> Laurent Gautier wrote:
>>>> 2007/1/6, Ron Adam <rrr at ronadam.com>:
>> [...]
>>> I'd like to know more about using the sandbox, I know it would be easy 
>>> for
>>> people to read the source there, but who all can have write access to 
>>> it without
>>> having write access to other python areas?  I would not mind giving 
>>> that a try
>>> if someone who already knows how could point me to the correct how-to
>>> documentation with some advice on what not to do.
>> Limiting where different people can commit code changes is possible... it's
>> just that I am not certain whether sourceforge allows it or not.
>> I asked A.M. Kuchling about that.
> 
> I'm not concerned about limiting changes to this project.  I want others to work 
> on it.  Can write access be *easily* granted to just one cvs sandbox directory, 
> for this project, without granting access to other directories in the sandbox or 
> trunk?
> 
An alternative is just to trust people only to scribble in the sandbox. 
I really don't think we have a bog security issue here as long as the 
individuals are "known the the community" (or should that be "known to 
the PSU"?).

After all, it *is* a configuration control system, which by definition 
can revert to previous content in the event of some unwanted change.

>>> I'm actually more concerned about the what not to do stuff.  I really 
>>> would not
>>> like to clobber someone else's work or create problems because of my
>>> inexperience with CVS.
>> I see that you are under Microsoft windows, so you may want to check
>> TortoiseSVN.
>> (The python project is stored on a SVN server, so it would make sense
>> to favor this  one over CVS - in the case the project administrators
>> have directory-level control -).
> 
> Thank's I'll try TortoiseSVN out. :-)
> 
I can heartily recommend it.
> 
>> Regarding the possibility of jeopardizing something in the repository,
>> the directory-level sandbox should only allow you trash your own work
>> ;-)
>> (but even then, you should always be able to recover from mistakes).
> 
> That would be fine then.  But I'll let you decide since you are offering to 
> manage getting it set up.
>
[ ... ]

Let's not spend too much time on paranoid administration, since we are 
supposed to be an open source community :)

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From python at rcn.com  Mon Jan  8 18:16:20 2007
From: python at rcn.com (Raymond Hettinger)
Date: Mon, 8 Jan 2007 09:16:20 -0800
Subject: [Python-Dev] Bug or not? Different behaviour iterating list
	andcollections.deque
References: <enqiif$ko4$1@sea.gmane.org> <45A19243.2000109@v.loewis.de>
	<ent2uf$5je$1@sea.gmane.org>
Message-ID: <003c01c73348$b869ecc0$ea146b0a@RaymondLaptop1>

"Christos Georgiou"
> In retrospection, the example code I chose, although it showed the two
> issues I thought important ('list / deque iteration discrepancy' and 'empty 
> / non-empty deque iteration discrepancy') was not as plain as needed, ...

Lists are unique in the way they allow mutation during iteration because
indexed lookup allows for a meaningful definition of what should be
done as the list mutates.

In contrast, deques are more like dictionaries and should not be mutated
during iteration.  The issue is that a deque could be so substantially
modified that there is not a clear, meaningful, and useful definition of what
item should be next served-up.

With respect to the second question, please assign an SF report to me
and I'll look at it in detail when I have time.  It appears to be an
idiosyncracy resulting from the ordering of the test for StopIteration
versus the test for RuntimeError.  The question is whether I can swap
the test order without introducing other anomalies.

Raymond



P.S.  The patch would look like this:

Index: collectionsmodule.c
===================================================================
--- collectionsmodule.c (revision 53281)
+++ collectionsmodule.c (working copy)
@@ -911,15 +911,14 @@
 {
        PyObject *item;

-       if (it->counter == 0)
-               return NULL;
-
        if (it->deque->state != it->state) {
                it->counter = 0;
                PyErr_SetString(PyExc_RuntimeError,
                                "deque mutated during iteration");
                return NULL;
        }
+       if (it->counter == 0)
+               return NULL;
        assert (!(it->b == it->deque->rightblock &&
                  it->index > it->deque->rightindex));

From mail at timgolden.me.uk  Tue Jan  9 10:33:05 2007
From: mail at timgolden.me.uk (Tim Golden)
Date: Tue, 9 Jan 2007 09:33:05 -0000 (GMT)
Subject: [Python-Dev] Apologies from the CBS autoresponder
Message-ID: <61679.81.171.156.66.1168335185.squirrel@81.171.156.66>

My many apologies to those who had their New Year
interrupted by my company's inane autoresponder.
I'm afraid that I went on leave for a week without
suspending list delivery just as my company changed
its name and sent out (unknown to me) an automatic
message to *everything* coming in to the old address.

I've pulled the plug on that address's subscription,
altho' I suspect some list admin got there before me.
Apologies again.

Tim Golden

From walter at livinglogic.de  Tue Jan  9 14:02:10 2007
From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=)
Date: Tue, 09 Jan 2007 14:02:10 +0100
Subject: [Python-Dev] DRAFT: pydev summary for 2006-12-01 to 2006-12-15
In-Reply-To: <76fd5acf0701080030t1cfc6d32n922efa64a29e4e43@mail.gmail.com>
References: <76fd5acf0701080030t1cfc6d32n922efa64a29e4e43@mail.gmail.com>
Message-ID: <45A39252.6070806@livinglogic.de>

Calvin Spealman wrote:

> [...]
> -----------------------------------
> Creating dicts from dict subclasses
> -----------------------------------
> 
> A common problem of subclassing built-in types (dict, in this case) arose trying
> to move weakref.WeakValueDictionary to a subclass of dict, rather than UserDict.
> 
> [NOTE: Could someone give a better description of this? I don't know enough of
> the issue here to write any content full enough. Thanks.]

The problem is that if you pass a mapping (i.e. something with a keys() 
method) to the dict constructor, it uses keys() and __getitem__() to 
make a copy. However if the argument is a subclass of dict, any 
overwritten keys() or __getitem__() methods are ignored.

Servus,
    Walter



From anthony at interlink.com.au  Wed Jan 10 05:24:05 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed, 10 Jan 2007 15:24:05 +1100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <200701091152.48048.anthony@interlink.com.au>
References: <200701081650.17353.anthony@interlink.com.au>
	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>
	<200701091152.48048.anthony@interlink.com.au>
Message-ID: <200701101524.06515.anthony@interlink.com.au>

cc'ing python-dev - followups should probably go there, rather than 
the p3yk list.

So here's my latest plan:

- Add a Py3KDeprecationWarning, as a subclass of DeprecationWarning 
(or maybe just of Warning)
- Add a -W py3k shortcut, which is the same as 
"-W once:Py3kDeprecationWarning"
- Add a C int (as in previous patch) that is also set to 1 if the 
command line flag is specified. This is because there's no easy or 
quick way to check that a warning has been set - and calling into 
the warnings code is expensive, even if the C code warnings module 
is done. We can revisit this if the C version of warnings grows 
some extra features to make this less awful. 

For 2.6, we make a single -t (mixing tabs and spaces) the default, 
and maybe add -T to suppress this.

DeprecationWarning for 2.6:
- `foo` (backticks)
- input()

The following are the list of things that get 
Py3kDeprecationWarnings (so far):

integer division - make "-Q warn" the default when -W py3k specified
coerce() - going away
apply() - going away
intern() - moving to sys (we should also move it to sys, and make 
intern() - call sys.intern())
file.xreadlines() - going away
dict.has_key() - use 'in dict'
<> - use != (aka "MakeBarryCryDeprecationWarning")
__(get|set|del)slice__
__coerce__ - gone
__(div|idiv|rdiv)__ - no replacement?
__nonzero__ - we should add __bool__ to 2.6, and then warn people. 
  Not sure we can rename the nb_nonzero slot, though.

For the various __foo__ slots, it might be nice to warn when they're 
defined, rather than used, but I've not looked into how hard this 
would be to do, or whether it would still work with .pyc files and 
the like. On the other hand, warning on use might also let us pick 
up when C code modules use the same functions. 

There's other changes that are probably too hard to warn about, 
because there's not an easy replacement - the exec and print 
statements come to mind here. 

Comments? What else should get warnings?
Anthony

From guido at python.org  Wed Jan 10 06:24:23 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 9 Jan 2007 21:24:23 -0800
Subject: [Python-Dev] [Python-3000-checkins] r53335 - in
	python/branches/p3yk: Doc/Makefile.deps Doc/lib/lib.tex
	Doc/lib/libsets.tex Doc/lib/libstdtypes.tex
	Lib/msilib/__init__.py Lib/sets.py Lib/test/test_cookielib.py
	Lib/test/test_set.py Lib/test/test_sets.py
In-Reply-To: <bbaeab100701091939r6e972491ue1eb86ab5e93ce3e@mail.gmail.com>
References: <20070110012835.B33831E4008@bag.python.org>
	<bbaeab100701091939r6e972491ue1eb86ab5e93ce3e@mail.gmail.com>
Message-ID: <ca471dc20701092124u370ed0d7v26e83c5d0e7d84c8@mail.gmail.com>

On 1/9/07, Brett Cannon <brett at python.org> wrote:
> Should we bother with deprecating 'sets' in the 2.x series?

I'd say so, yes.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From raymond.hettinger at verizon.net  Wed Jan 10 19:13:39 2007
From: raymond.hettinger at verizon.net (Raymond Hettinger)
Date: Wed, 10 Jan 2007 10:13:39 -0800
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
References: <200701081650.17353.anthony@interlink.com.au><9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com><200701091152.48048.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
Message-ID: <00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>

<"Anthony Baxter">
> Comments? What else should get warnings?

It is my strong preference that we not go down this path.  
Instead, the 2.6 vs 3.0 difference analysis should go in an 
external lint utility.

The Py2.x series may live-on for some time and should do so
as if Py3.x did not exist.  Burdening the 2.x code with loads
of warnings will only clutter the source code and make maintenance 
more difficult.  There may also be some performance impact.

We should resolve that Py2.6 remain as clean as possible
and that Py3.0 be kept in its own world.  Forging a new
blade does not have to entail dulling the trusty old blade.


Raymond




From brett at python.org  Wed Jan 10 19:52:03 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 10 Jan 2007 10:52:03 -0800
Subject: [Python-Dev] [Python-3000-checkins] r53335 - in
	python/branches/p3yk: Doc/Makefile.deps Doc/lib/lib.tex
	Doc/lib/libsets.tex Doc/lib/libstdtypes.tex
	Lib/msilib/__init__.py Lib/sets.py Lib/test/test_cookielib.py
	Lib/test/test_set.py Lib/test/test_sets.py
In-Reply-To: <ca471dc20701092124u370ed0d7v26e83c5d0e7d84c8@mail.gmail.com>
References: <20070110012835.B33831E4008@bag.python.org>
	<bbaeab100701091939r6e972491ue1eb86ab5e93ce3e@mail.gmail.com>
	<ca471dc20701092124u370ed0d7v26e83c5d0e7d84c8@mail.gmail.com>
Message-ID: <bbaeab100701101052n1a56d423g7464e2737930da30@mail.gmail.com>

On 1/9/07, Guido van Rossum <guido at python.org> wrote:
> On 1/9/07, Brett Cannon <brett at python.org> wrote:
> > Should we bother with deprecating 'sets' in the 2.x series?
>
> I'd say so, yes.
>

OK, I work on deprecating the sets module in 2.6 .

-Brett

From thomas at python.org  Wed Jan 10 20:42:16 2007
From: thomas at python.org (Thomas Wouters)
Date: Wed, 10 Jan 2007 20:42:16 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
References: <200701081650.17353.anthony@interlink.com.au>
	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>
	<200701091152.48048.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
Message-ID: <9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>

On 1/10/07, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
>
> <"Anthony Baxter">
> > Comments? What else should get warnings?
>
> It is my strong preference that we not go down this path.
> Instead, the 2.6 vs 3.0 difference analysis should go in an
> external lint utility.
>
> The Py2.x series may live-on for some time and should do so
> as if Py3.x did not exist.  Burdening the 2.x code with loads
> of warnings will only clutter the source code and make maintenance
> more difficult.  There may also be some performance impact.
>
> We should resolve that Py2.6 remain as clean as possible
> and that Py3.0 be kept in its own world.  Forging a new
> blade does not have to entail dulling the trusty old blade.


The idea is that we only generate the warnings optionally, only for things
that can be written in a manner compatible with prevalent Python versions,
and in the most efficient manner we can manage, *except* for the few things
that are already considered (by many) criminal to use: input(), backtics,
mixed tabs and spaces. In other words, for any code written even remotely
sane in the last five years, no extra warnings will be generated. By Guido's
plan, 3.0 will arrive well before 2.6, and the migration step is not as
large as many fear it to be. Having Python 2.6 optionally warn for
3.0-compatibility is a lot easier for the average developer than having a
separate tool or a separately compiled Python.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070110/6483188f/attachment.htm 

From g.brandl at gmx.net  Wed Jan 10 20:46:25 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 10 Jan 2007 20:46:25 +0100
Subject: [Python-Dev] python-ideas on gmane
Message-ID: <eo3fqi$af9$1@sea.gmane.org>

Hi,

is there any objection against making python-ideas available on gmane?
Should I just suggest it myself, or has someone to do extra admin work
on the mailing list side?

Georg


From barry at python.org  Wed Jan 10 20:59:06 2007
From: barry at python.org (Barry Warsaw)
Date: Wed, 10 Jan 2007 14:59:06 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
References: <200701081650.17353.anthony@interlink.com.au>
	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>
	<200701091152.48048.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
Message-ID: <9B10BE24-2A2F-4BAE-A6CF-82E3E3BA76CD@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 10, 2007, at 2:42 PM, Thomas Wouters wrote:

> The idea is that we only generate the warnings optionally, only for  
> things
> that can be written in a manner compatible with prevalent Python  
> versions,
> and in the most efficient manner we can manage, *except* for the  
> few things
> that are already considered (by many) criminal to use: input(),  
> backtics,
> mixed tabs and spaces. In other words, for any code written even  
> remotely
> sane in the last five years, no extra warnings will be generated.  
> By Guido's
> plan, 3.0 will arrive well before 2.6, and the migration step is  
> not as
> large as many fear it to be. Having Python 2.6 optionally warn for
> 3.0-compatibility is a lot easier for the average developer than  
> having a
> separate tool or a separately compiled Python.

I tend to agree with Raymond that Python 2.x will live longer than  
any of us think it might, or would hope for.  I think it's even  
possible that there will be enough commercial impetus to keep 2.x  
alive with security updates (though with no new features) for a long  
while after 2.9 is released.  I also think that the migration path  
may be more difficult than we think, especially with the unicode/str  
changes.  Maybe it will be smooth and easy, but we don't know yet.   
(Heck, my company probably won't even upgrade from Python 2.4 to  
Python 2.5 any time soon.)

Having said that, I don't have too much of a problem with the general  
guidelines Thomas outlines above.  Just be really careful about what  
you criminalize in 2.6 because each will cause some current users  
pain.  I think the other thing we've learned is that we may not know  
how much pain we cause until after Python 2.6.0 is released and ultra- 
conservative users start to upgrade.  I can probably get on board  
with the three specific cases you mention above, because /my/ code is  
probably safe (although I wouldn't be surprised if some 10+ y.o. code  
still has a few lurking backticks), but I might balk if that slope  
gets any more slippery.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRaVFinEjvBPtnXfVAQJKYgQAg3RjH5OPF9TkdtWHqp9ylwnsRaBwuUdI
MGSD5ukkddl7XuluHMxVDIduxxTrQos9bCacLPtK5YejhxKnXU0V8jfDYyXv1pDB
j+DHJZ6cHQFTWfA7M48DMNPGoBSv0RyOSnrE9fMVrQ5D1dKO4sBIczzAMZ83y405
Zg3aDGE2voI=
=wo27
-----END PGP SIGNATURE-----

From collinw at gmail.com  Wed Jan 10 21:16:09 2007
From: collinw at gmail.com (Collin Winter)
Date: Wed, 10 Jan 2007 14:16:09 -0600
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
References: <200701081650.17353.anthony@interlink.com.au>
	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>
	<200701091152.48048.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
Message-ID: <43aa6ff70701101216x307f5b8cu4513bf01eb7c357d@mail.gmail.com>

On 1/10/07, Thomas Wouters <thomas at python.org> wrote:
> On 1/10/07, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
> > It is my strong preference that we not go down this path.
> > Instead, the 2.6 vs 3.0 difference analysis should go in an
> > external lint utility.
> >
> > The Py2.x series may live-on for some time and should do so
> > as if Py3.x did not exist.  Burdening the 2.x code with loads
> > of warnings will only clutter the source code and make maintenance
> > more difficult.  There may also be some performance impact.
> >
> > We should resolve that Py2.6 remain as clean as possible
> > and that Py3.0 be kept in its own world.  Forging a new
> > blade does not have to entail dulling the trusty old blade.
>
> The idea is that we only generate the warnings optionally, only for things
> that can be written in a manner compatible with prevalent Python versions,
> and in the most efficient manner we can manage, *except* for the few things
> that are already considered (by many) criminal to use: input(), backtics,
> mixed tabs and spaces. In other words, for any code written even remotely
> sane in the last five years, no extra warnings will be generated.

I'd rather see this effort invested in a tool like Guido's 2to3,
rather than in sprinkling warnings all through the 2.x codebase (even
if they are wrapped in #ifdef's). I don't imagine people developing
day-to-day in a 2.x environment are going to be so terribly interested
in "is this 3.x compliant?" that downloading a separate tool would be
an undue burden.

Collin Winter

From steve at holdenweb.com  Wed Jan 10 21:40:32 2007
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 10 Jan 2007 20:40:32 +0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <43aa6ff70701101216x307f5b8cu4513bf01eb7c357d@mail.gmail.com>
References: <200701081650.17353.anthony@interlink.com.au>	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>	<200701091152.48048.anthony@interlink.com.au>	<200701101524.06515.anthony@interlink.com.au>	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
	<43aa6ff70701101216x307f5b8cu4513bf01eb7c357d@mail.gmail.com>
Message-ID: <45A54F40.7080305@holdenweb.com>

Collin Winter wrote:
> On 1/10/07, Thomas Wouters <thomas at python.org> wrote:
>> On 1/10/07, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
>>> It is my strong preference that we not go down this path.
>>> Instead, the 2.6 vs 3.0 difference analysis should go in an
>>> external lint utility.
>>>
>>> The Py2.x series may live-on for some time and should do so
>>> as if Py3.x did not exist.  Burdening the 2.x code with loads
>>> of warnings will only clutter the source code and make maintenance
>>> more difficult.  There may also be some performance impact.
>>>
>>> We should resolve that Py2.6 remain as clean as possible
>>> and that Py3.0 be kept in its own world.  Forging a new
>>> blade does not have to entail dulling the trusty old blade.
>> The idea is that we only generate the warnings optionally, only for things
>> that can be written in a manner compatible with prevalent Python versions,
>> and in the most efficient manner we can manage, *except* for the few things
>> that are already considered (by many) criminal to use: input(), backtics,
>> mixed tabs and spaces. In other words, for any code written even remotely
>> sane in the last five years, no extra warnings will be generated.
> 
> I'd rather see this effort invested in a tool like Guido's 2to3,
> rather than in sprinkling warnings all through the 2.x codebase (even
> if they are wrapped in #ifdef's). I don't imagine people developing
> day-to-day in a 2.x environment are going to be so terribly interested
> in "is this 3.x compliant?" that downloading a separate tool would be
> an undue burden.
> 
I'm all for helping people to prepare for 3.0, since I don't want to see 
it languish in Perl 6 country. At the same time I agree with Raymond 
that migration to 3.0 can't be allowed to place obstacles in the way of 
2.X users. They shouldn't be penalised for using 2.X, particularly if 
they are new to the language, otherwise we will run the risk of 
adversely affecting the Python adoption rate - which I hope no reader of 
this list wants.

So, why not a new warning category: MigrationWarning?

This type of warning should appear only when the user specifically 
configures their Python installation to provide them, thereby allowing 
existing production code to continue working without users needing to 
switch warnings off.

There could even be levels, so it was possible to configure your Python 
2.X to give specific advice about migration to particular future 
versions, without any change in the action of the interpreter in the 
absence of any indication that the user wanted migration warnings. That 
way we are guiding our forward-looking users towards the future without 
chastising others for adopting, or sticking with, older versions.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From thomas at python.org  Wed Jan 10 21:48:22 2007
From: thomas at python.org (Thomas Wouters)
Date: Wed, 10 Jan 2007 21:48:22 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <45A54F40.7080305@holdenweb.com>
References: <200701081650.17353.anthony@interlink.com.au>
	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>
	<200701091152.48048.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
	<43aa6ff70701101216x307f5b8cu4513bf01eb7c357d@mail.gmail.com>
	<45A54F40.7080305@holdenweb.com>
Message-ID: <9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>

On 1/10/07, Steve Holden <steve at holdenweb.com> wrote:
>
> Collin Winter wrote:
> > On 1/10/07, Thomas Wouters <thomas at python.org> wrote:
> >> On 1/10/07, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
> >>> It is my strong preference that we not go down this path.
> >>> Instead, the 2.6 vs 3.0 difference analysis should go in an
> >>> external lint utility.
> >>>
> >>> The Py2.x series may live-on for some time and should do so
> >>> as if Py3.x did not exist.  Burdening the 2.x code with loads
> >>> of warnings will only clutter the source code and make maintenance
> >>> more difficult.  There may also be some performance impact.
> >>>
> >>> We should resolve that Py2.6 remain as clean as possible
> >>> and that Py3.0 be kept in its own world.  Forging a new
> >>> blade does not have to entail dulling the trusty old blade.
> >> The idea is that we only generate the warnings optionally, only for
> things
> >> that can be written in a manner compatible with prevalent Python
> versions,
> >> and in the most efficient manner we can manage, *except* for the few
> things
> >> that are already considered (by many) criminal to use: input(),
> backtics,
> >> mixed tabs and spaces. In other words, for any code written even
> remotely
> >> sane in the last five years, no extra warnings will be generated.
> >
> > I'd rather see this effort invested in a tool like Guido's 2to3,


They serve a different purpose, and it isn't taking any time away from me
(or Anthony, I can confidently guess) working on 2to3.

> rather than in sprinkling warnings all through the 2.x codebase (even
> > if they are wrapped in #ifdef's). I don't imagine people developing
> > day-to-day in a 2.x environment are going to be so terribly interested
> > in "is this 3.x compliant?" that downloading a separate tool would be
> > an undue burden.
> >
> I'm all for helping people to prepare for 3.0, since I don't want to see
> it languish in Perl 6 country. At the same time I agree with Raymond
> that migration to 3.0 can't be allowed to place obstacles in the way of
> 2.X users. They shouldn't be penalised for using 2.X, particularly if
> they are new to the language, otherwise we will run the risk of
> adversely affecting the Python adoption rate - which I hope no reader of
> this list wants.
>
> So, why not a new warning category: MigrationWarning?


I believe Anthony suggested Py3KDeprecationWarning or such. I don't think
the name really matters.

This type of warning should appear only when the user specifically
> configures their Python installation to provide them, thereby allowing
> existing production code to continue working without users needing to
> switch warnings off.


Sorry, that doesn't work -- most programmers don't build their own Python,
and it's way too much to expect all distributions to provide two versions of
the same Python. Let's not over-engineer this. We'll add the warning flag
the way Anthony suggested, and measure the speed impact. If the speed impact
is too high, we should just forget about it. This isn't a spaceship, this is
just a little deprecation warning for people who want to see it -- people
who wonder whether python3.0 will really be all that different, whether
their code will need extensive modification or not, but don't want to
download a conversion tool and/or a python3.0 installation just to figure
that out.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070110/50d98120/attachment-0001.htm 

From brett at python.org  Wed Jan 10 22:02:03 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 10 Jan 2007 13:02:03 -0800
Subject: [Python-Dev] python-ideas on gmane
In-Reply-To: <eo3fqi$af9$1@sea.gmane.org>
References: <eo3fqi$af9$1@sea.gmane.org>
Message-ID: <bbaeab100701101302h2727f057p52fefd6e034892e6@mail.gmail.com>

On 1/10/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Hi,
>
> is there any objection against making python-ideas available on gmane?
> Should I just suggest it myself, or has someone to do extra admin work
> on the mailing list side?
>

Just email me personally the email address to subscribe.

-Brett

From python at rcn.com  Wed Jan 10 22:11:33 2007
From: python at rcn.com (Raymond Hettinger)
Date: Wed, 10 Jan 2007 13:11:33 -0800
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
References: <200701081650.17353.anthony@interlink.com.au>
	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>
	<200701091152.48048.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
Message-ID: <00b001c734fb$e9433540$ea146b0a@RaymondLaptop1>

[Thomas Wouters]
>  By Guido's plan, 3.0 will arrive well before 2.6, and the migration step is 
> not as
> large as many fear it to be. Having Python 2.6 optionally warn for 
> 3.0-compatibility
> is a lot easier for the average developer than having a separate tool or a 
> separately
> compiled Python.

If 3.0 comes out before Py2.6, ISTM that the authorative tool for checking Py3.0
compatibility is to simply run the code in Py3.0.  Anything less is sure to miss 
something.

Also, the usual reasons apply for having a stand-alone pychecker/pylint tool or 
module.
As a separate tool, we can make rapid updates, add new tests, fix bugs, and 
enhance
this usability of the tool in response to user feedback.

One other thought is that warnings framework may not be the best way for a 
developer
to go about updating his or her code.  For me the warnings framework has almost
never been of use for code updates..  It is easier and more comprehensive to 
grep for
backticks or for calls to "coerce" than to run code and hope that every code 
path gets
exercised.



Raymond

From sylvain.thenault at logilab.fr  Wed Jan 10 22:06:18 2007
From: sylvain.thenault at logilab.fr (Sylvain =?iso-8859-1?Q?Th=E9nault?=)
Date: Wed, 10 Jan 2007 22:06:18 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
References: <200701081650.17353.anthony@interlink.com.au>
	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>
	<200701091152.48048.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
Message-ID: <20070110210601.GA18942@crater.logilab.fr>

sorry this is actually more an answer to Raymond's email but I
accendidentally delete it some I'm replying there.

On Wednesday 10 January ? 20:42, Thomas Wouters wrote:
> On 1/10/07, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
> 
>     <"Anthony Baxter">
>     > Comments? What else should get warnings?
> 
>     It is my strong preference that we not go down this path.
>     Instead, the 2.6 vs 3.0 difference analysis should go in an
>     external lint utility.
> 
>     The Py2.x series may live-on for some time and should do so
>     as if Py3.x did not exist.  Burdening the 2.x code with loads
>     of warnings will only clutter the source code and make maintenance
>     more difficult.  There may also be some performance impact.
> 
>     We should resolve that Py2.6 remain as clean as possible
>     and that Py3.0 be kept in its own world.  Forging a new
>     blade does not have to entail dulling the trusty old blade.

Just notice that pylint is already warning for some py3k deprecation
such as input(), <> and so on. It would be pretty easy to add warnings
for the missing stuff provided a complete list of changes. 
Even better, pylint sorts its messages between various categories, and
it would be as easy to get a py3k migration category so users can launch
pylint to get only migration related messages (or filter them out as
well). IMO that could be acheive in a couple of hours without anymore
work involved.

-- 
Sylvain Th?nault                               LOGILAB, Paris (France)
Formations Python, Zope, Plone, Debian:  http://www.logilab.fr/formations
D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
Python et calcul scientifique:           http://www.logilab.fr/science


From glyph at divmod.com  Wed Jan 10 22:41:50 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Wed, 10 Jan 2007 21:41:50 -0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Message-ID: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>

On 07:42 pm, thomas at python.org wrote:
>On 1/10/07, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
>>
>><"Anthony Baxter">
>> > Comments? What else should get warnings?
>>
>>It is my strong preference that we not go down this path.
>>Instead, the 2.6 vs 3.0 difference analysis should go in an
>>external lint utility.

>Having Python 2.6 optionally warn for
>3.0-compatibility is a lot easier for the average developer than having a
>separate tool or a separately compiled Python.

I could not possibly agree more.

Given the highly dynamic nature of Python, such a tool will *at best* catch only the most egregious uses of deprecated features.  Backticks are easy enough to find, but the *only* way that I can reasonably imagine migrating a body of code like Twisted (or any non-trivial Python library) would be having a way to examine the warning output of its test suite.

I am suuuuper +1 on the warnings for 2.6, as well as forward-compatibility at some point in the 2.x series for new syntax.  Without the ability to bridge 2.x->3.0 during some interim period, I can say for sure that Twisted _will not_ migrate to 3.0, ever.  We are really a small project and just don't have the manpower to maintain two overlapping but mutually incompatible codebases.

I've been assuming for some time that the only hope for Py3k compatibility within Twisted would be using PyPy as a translation layer.  With the addition of runtime compatibility warnings, it might be feasible that we could run on the "bare metal" (ha ha) of Python3's VM.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070110/cfd42769/attachment.htm 

From steve at holdenweb.com  Wed Jan 10 23:39:54 2007
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 10 Jan 2007 22:39:54 +0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>
References: <200701081650.17353.anthony@interlink.com.au>	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>	<200701091152.48048.anthony@interlink.com.au>	<200701101524.06515.anthony@interlink.com.au>	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>	<43aa6ff70701101216x307f5b8cu4513bf01eb7c357d@mail.gmail.com>	<45A54F40.7080305@holdenweb.com>
	<9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>
Message-ID: <45A56B3A.4000000@holdenweb.com>

Thomas Wouters wrote:
> 
> 
> On 1/10/07, *Steve Holden* <steve at holdenweb.com 
> <mailto:steve at holdenweb.com>> wrote:
> 
>     Collin Winter wrote:
>      > On 1/10/07, Thomas Wouters <thomas at python.org
>     <mailto:thomas at python.org>> wrote:
>      >> On 1/10/07, Raymond Hettinger < raymond.hettinger at verizon.net
>     <mailto:raymond.hettinger at verizon.net>> wrote:
>      >>> It is my strong preference that we not go down this path.
>      >>> Instead, the 2.6 vs 3.0 difference analysis should go in an
>      >>> external lint utility.
>      >>>
>      >>> The Py2.x series may live-on for some time and should do so
>      >>> as if Py3.x did not exist.  Burdening the 2.x code with loads
>      >>> of warnings will only clutter the source code and make maintenance
>      >>> more difficult.  There may also be some performance impact.
>      >>>
>      >>> We should resolve that Py2.6 remain as clean as possible
>      >>> and that Py3.0 be kept in its own world.  Forging a new
>      >>> blade does not have to entail dulling the trusty old blade.
>      >> The idea is that we only generate the warnings optionally, only
>     for things
>      >> that can be written in a manner compatible with prevalent Python
>     versions,
>      >> and in the most efficient manner we can manage, *except* for the
>     few things
>      >> that are already considered (by many) criminal to use: input(),
>     backtics,
>      >> mixed tabs and spaces. In other words, for any code written even
>     remotely
>      >> sane in the last five years, no extra warnings will be generated.
>      >
>      > I'd rather see this effort invested in a tool like Guido's 2to3,
> 
The above appears to be a quoting error, attributing comments to me that 
were actually made by Collin Winter.
> 
> They serve a different purpose, and it isn't taking any time away from 
> me (or Anthony, I can confidently guess) working on 2to3.
> 
>      > rather than in sprinkling warnings all through the 2.x codebase (even
>      > if they are wrapped in #ifdef's). I don't imagine people developing
>      > day-to-day in a 2.x environment are going to be so terribly
>     interested
>      > in "is this 3.x compliant?" that downloading a separate tool would be
>      > an undue burden.
>      >
>     I'm all for helping people to prepare for 3.0, since I don't want to see
>     it languish in Perl 6 country. At the same time I agree with Raymond
>     that migration to 3.0 can't be allowed to place obstacles in the way of
>     2.X users. They shouldn't be penalised for using 2.X, particularly if
>     they are new to the language, otherwise we will run the risk of
>     adversely affecting the Python adoption rate - which I hope no reader of
>     this list wants.
> 
>     So, why not a new warning category: MigrationWarning? 
> 
> 
> I believe Anthony suggested Py3KDeprecationWarning or such. I don't 
> think the name really matters.
> 
I quite agree. I was really disagreeing with the proposal that the new 
warning be a subclass of DeprecationWarning, since that implies that 
warnings will appear without being requested - that would, IMHO, be a 
sad approach to migration. I'd like users who decide to remain with the 
2.x series not to suffer at all as a result of that decision (except for 
missing out on a major language development, of course).

>     This type of warning should appear only when the user specifically
>     configures their Python installation to provide them, thereby allowing
>     existing production code to continue working without users needing to
>     switch warnings off. 
> 
> 
> Sorry, that doesn't work -- most programmers don't build their own 
> Python, and it's way too much to expect all distributions to provide two 
> versions of the same Python. Let's not over-engineer this. We'll add the 
> warning flag the way Anthony suggested, and measure the speed impact. If 
> the speed impact is too high, we should just forget about it. This isn't 
> a spaceship, this is just a little deprecation warning for people who 
> want to see it -- people who wonder whether python3.0 will really be all 
> that different, whether their code will need extensive modification or 
> not, but don't want to download a conversion tool and/or a python3.0 
> installation just to figure that out.
> 
Two versions aren't required, and that's not what I was proposing. There 
should be a command-line an option to switch these warnings *on*, rather 
than having to take the usual action of turning DeprecationWarning 
*off*. In other words, only intrude on the lives of those who 
consciously seek advice on migration.

Quite happy to drop the idea if the impact on execution speed is too 
great, but I anticipate most of the warnings would occur during parsing 
and code generation, and having them off by default ought to minimise 
the impact.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From steve at holdenweb.com  Wed Jan 10 23:39:54 2007
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 10 Jan 2007 22:39:54 +0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>
References: <200701081650.17353.anthony@interlink.com.au>	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>	<200701091152.48048.anthony@interlink.com.au>	<200701101524.06515.anthony@interlink.com.au>	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>	<43aa6ff70701101216x307f5b8cu4513bf01eb7c357d@mail.gmail.com>	<45A54F40.7080305@holdenweb.com>
	<9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>
Message-ID: <45A56B3A.4000000@holdenweb.com>

Thomas Wouters wrote:
> 
> 
> On 1/10/07, *Steve Holden* <steve at holdenweb.com 
> <mailto:steve at holdenweb.com>> wrote:
> 
>     Collin Winter wrote:
>      > On 1/10/07, Thomas Wouters <thomas at python.org
>     <mailto:thomas at python.org>> wrote:
>      >> On 1/10/07, Raymond Hettinger < raymond.hettinger at verizon.net
>     <mailto:raymond.hettinger at verizon.net>> wrote:
>      >>> It is my strong preference that we not go down this path.
>      >>> Instead, the 2.6 vs 3.0 difference analysis should go in an
>      >>> external lint utility.
>      >>>
>      >>> The Py2.x series may live-on for some time and should do so
>      >>> as if Py3.x did not exist.  Burdening the 2.x code with loads
>      >>> of warnings will only clutter the source code and make maintenance
>      >>> more difficult.  There may also be some performance impact.
>      >>>
>      >>> We should resolve that Py2.6 remain as clean as possible
>      >>> and that Py3.0 be kept in its own world.  Forging a new
>      >>> blade does not have to entail dulling the trusty old blade.
>      >> The idea is that we only generate the warnings optionally, only
>     for things
>      >> that can be written in a manner compatible with prevalent Python
>     versions,
>      >> and in the most efficient manner we can manage, *except* for the
>     few things
>      >> that are already considered (by many) criminal to use: input(),
>     backtics,
>      >> mixed tabs and spaces. In other words, for any code written even
>     remotely
>      >> sane in the last five years, no extra warnings will be generated.
>      >
>      > I'd rather see this effort invested in a tool like Guido's 2to3,
> 
The above appears to be a quoting error, attributing comments to me that 
were actually made by Collin Winter.
> 
> They serve a different purpose, and it isn't taking any time away from 
> me (or Anthony, I can confidently guess) working on 2to3.
> 
>      > rather than in sprinkling warnings all through the 2.x codebase (even
>      > if they are wrapped in #ifdef's). I don't imagine people developing
>      > day-to-day in a 2.x environment are going to be so terribly
>     interested
>      > in "is this 3.x compliant?" that downloading a separate tool would be
>      > an undue burden.
>      >
>     I'm all for helping people to prepare for 3.0, since I don't want to see
>     it languish in Perl 6 country. At the same time I agree with Raymond
>     that migration to 3.0 can't be allowed to place obstacles in the way of
>     2.X users. They shouldn't be penalised for using 2.X, particularly if
>     they are new to the language, otherwise we will run the risk of
>     adversely affecting the Python adoption rate - which I hope no reader of
>     this list wants.
> 
>     So, why not a new warning category: MigrationWarning? 
> 
> 
> I believe Anthony suggested Py3KDeprecationWarning or such. I don't 
> think the name really matters.
> 
I quite agree. I was really disagreeing with the proposal that the new 
warning be a subclass of DeprecationWarning, since that implies that 
warnings will appear without being requested - that would, IMHO, be a 
sad approach to migration. I'd like users who decide to remain with the 
2.x series not to suffer at all as a result of that decision (except for 
missing out on a major language development, of course).

>     This type of warning should appear only when the user specifically
>     configures their Python installation to provide them, thereby allowing
>     existing production code to continue working without users needing to
>     switch warnings off. 
> 
> 
> Sorry, that doesn't work -- most programmers don't build their own 
> Python, and it's way too much to expect all distributions to provide two 
> versions of the same Python. Let's not over-engineer this. We'll add the 
> warning flag the way Anthony suggested, and measure the speed impact. If 
> the speed impact is too high, we should just forget about it. This isn't 
> a spaceship, this is just a little deprecation warning for people who 
> want to see it -- people who wonder whether python3.0 will really be all 
> that different, whether their code will need extensive modification or 
> not, but don't want to download a conversion tool and/or a python3.0 
> installation just to figure that out.
> 
Two versions aren't required, and that's not what I was proposing. There 
should be a command-line an option to switch these warnings *on*, rather 
than having to take the usual action of turning DeprecationWarning 
*off*. In other words, only intrude on the lives of those who 
consciously seek advice on migration.

Quite happy to drop the idea if the impact on execution speed is too 
great, but I anticipate most of the warnings would occur during parsing 
and code generation, and having them off by default ought to minimise 
the impact.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From thomas at python.org  Wed Jan 10 23:48:01 2007
From: thomas at python.org (Thomas Wouters)
Date: Wed, 10 Jan 2007 23:48:01 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <45A56B3A.4000000@holdenweb.com>
References: <200701081650.17353.anthony@interlink.com.au>
	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>
	<200701091152.48048.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>
	<43aa6ff70701101216x307f5b8cu4513bf01eb7c357d@mail.gmail.com>
	<45A54F40.7080305@holdenweb.com>
	<9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>
	<45A56B3A.4000000@holdenweb.com>
Message-ID: <9e804ac0701101448x527d98ecmc7dd016ad295fb37@mail.gmail.com>

On 1/10/07, Steve Holden <steve at holdenweb.com> wrote:
>
> Thomas Wouters wrote:
> >
> >
> > On 1/10/07, *Steve Holden* <steve at holdenweb.com
> > <mailto:steve at holdenweb.com>> wrote:
> >
> >     Collin Winter wrote:
> >      > On 1/10/07, Thomas Wouters <thomas at python.org
> >     <mailto:thomas at python.org>> wrote:
> >      >> On 1/10/07, Raymond Hettinger < raymond.hettinger at verizon.net
> >     <mailto:raymond.hettinger at verizon.net>> wrote:
> >      >>> It is my strong preference that we not go down this path.
> >      >>> Instead, the 2.6 vs 3.0 difference analysis should go in an
> >      >>> external lint utility.
> >      >>>
> >      >>> The Py2.x series may live-on for some time and should do so
> >      >>> as if Py3.x did not exist.  Burdening the 2.x code with loads
> >      >>> of warnings will only clutter the source code and make
> maintenance
> >      >>> more difficult.  There may also be some performance impact.
> >      >>>
> >      >>> We should resolve that Py2.6 remain as clean as possible
> >      >>> and that Py3.0 be kept in its own world.  Forging a new
> >      >>> blade does not have to entail dulling the trusty old blade.
> >      >> The idea is that we only generate the warnings optionally, only
> >     for things
> >      >> that can be written in a manner compatible with prevalent Python
> >     versions,
> >      >> and in the most efficient manner we can manage, *except* for the
> >     few things
> >      >> that are already considered (by many) criminal to use: input(),
> >     backtics,
> >      >> mixed tabs and spaces. In other words, for any code written even
> >     remotely
> >      >> sane in the last five years, no extra warnings will be
> generated.
> >      >
> >      > I'd rather see this effort invested in a tool like Guido's 2to3,
> >
> The above appears to be a quoting error, attributing comments to me that
> were actually made by Collin Winter.


I'm sorry, that was unintentional. I was actually replying to Colin; I took
the opportunity to reply to two mails. I'm not sure what happened, it looked
right in gmail (and still does.)

I quite agree. I was really disagreeing with the proposal that the new
> warning be a subclass of DeprecationWarning, since that implies that
> warnings will appear without being requested - that would, IMHO, be a
> sad approach to migration. I'd like users who decide to remain with the
> 2.x series not to suffer at all as a result of that decision (except for
> missing out on a major language development, of course).


Ok, so, you're actually agreeing, except for the DeprecationWarning
subclassing. There was never an intent to display these py3k deprecation
warnings without an explicit flag (at least, not in this thread.) Hopefully
that puts some people at ease.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070110/8d89168e/attachment.htm 

From steve at holdenweb.com  Wed Jan 10 23:52:42 2007
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 10 Jan 2007 22:52:42 +0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9e804ac0701101448x527d98ecmc7dd016ad295fb37@mail.gmail.com>
References: <200701081650.17353.anthony@interlink.com.au>	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>	<200701091152.48048.anthony@interlink.com.au>	<200701101524.06515.anthony@interlink.com.au>	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>	<43aa6ff70701101216x307f5b8cu4513bf01eb7c357d@mail.gmail.com>	<45A54F40.7080305@holdenweb.com>	<9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>	<45A56B3A.4000000@holdenweb.com>
	<9e804ac0701101448x527d98ecmc7dd016ad295fb37@mail.gmail.com>
Message-ID: <45A56E3A.6070408@holdenweb.com>

Thomas Wouters wrote:
[...]
> 
> Ok, so, you're actually agreeing, except for the DeprecationWarning 
> subclassing. There was never an intent to display these py3k deprecation 
> warnings without an explicit flag (at least, not in this thread.) 
> Hopefully that puts some people at ease.
> 
If the action is as stated above you could make it a subclass of 
DeprecationWarning as far as I am concerned.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From steve at holdenweb.com  Wed Jan 10 23:52:42 2007
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 10 Jan 2007 22:52:42 +0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9e804ac0701101448x527d98ecmc7dd016ad295fb37@mail.gmail.com>
References: <200701081650.17353.anthony@interlink.com.au>	<9e804ac0701080353x6e84bd55q72848de26783635d@mail.gmail.com>	<200701091152.48048.anthony@interlink.com.au>	<200701101524.06515.anthony@interlink.com.au>	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>	<9e804ac0701101142x4b44b9fcn7b266c7dd16d290e@mail.gmail.com>	<43aa6ff70701101216x307f5b8cu4513bf01eb7c357d@mail.gmail.com>	<45A54F40.7080305@holdenweb.com>	<9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>	<45A56B3A.4000000@holdenweb.com>
	<9e804ac0701101448x527d98ecmc7dd016ad295fb37@mail.gmail.com>
Message-ID: <45A56E3A.6070408@holdenweb.com>

Thomas Wouters wrote:
[...]
> 
> Ok, so, you're actually agreeing, except for the DeprecationWarning 
> subclassing. There was never an intent to display these py3k deprecation 
> warnings without an explicit flag (at least, not in this thread.) 
> Hopefully that puts some people at ease.
> 
If the action is as stated above you could make it a subclass of 
DeprecationWarning as far as I am concerned.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From p.f.moore at gmail.com  Thu Jan 11 00:10:10 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 10 Jan 2007 23:10:10 +0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
Message-ID: <79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>

On 10/01/07, glyph at divmod.com <glyph at divmod.com> wrote:
> I've been assuming for some time that the only hope for Py3k compatibility
> within Twisted would be using PyPy as a translation layer.

Does this ring as many warning bells for me as it does for others? I
know very little about the current state of PyPy, but I read your
comment as implying that you expect Twisted to be unavailable (at some
level or other) for users of Py3K. Is that right?

How many other projects/packages anticipate *not* migrating to Py3K, I wonder?

Paul.

From p.f.moore at gmail.com  Thu Jan 11 00:11:01 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 10 Jan 2007 23:11:01 +0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
Message-ID: <79990c6b0701101511k7e04b5d2u6e6eaa11dc66179c@mail.gmail.com>

On 10/01/07, Paul Moore <p.f.moore at gmail.com> wrote:
> Does this ring as many warning bells for me as it does for others?

... for others as it does for me ...

Doh.
Paul

From benji at benjiyork.com  Thu Jan 11 00:46:40 2007
From: benji at benjiyork.com (Benji York)
Date: Wed, 10 Jan 2007 18:46:40 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
Message-ID: <45A57AE0.2060600@benjiyork.com>

Paul Moore wrote:
> How many other projects/packages anticipate *not* migrating to Py3K, I wonder?

I certainly can't speak for the project as a whole, but I anticipate a 
fair bit of work to port Zope 3 (100+ KLOC) to Python 3.0.
--
Benji York

From goodger at python.org  Thu Jan 11 01:04:27 2007
From: goodger at python.org (David Goodger)
Date: Wed, 10 Jan 2007 19:04:27 -0500
Subject: [Python-Dev] python-ideas on gmane
In-Reply-To: <eo3fqi$af9$1@sea.gmane.org>
References: <eo3fqi$af9$1@sea.gmane.org>
Message-ID: <45A57F0B.4050608@python.org>

[Georg Brandl]
> is there any objection against making python-ideas available on gmane?

It's already there. That's how I read it.

-- 
David Goodger <http://python.net/~goodger>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
Url : http://mail.python.org/pipermail/python-dev/attachments/20070110/6a18f1e1/attachment.pgp 

From pje at telecommunity.com  Thu Jan 11 01:30:49 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 10 Jan 2007 19:30:49 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.co
 m>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
Message-ID: <5.1.1.6.0.20070110193017.0296e388@sparrow.telecommunity.com>

At 11:10 PM 1/10/2007 +0000, Paul Moore wrote:
>On 10/01/07, glyph at divmod.com <glyph at divmod.com> wrote:
> > I've been assuming for some time that the only hope for Py3k compatibility
> > within Twisted would be using PyPy as a translation layer.
>
>Does this ring as many warning bells for me as it does for others? I
>know very little about the current state of PyPy, but I read your
>comment as implying that you expect Twisted to be unavailable (at some
>level or other) for users of Py3K. Is that right?
>
>How many other projects/packages anticipate *not* migrating to Py3K, I wonder?

I think it's safe to say that any package of comparable size is going to 
think twice about it without transitional compatibility.


From anthony at interlink.com.au  Thu Jan 11 02:44:39 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu, 11 Jan 2007 12:44:39 +1100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>
References: <200701081650.17353.anthony@interlink.com.au>
	<45A54F40.7080305@holdenweb.com>
	<9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>
Message-ID: <200701111244.41942.anthony@interlink.com.au>

On Thursday 11 January 2007 07:48, Thomas Wouters wrote:
> They serve a different purpose, and it isn't taking any time away
> from me (or Anthony, I can confidently guess) working on 2to3.

Correct. Note that checking for something like dict.has_key is going 
to be far far more reliable from inside the interpreter. Heck, one 
of the PEP-3xxx's actually calls for doing this.

> > I'm all for helping people to prepare for 3.0, since I don't
> > want to see it languish in Perl 6 country. At the same time I
> > agree with Raymond that migration to 3.0 can't be allowed to
> > place obstacles in the way of 2.X users. They shouldn't be
> > penalised for using 2.X, particularly if they are new to the
> > language, otherwise we will run the risk of adversely affecting
> > the Python adoption rate - which I hope no reader of this list
> > wants.
> >
> > So, why not a new warning category: MigrationWarning?
>
> I believe Anthony suggested Py3KDeprecationWarning or such. I
> don't think the name really matters.

Correct. In addition, please read what I posted - these warnings are 
off by default, and won't go through the warnings mechanism at all 
unless you specify the command line flag.

I've had a number of people say that this is something they would 
really, really like to see - the idea is both to let people migrate 
more easily, and provide reassurance that it won't be that bad to 
migrate!

-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From anthony at interlink.com.au  Thu Jan 11 02:48:10 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu, 11 Jan 2007 12:48:10 +1100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <eo3evi$7dr$1@sea.gmane.org>
References: <200701081650.17353.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
	<eo3evi$7dr$1@sea.gmane.org>
Message-ID: <200701111248.10791.anthony@interlink.com.au>

On Thursday 11 January 2007 06:32, Georg Brandl wrote:
> I guess there are quite a few people who won't start moving to
> Python 3.0 with 2.6, or even when 3.1 is out, as long as their
> program works fine with the 2.x line. There's no need to punish
> them with extra overhead.

Checking a single C global int is hardly going to make a huge impact 
at all.


-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From python at rcn.com  Thu Jan 11 03:04:05 2007
From: python at rcn.com (Raymond Hettinger)
Date: Wed, 10 Jan 2007 18:04:05 -0800
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
References: <200701081650.17353.anthony@interlink.com.au><45A54F40.7080305@holdenweb.com><9e804ac0701101248k55604b0dlc48dbd1d0d39c84d@mail.gmail.com>
	<200701111244.41942.anthony@interlink.com.au>
Message-ID: <011101c73524$c8692fe0$ea146b0a@RaymondLaptop1>

[Anthony Baxter]
> I've had a number of people say that this is something they would 
> really, really like to see - the idea is both to let people migrate 
> more easily, and provide reassurance that it won't be that bad to 
> migrate!

If Py3.0 is going to come out before Py2.6, can we table the discussion
until then?  We may find that a) migration was easier than we thought,
b) that stand-alone migration tools are sufficient, or c) by the time
Py2.6 comes-out, no one cares about having 2.x vs 3.x warnings.
OTOH, if people do care, then we'll have a strong case for loading
these warnings into Py2.6 before it gets close to being final.

Also, I'm wondering if the desire for 2.6 warnings is based on the notion that 
it will be possible to get large tools to work under both Py2.x and Py3.x.
With all the module renaming/packaging, old-style classes disappearing,
encoded text objects, real division and whatnot; that notion may be
a pipe-dream.

As far as "reassurance that it won't be that bad to migrate", screens full
of warnings may be less than reassuring.


Raymond

From jackdied at jackdied.com  Thu Jan 11 03:47:57 2007
From: jackdied at jackdied.com (Jack Diederich)
Date: Wed, 10 Jan 2007 21:47:57 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <011101c73524$c8692fe0$ea146b0a@RaymondLaptop1>
References: <200701111244.41942.anthony@interlink.com.au>
	<011101c73524$c8692fe0$ea146b0a@RaymondLaptop1>
Message-ID: <20070111024757.GH17734@performancedrivers.com>

On Wed, Jan 10, 2007 at 06:04:05PM -0800, Raymond Hettinger wrote:
> [Anthony Baxter]
> > I've had a number of people say that this is something they would 
> > really, really like to see - the idea is both to let people migrate 
> > more easily, and provide reassurance that it won't be that bad to 
> > migrate!
> 
> If Py3.0 is going to come out before Py2.6, can we table the discussion
> until then?  We may find that a) migration was easier than we thought,
> b) that stand-alone migration tools are sufficient, or c) by the time
> Py2.6 comes-out, no one cares about having 2.x vs 3.x warnings.
> OTOH, if people do care, then we'll have a strong case for loading
> these warnings into Py2.6 before it gets close to being final.

I'm also a fan of not scratching something until it itches but if
someone else already feels the itch and wants to do the work +0.
The pro warnings camp has said it won't add interpreter overhead unless
you ask for it (and they are willing to test that it is so).

> Also, I'm wondering if the desire for 2.6 warnings is based on the notion that 
> it will be possible to get large tools to work under both Py2.x and Py3.x.
> With all the module renaming/packaging, old-style classes disappearing,
> encoded text objects, real division and whatnot; that notion may be
> a pipe-dream.

No one has seriously suggested that it would be easy or if you prefer
no one serious has suggested it would be easy ;)

> As far as "reassurance that it won't be that bad to migrate", screens full
> of warnings may be less than reassuring.

If folks want to put in the effort (and people heavier than me have 
offered) to support light-weight optional warnings in addition to the
2to3 tool I can't complain.  It seems redundant to me but their time isn't
mine.

-Jack

From jimjjewett at gmail.com  Thu Jan 11 17:12:02 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Thu, 11 Jan 2007 11:12:02 -0500
Subject: [Python-Dev] Code working in both 2.x and 3.X
Message-ID: <fb6fbf560701110812p71b12567y6baae509c9747681@mail.gmail.com>

Raymond Hettinger wrote:

> Also, I'm wondering if the desire for 2.6 warnings is based on the notion that
> it will be possible to get large tools to work under both Py2.x and Py3.x.

I had certainly assumed it would be possible.

In fact, I had assumed that the 2->3 translator would have a mode
which left the code running in both.

This might not be the cleanest possible code for either line (parens
in 2.x, some extra iter calls or missed shortcuts, etc), but it should
certainly exist.  If it doesn't, then people who do want to support
both lines will themselves have to work exclusively in 2.x and
"compile" to 3.x.

> With all the module renaming/packaging,

I had believed that 2.x would support both names, typically as

    >>> # make an deprecation warning, or log entry
    >>> from oldname import *

So code using the new names may not work in 2.5, but should work in
2.6.  If you want it to work in 2.3, you either use conditional
imports or include your own forwarding modules for older pythons.

> old-style classes disappearing,

This is only a problem if you rely on specfically old-style semantics.
 Most old-style classes that I see work fine as new-style; the author
just didn't want to put in the extra boilerplate (or wanted 1.5
compatibility, or just didn't bother to make a change).

> encoded text objects,

This one I'm less sure about, but it seems that programs which work
correctly with both str and unicode in 2.x should work with 3, and
that programs which (in practice) restrict themselves to ASCII will
continue to work.

> real division

You can already get this a __future__ import, and you can explicitly
request the old form (in both lines) by truncating the result.

> and whatnot; that notion may be a pipe-dream.

I hope not.

That said, I'm not so sure about C extension modules...

-jJ

From glyph at divmod.com  Thu Jan 11 19:29:06 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 11 Jan 2007 18:29:06 -0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Message-ID: <20070111182906.17094.57269068.divmod.xquotient.1191@joule.divmod.com>

On 10 Jan, 11:10 pm, p.f.moore at gmail.com wrote:
>On 10/01/07, glyph at divmod.com <glyph at divmod.com> wrote:
>>I've been assuming for some time that the only hope for Py3k compatibility
>>within Twisted would be using PyPy as a translation layer.

>Does this ring as many warning bells for me as it does for others? I
>know very little about the current state of PyPy, but I read your
>comment as implying that you expect Twisted to be unavailable (at some
>level or other) for users of Py3K. Is that right?

This entirely depends on the migration path available.  My understanding of the situation is that Py3K is a different, incompatible language.  In that sense, Twisted will only be as unavailable to users of Py3K as it is unavailable to users of Ruby, Perl, or any other dynamic-but-not-Python language.

If the 2.x series grows deprecation warnings and gradually has all of py3k's features backported, such that there is a smooth transitional period of a few years where we can support one or two versions of 2.x and a version of 3.x, then eventually perhaps it will be worthwhile.

Alternately, if they really aren't compatible *but* there is such a compelling featureset drawing other developers that there's really a critical mass of projects that migrate to 3.x and stop maintaining their 2.x counterparts, it might be worthwhile to jump the chasm to follow them.  At a minimum, for it to be worthwhile for Twisted to attempt this if the following packages are all moved to 3.x:

 - pyopenssl
 - pygtk
 - pysqlite
 - PyObjC
 - win32all
 - zope interface (and therefore, probably, all of Zope)
 - pycrypto
 - pypam
 - pygame
 - pylucene

Those projects that do move I'm sure will take the "opportunity" to incompatibly break all of their APIs as well, since no existing code will actually use them, exacerbating the problems involved in porting.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070111/6d631903/attachment.html 

From foom at fuhm.net  Thu Jan 11 20:09:42 2007
From: foom at fuhm.net (James Y Knight)
Date: Thu, 11 Jan 2007 14:09:42 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <45A57AE0.2060600@benjiyork.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
	<45A57AE0.2060600@benjiyork.com>
Message-ID: <5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>

On Jan 10, 2007, at 6:46 PM, Benji York wrote:

> Paul Moore wrote:
>> How many other projects/packages anticipate *not* migrating to  
>> Py3K, I wonder?
>
> I certainly can't speak for the project as a whole, but I anticipate a
> fair bit of work to port Zope 3 (100+ KLOC) to Python 3.0.

I (another Twisted developer, among other hats I wear) am also very  
worried about the Python 3.0 transition.

If the goal is really to have Py 3.0 be released later this year,  
Twisted will almost certainly want to still be compatible with Python  
2.4 at that time. (Note that it's currently compatible down to Py  
2.3). Even discounting that level of compatibility, it's gonna have  
to be _at least_ compatible with whatever the latest 2.X release is  
(2.5, I guess, is the plan?).

Due to a lack of manpower, it is also not in all probability a  
realistic option to maintain two parallel almost-identical  
development branches, one that supports Python 2.x and one that  
supports Python 3 [1]. It would be quite a shame if this meant that  
Twisted effectively cannot move to Python 3.x for years after its  
release. I don't think anyone wants to see this happen. (and when I  
say "Twisted" here, I really mean "*every* sizeable python module or  
program with any current user base whatsoever".)

If Python 3.0 was simply a release which removed deprecated features,  
there would clearly be no issue. I would update my code in advance of  
the 3.0 release to not use any of those features being removed, and  
I'm all set. But that's not what I'm hearing. Python 3 is both adding  
new ways to do things, and removing the older way, in the same  
version, with no overlap. This makes me very anxious.

Basically: my plea is: please don't remove the old way of doing  
things in Py 3.0 at the same time as you add the new way of doing  
things. Have a couple versions of compatibility, just for me (and the  
other thousands of people in the same situation). Please do make the  
language better, and please do remove huge swaths of old deprecated  
APIs and functionality. Just, please don't gratuitously remove an old  
API to make Python a tiny bit cleaner, when there's not already  
something I can reasonably do *today* to replace it.

The dict.keys() change is a simple illustrative case.

Here's the situation as I understand it:
In Python 2.X, if I want an iterator of items, I call d.iteritems().  
d.items() returns a list.
In Python 3.X, if I want a iterator (or maybe an set-view; doesn't  
matter to me) of items, I call d.items(). There is no d.iteritems()

I'm completely positive about the change to make d.items() not return  
a list. It's a great thing to do. But note that there is NOTHING [2]  
I can call in order to get an iterator of items on both versions! If  
I want my code to work properly on both, I'll have to write a wrapper  
function that depends on the python version, like:
if py3k:
   def dict_iteritems(d):
     return d.items()
else:
   def dict_iteritems(d):
     return d.iteritems()

Clearly this is one of the simplest possible wrappers to write, but  
It's going to make code really ugly to have to use wrapper functions  
everywhere instead of methods. My point: is it *truly* necessary to  
completely remove .iteritems() in the same release that .items() gets  
iterator behavior? It would make every 3rd party developers' life so  
much easier if iteritems() stuck around a for another few releases  
after 3.X gains traction, and I really can't see any downside.

Of course, this is only one simple example, and if this was the  
_only_ incompatible change in Py3k, it would be non-ideal, but not a  
huge problem. But this same point goes for every single change. All  
the bidirectionally-incompatible changes put together may very well  
turn into one huge problem.

James

[1] Unless of course there's a perfect automated conversion script  
that can generate the 3.X compatible source code from the 2.X  
compatible source code. But since such a script is basically  
impossible, I doubt that's going to be the case. Feel free to prove  
me wrong here, of course. (but if it is possible, simply build it  
into python 3.0 so that it can convert old modules at load time and  
make everyone happy!)

[2] iter(d.items()) _does not count_. That is unacceptable on Python  
2.X due to the overhead of making the huge list.


From g.brandl at gmx.net  Thu Jan 11 20:32:41 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 11 Jan 2007 20:32:41 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>	<45A57AE0.2060600@benjiyork.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
Message-ID: <eo63cp$1ak$1@sea.gmane.org>

James Y Knight schrieb:

> If Python 3.0 was simply a release which removed deprecated features,  
> there would clearly be no issue. I would update my code in advance of  
> the 3.0 release to not use any of those features being removed, and  
> I'm all set. But that's not what I'm hearing. Python 3 is both adding  
> new ways to do things, and removing the older way, in the same  
> version, with no overlap. This makes me very anxious.

It has always been planned that in those cases that allow it, the new way to do
it will be introduced in a 2.x release too, and the old way removed only in 3.x.

Georg


From sluggoster at gmail.com  Thu Jan 11 21:22:50 2007
From: sluggoster at gmail.com (Mike Orr)
Date: Thu, 11 Jan 2007 12:22:50 -0800
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
	<45A57AE0.2060600@benjiyork.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
Message-ID: <6e9196d20701111222v49d70d0cw7cd00babf04d410a@mail.gmail.com>

On 1/11/07, James Y Knight <foom at fuhm.net> wrote:
> If the goal is really to have Py 3.0 be released later this year,

3.0 alpha is scheduled for this year.  3.0 final is not scheduled till
next year, and of course another level of tweaks will have to be made
after it's been in the Real World for a few months, so it won't fully
stabilize until 3.1 or 3.2.  That gives 1 1/2 - 2 years if you want to
target "production-level" 3.1.  Also, major parts of the 3.0 core have
not been written yet, so it remains to be seen how realistic the
schedule is (or how radical the changes will be if the schedule is
kept).

There will certainly be demand for an asynchronous server in 3.0,
along with the large set of protocols support in Twisted.  So either
Twisted will be ported or a new package will be written, possibly by
some of the Twisted developers, and very likely modelled after Twisted
even if the literal code can't be copied.

> Due to a lack of manpower, it is also not in all probability a
> realistic option to maintain two parallel almost-identical
> development branches,

So the two projects will operate independently, and the 3.0 one may be
smaller and less ambitious than Twisted.  But if the need is there it
will be written.

> Just, please don't gratuitously remove an old
> API to make Python a tiny bit cleaner, when there's not already
> something I can reasonably do *today* to replace it.

How did Perl 4 and Perl 5 handle the situation?  I basically waited
2-3 years after Perl 5 came out, then started programming the new way.
 If it mattered (it didn't), I would have tied my applications
specifically to Perl 4.

-- 
Mike Orr <sluggoster at gmail.com>

From steve at holdenweb.com  Thu Jan 11 22:24:57 2007
From: steve at holdenweb.com (Steve Holden)
Date: Thu, 11 Jan 2007 21:24:57 +0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <6e9196d20701111222v49d70d0cw7cd00babf04d410a@mail.gmail.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>	<45A57AE0.2060600@benjiyork.com>	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<6e9196d20701111222v49d70d0cw7cd00babf04d410a@mail.gmail.com>
Message-ID: <45A6AB29.4040604@holdenweb.com>

Mike Orr wrote:
[...]
> 
> How did Perl 4 and Perl 5 handle the situation?  I basically waited
> 2-3 years after Perl 5 came out, then started programming the new way.
>  If it mattered (it didn't), I would have tied my applications
> specifically to Perl 4.
> 
It's probably a good job you aren't holding your breath for Perl 6, 
though, eh? How *do* we make it easy for the authors of significant code 
bases to migrate to 3.0 *and* continue to support hte 2.X "market"?

These questions are all very important. I only wish I were serious 
enough to discuss them ;-).

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From arigo at tunes.org  Thu Jan 11 22:46:02 2007
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 11 Jan 2007 22:46:02 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
Message-ID: <20070111214601.GA10725@code0.codespeak.net>

Hi Paul,

On Wed, Jan 10, 2007 at 11:10:10PM +0000, Paul Moore wrote:
> How many other projects/packages anticipate *not* migrating to Py3K, I wonder?

FWIW: Psyco.


Armin

From g.brandl at gmx.net  Thu Jan 11 22:57:58 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 11 Jan 2007 22:57:58 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <20070111214601.GA10725@code0.codespeak.net>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
	<20070111214601.GA10725@code0.codespeak.net>
Message-ID: <eo6bt6$3oo$1@sea.gmane.org>

Armin Rigo schrieb:
> Hi Paul,
> 
> On Wed, Jan 10, 2007 at 11:10:10PM +0000, Paul Moore wrote:
>> How many other projects/packages anticipate *not* migrating to Py3K, I wonder?
> 
> FWIW: Psyco.

What will PyPy do? It will certainly support compiling Py3k code at some point,
but will the codebase itself be converted?

Georg


From guido at python.org  Thu Jan 11 23:54:09 2007
From: guido at python.org (Guido van Rossum)
Date: Thu, 11 Jan 2007 14:54:09 -0800
Subject: [Python-Dev] [Python-3000] Code working in both 2.x and 3.X
In-Reply-To: <eo6c90$5ea$1@sea.gmane.org>
References: <fb6fbf560701110812p71b12567y6baae509c9747681@mail.gmail.com>
	<eo6c90$5ea$1@sea.gmane.org>
Message-ID: <ca471dc20701111454m2954a817tb68adafed161a631@mail.gmail.com>

On 1/11/07, Ron Adam <rrr at ronadam.com> wrote:
> Jim Jewett wrote:
> > Raymond Hettinger wrote:
> >
> >> Also, I'm wondering if the desire for 2.6 warnings is based on the notion that
> >> it will be possible to get large tools to work under both Py2.x and Py3.x.
> >
> > I had certainly assumed it would be possible.
> >
> > In fact, I had assumed that the 2->3 translator would have a mode
> > which left the code running in both.
> >
> > This might not be the cleanest possible code for either line (parens
> > in 2.x, some extra iter calls or missed shortcuts, etc), but it should
> > certainly exist.  If it doesn't, then people who do want to support
> > both lines will themselves have to work exclusively in 2.x and
> > "compile" to 3.x.
> >
> >> With all the module renaming/packaging,
> [etc... clipped to shorten]
>
> There seems to be a fair amount of anxiety starting to appear about the
> difficulty of trying to support both 2.x and 3.x at the same time.  It's
> understandable, but I think it may be the earlier than expected 3.0 release date
> that is adding to that.
>
> Would it be reasonable to define X.0.X versions as official transition versions
> which need not have both back word compatibility, or overly strict future
> compatibility?
>
> I thinking that the 3.0.X version be considered a try it out (alpha) release to
> generate plenty of feed back, and the 3.1.X version be the first version meant
> for actual development use.
>
>     3.0.X <=> 3.alpha.X
>     3.1.X <=> 3.first.X
>
> So the release order may be 3.0.0, 2.6.0, 3.1.0. ...
>
> (With minor bug fix releases in between, of course.)
>
> My thinking is developers won't need to support the 3.0.X version *ever*.  And
> even 3.1.X need not be "too strongly" constrained to be back word compatible
> with version 3.0.X, as it may still have quite a few changes.  Version 3.1.0
> would then be the first version that future, 3.(1+n), versions will need to
> maintain back words compatibility with.
>
> This gives a lot more time for developers to try it, give feed back, and work
> out how to do things like move stuff from 2.X to 3.X, etc...  It also may give
> more freedom to make changes in version 3.0 with less anxiety to third party
> developers.
>
> Version 3.0  ->  a (relatively) clean start with big changes.
>
> Version 2.6  ->  may take into account things in version 3.0.
>
> Version 3.1  ->  builds on 3.0 and may take into account version 2.6

This last line is news to me, and I'm not in favor of it. My plans for
3.1 are different -- to take into account user feedback on 3.0.

> Isn't this the general (informal) context already being considered?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony at interlink.com.au  Fri Jan 12 02:12:18 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 12 Jan 2007 12:12:18 +1100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<45A57AE0.2060600@benjiyork.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
Message-ID: <200701121212.19187.anthony@interlink.com.au>

On Friday 12 January 2007 06:09, James Y Knight wrote:
> On Jan 10, 2007, at 6:46 PM, Benji York wrote:
> > Paul Moore wrote:
> >> How many other projects/packages anticipate *not* migrating to
> >> Py3K, I wonder?
> >
> > I certainly can't speak for the project as a whole, but I
> > anticipate a fair bit of work to port Zope 3 (100+ KLOC) to
> > Python 3.0.
>
> I (another Twisted developer, among other hats I wear) am also
> very worried about the Python 3.0 transition.

I'm plan to try and make the transition as painless as possible. A 
goal I have is to try and make it possible to write code that works 
in both 2.6 and 3.0. Obviously 2.6 will be backwards-compatible 
with previous versions, but I'd like to see it get either a command 
line option or a from __future__ statement to enable compatibility 
with 3.0-isms, like the dict.items change. 

> If the goal is really to have Py 3.0 be released later this year,

The goal is to have a first alpha sometime this year - there is 
absolutely no chance of a 3.0 final this year.

> Basically: my plea is: please don't remove the old way of doing
> things in Py 3.0 at the same time as you add the new way of doing
> things.

If there was a way to make 2.6 as compatible as possible with 3.0,
would this make life less painful? Obviously there'd have to be 
breakages in a backwards direction, but I'd hope it would make it 
easier to go forward. Some things should also be OK to backport to 
2.6 - for instance, I can't see an obvious reason why 2.6 can't 
support "except FooError as oopsie" as an alternate spelling of 
"except FooError, oopsie".

Similarly, where the stdlib has been shuffled around, there should 
be shims in 2.6 that allow people to work with the new names.

I don't think waiting for 2.7 to make the compatibility work is a 
workable approach - we're something like 2.5-3 years away from a 
2.7 release, and (optimistically) 12-18 months from a 3.0 final. 
That leaves a window of 1.5-2 years where we are missing an 
important tool for people.

> [1] Unless of course there's a perfect automated conversion
> script that can generate the 3.X compatible source code from the
> 2.X compatible source code. 

I doubt that the 2to3 script will be perfect, but hopefully it can 
get most things. I can't see it easily fixing up things like

check = mydict.has_key
...
if check(foo):

This is why I also want to add Py3kDeprecationWarning to 2.6. 

On that note, I'd like to see changes like the mass-change to the 
stdlib in the 3.0 branch that changed raise A, B into raise A(B) 
applied to the trunk. This makes it much easier to apply patches to 
both the 3.0 branch and the trunk. Similar changes should be 
applied to remove, for instance, use of <> and dict.has_key from 
the stdlib. Simply put, I'd like the stdlib between 2 and 3 to be 
as similar as possible.


-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From foom at fuhm.net  Fri Jan 12 06:12:34 2007
From: foom at fuhm.net (James Y Knight)
Date: Fri, 12 Jan 2007 00:12:34 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <200701121212.19187.anthony@interlink.com.au>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<45A57AE0.2060600@benjiyork.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<200701121212.19187.anthony@interlink.com.au>
Message-ID: <560D143B-136B-463E-BC87-3903D313D88D@fuhm.net>

On Jan 11, 2007, at 8:12 PM, Anthony Baxter wrote:
> I'm plan to try and make the transition as painless as possible.

I'm glad to hear it.

> The goal is to have a first alpha sometime this year - there is
> absolutely no chance of a 3.0 final this year.

Duly noted.

>> Basically: my plea is: please don't remove the old way of doing
>> things in Py 3.0 at the same time as you add the new way of doing
>> things.
>
> If there was a way to make 2.6 as compatible as possible with 3.0,
> would this make life less painful?

At the risk of repeating myself:

it's *all* about the overlap between a new way of doing things being  
added and the previous way being deleted. The overlap that's  
important here isn't necessarily version numbers, but *time*. Time- 
since-release correlates directly with the installed userbase of a  
version of python. I'm supposing the plan is for a nearly- 
simultaneous 2.6 & 3.0 release (true?). Then, it doesn't really even  
matter if 2.6 achieves perfect forward compatibility with 3.0, it's  
too late. 3.0 is out, and if I want to be compatible with it, I'd  
then have to then require 2.6+. But as 2.6 was just released, I can't  
require it: most people won't have it installed, and abandoning those  
users is simply not an option. I need *time* between the addition of  
the new API and the deletion of the old.

When Python 3.0 is released, unless it comes with prominent warnings  
saying "nobody should actually use this, don't try to make your code  
run on it, if you run this program all your friends will laugh at  
you.", early adopter users are going to want to use 3rd party  
software with Python 3.0. If my only choice at that point is to tell  
these people "sorry, no, it's simply impossible to support Python 3.0  
with a reasonable amount of effort, while still keeping compatible  
with older Pythons that most users still have", that is not a happy  
situation. I'd like to avoid this situation.

Of course, if 2.6 really is going to be practically identical to 3.0,  
except that 3.0 will have removed a bunch of functionality, well,  
then there's absolutely no reason for *anyone* to use 3.0: the only  
major difference is that nothing can run on it. If that's how things  
will be, is there even any point in releasing 3.0? What good is it?

> Obviously there'd have to be breakages in a backwards direction,

I fail to see why this is obvious. Can't 3.0 be content with only  
removing syntax/features/APIs which can already be replaced by using  
a newer, better thing that already exists in Python 2.4? There's  
plenty of that to do! Py 3.0 can also deprecate a whole new pile of  
stuff to be removed in 3.2, and add a bunch of awesome new features.  
That's great, no objection. But *some* time between addition of new  
feature and removal of old feature, please?

> I don't think waiting for 2.7 to make the compatibility work is a
> workable approach -

Well, yes, that would, of course, be even worse.

> On that note, I'd like to see changes like the mass-change to the
> stdlib in the 3.0 branch that changed raise A, B into raise A(B)
> applied to the trunk. This makes it much easier to apply patches to
> both the 3.0 branch and the trunk. Similar changes should be
> applied to remove, for instance, use of <> and dict.has_key from
> the stdlib. Simply put, I'd like the stdlib between 2 and 3 to be
> as similar as possible.

This suggests an intriguing idea: require all the high level modules  
in the stdlib in python 3.0 to continue to be compatible with Python  
2.5. If the developers of python are forced to make their own code  
also work with both 2.5 and 3.0, I think any poorly conceived  
bidirectionally-incompatible changes will be quickly rejected. :)

James



From anthony at interlink.com.au  Fri Jan 12 08:56:25 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 12 Jan 2007 18:56:25 +1100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
References: <200701081650.17353.anthony@interlink.com.au>
	<200701101524.06515.anthony@interlink.com.au>
	<00ee01c734e3$0f1e4110$ea146b0a@RaymondLaptop1>
Message-ID: <200701121856.28557.anthony@interlink.com.au>

On Thursday 11 January 2007 05:13, Raymond Hettinger wrote:
> It is my strong preference that we not go down this path.
> Instead, the 2.6 vs 3.0 difference analysis should go in an
> external lint utility.

I don't see how an external utility can possibly catch everything 
that's going to break - it's going to be much much easier to catch 
things like use of has_key and the like from inside the 
interpreter. Additionally, catching things like C code extensions 
using a slot or a method or a function that's going away is just 
not going to be possible otherwise.

> The Py2.x series may live-on for some time and should do so
> as if Py3.x did not exist.  Burdening the 2.x code with loads
> of warnings will only clutter the source code and make
> maintenance more difficult.  There may also be some performance
> impact.

If it's a single check of a C int, the performance impact will be 
minimal. I agree that 2.x will live for a long time - but unless we 
provide the best possible upgrade path, we will be stuck 
maintaining both 2.x and 3.x for ever.

Additionally, without a 2.x<->3.x upgrade path 3.x is essentially a 
new language, having to build a new userbase from scratch. Worse 
yet, 2.x will suffer as people have the perception "Python 2? 
That's a dead/abandoned language"

> We should resolve that Py2.6 remain as clean as possible
> and that Py3.0 be kept in its own world.  Forging a new
> blade does not have to entail dulling the trusty old blade.

I completely disagree here. We cannot simply ignore 3.0 in the 2.x 
series. We need to provide (as much as possible) an upgrade path 
for people who write and use code in the language.

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From martin at v.loewis.de  Fri Jan 12 09:19:52 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 12 Jan 2007 09:19:52 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <eo63cp$1ak$1@sea.gmane.org>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>	<45A57AE0.2060600@benjiyork.com>	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<eo63cp$1ak$1@sea.gmane.org>
Message-ID: <45A744A8.40707@v.loewis.de>

Georg Brandl schrieb:
>> If Python 3.0 was simply a release which removed deprecated features,  
>> there would clearly be no issue. I would update my code in advance of  
>> the 3.0 release to not use any of those features being removed, and  
>> I'm all set. But that's not what I'm hearing. Python 3 is both adding  
>> new ways to do things, and removing the older way, in the same  
>> version, with no overlap. This makes me very anxious.
> 
> It has always been planned that in those cases that allow it, the new way to do
> it will be introduced in a 2.x release too, and the old way removed only in 3.x.

What does that mean for the example James gave: if dict.items is going
to be an iterator in 3.0, what 2.x version can make it return an
iterator, when it currently returns a list?

There simply can't be a 2.x version that *introduces* the new way, as it
is not merely a new API, but a changed API.

Regards,
Martin

From anthony at interlink.com.au  Fri Jan 12 10:03:55 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 12 Jan 2007 20:03:55 +1100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <45A744A8.40707@v.loewis.de>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<eo63cp$1ak$1@sea.gmane.org> <45A744A8.40707@v.loewis.de>
Message-ID: <200701122003.56210.anthony@interlink.com.au>

On Friday 12 January 2007 19:19, Martin v. L?wis wrote:
> Georg Brandl schrieb:
> > It has always been planned that in those cases that allow it,
> > the new way to do it will be introduced in a 2.x release too,
> > and the old way removed only in 3.x.
>
> What does that mean for the example James gave: if dict.items is
> going to be an iterator in 3.0, what 2.x version can make it
> return an iterator, when it currently returns a list?
>
> There simply can't be a 2.x version that *introduces* the new
> way, as it is not merely a new API, but a changed API.

There's a couple of ways I see it - we could add a "-3" command line 
flag to enable 3.x compat, or maybe a from __future__ statement. 
Although the latter would be a global thing, which is different to 
how all existing from __future__s work, so probably not good.

I don't see a path forward that doesn't involve something painful, 
so long as 3.0 is going to be the clean break. As I mentioned, 
though, I'd like as far as possible to make it so that 2.6 (with a 
flag) can be at least vaguely compatible with 3.0.

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From mwh at python.net  Fri Jan 12 11:12:21 2007
From: mwh at python.net (Michael Hudson)
Date: Fri, 12 Jan 2007 11:12:21 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <eo6bt6$3oo$1@sea.gmane.org> (Georg Brandl's message of "Thu,
	11 Jan 2007 22:57:58 +0100")
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
	<20070111214601.GA10725@code0.codespeak.net>
	<eo6bt6$3oo$1@sea.gmane.org>
Message-ID: <8764bcmu4a.fsf@starship.python.net>

Georg Brandl <g.brandl at gmx.net> writes:

> Armin Rigo schrieb:
>> Hi Paul,
>> 
>> On Wed, Jan 10, 2007 at 11:10:10PM +0000, Paul Moore wrote:
>>> How many other projects/packages anticipate *not* migrating to Py3K, I wonder?
>> 
>> FWIW: Psyco.
>
> What will PyPy do? It will certainly support compiling Py3k code at some point,
> but will the codebase itself be converted?

For practical reasons (we have enough work to be getting on with) PyPy
is more-or-less ignoring Python 2.5 at the moment.  After funding and
so on, when there's less pressure, maybe it will seem worth it.  Not
soon though.

Cheers,
mwh

-- 
  I'm okay with intellegent buildings, I'm okay with non-sentient
  buildings. I have serious reservations about stupid buildings.
     -- Dan Sheppard, ucam.chat (from Owen Dunn's summary of the year)

From glyph at divmod.com  Fri Jan 12 11:12:36 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 12 Jan 2007 10:12:36 -0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Message-ID: <20070112101236.17094.1688023152.divmod.xquotient.1279@joule.divmod.com>

On 11 Jan, 08:22 pm, sluggoster at gmail.com wrote:
>On 1/11/07, James Y Knight <foom at fuhm.net> wrote:
>> If the goal is really to have Py 3.0 be released later this year,

>There will certainly be demand for an asynchronous server in 3.0,

To flip the question around: there might be a demand for Twisted in 3.0, but will there be a demand for 3.0 in Twisted?  It might just be easier for everyone concerned to just continue maintaining 2.x forever.  I have yet to see a reason why, other than continued maintenance, 3.0 would be a preferable development platform.

>So the two projects will operate independently, and the 3.0 one may be
>smaller and less ambitious than Twisted.  But if the need is there it
>will be written.

It is quite likely that someone else will write some completely different code for python 3.0 that calls select().  I hadn't considered that the goal of 3.0 was to *discover* these people by alienating existing Python developers - that's crafty!  If so, though, you'll have to figure out a way to stop Anthony from providing all this compatibility stuff.  He might make it too attractive for us to continue development on future versions :).

>How did Perl 4 and Perl 5 handle the situation?  I basically waited
>2-3 years after Perl 5 came out, then started programming the new way.
> If it mattered (it didn't), I would have tied my applications
>specifically to Perl 4.

I handled the Perl 4 to 5 transition by dropping Perl and moving to Python, because if I was going to port all my code to another language I wanted to at least port to a better language.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070112/a7c20083/attachment-0001.htm 

From glyph at divmod.com  Fri Jan 12 11:24:57 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 12 Jan 2007 10:24:57 -0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Message-ID: <20070112102457.17094.1135744655.divmod.xquotient.1303@joule.divmod.com>

On 01:12 am, anthony at interlink.com.au wrote:
>On Friday 12 January 2007 06:09, James Y Knight wrote:
>> On Jan 10, 2007, at 6:46 PM, Benji York wrote:
>> > Paul Moore wrote:
>> >> How many other projects/packages anticipate *not* migrating to
>> >> Py3K, I wonder?
>> >
>> > I certainly can't speak for the project as a whole, but I
>> > anticipate a fair bit of work to port Zope 3 (100+ KLOC) to
>> > Python 3.0.
>>
>> I (another Twisted developer, among other hats I wear) am also
>> very worried about the Python 3.0 transition.
>
>I'm plan to try and make the transition as painless as possible. A
>goal I have is to try and make it possible to write code that works
>in both 2.6 and 3.0. Obviously 2.6 will be backwards-compatible
>with previous versions, but I'd like to see it get either a command
>line option or a from __future__ statement to enable compatibility
>with 3.0-isms, like the dict.items change.

This is all very encouraging.

Please stick with a __future__ statement if at all possible though.  The biggest challenge in migration is to reduce the impact so that it can be done partially in a real system.

If you have a module X wants to be "3.0 safe" which imports a module Y from a different developer that is not, command line options for compatibility might not help at all.  A __future__ statment in X but not in Y, though, would allow for a smooth transition.

I can see how that would be tricky for things like dictionary methods, but it does seem doable.  For example, have a has_key descriptor which can raise an AttributeError if the globals() of the calling stack frame has been marked in some way.

>> Basically: my plea is: please don't remove the old way of doing
>> things in Py 3.0 at the same time as you add the new way of doing
>> things.
>
>If there was a way to make 2.6 as compatible as possible with 3.0,
>would this make life less painful? Obviously there'd have to be
>breakages in a backwards direction, but I'd hope it would make it
>easier to go forward. Some things should also be OK to backport to
>2.6 - for instance, I can't see an obvious reason why 2.6 can't
>support "except FooError as oopsie" as an alternate spelling of
>"except FooError, oopsie".

You happen to have hit my favorite hot-button 3.0 issue right there :).  I don't care about backticks but sometimes I _do_ have to catch exceptions.

>Similarly, where the stdlib has been shuffled around, there should
>be shims in 2.6 that allow people to work with the new names.

This part I wouldn't even mind having to write myself.  It would certainly be good to have somewhere more official though.

>I don't think waiting for 2.7 to make the compatibility work is a
>workable approach - we're something like 2.5-3 years away from a
>2.7 release, and (optimistically) 12-18 months from a 3.0 final.
>That leaves a window of 1.5-2 years where we are missing an
>important tool for people.

It would be nice if the versioning would actually happen in order.

>> [1] Unless of course there's a perfect automated conversion
>> script that can generate the 3.X compatible source code from the
>> 2.X compatible source code.
>
>I doubt that the 2to3 script will be perfect, but hopefully it can
>get most things. I can't see it easily fixing up things like
>
>check = mydict.has_key
>...
>if check(foo):

Currently a common idiom, by the way, used throughout the lower levels of Twisted.

>This is why I also want to add Py3kDeprecationWarning to 2.6.
>
>On that note, I'd like to see changes like the mass-change to the
>stdlib in the 3.0 branch that changed raise A, B into raise A(B)
>applied to the trunk. This makes it much easier to apply patches to
>both the 3.0 branch and the trunk. Similar changes should be
>applied to remove, for instance, use of <> and dict.has_key from
>the stdlib. Simply put, I'd like the stdlib between 2 and 3 to be
>as similar as possible.

It would be nice if the stdlib could be used as a case study - if the 3 stdlib tests can pass on some version of 2 (or vice versa) that should be a minimum bar for application portability.

Maybe a better way to handle the similarity is that the reorganized stdlib should simply be available as a separate piece of code on 2.x?  That would allow 2.x to use any new features added and distinguish between code which had been moved to use new stdlib APIs and that which hadn't.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070112/ec920900/attachment.htm 

From glyph at divmod.com  Fri Jan 12 11:42:25 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 12 Jan 2007 10:42:25 -0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Message-ID: <20070112104225.17094.763678682.divmod.xquotient.1329@joule.divmod.com>

On 07:56 am, anthony at interlink.com.au wrote:

>Additionally, without a 2.x<->3.x upgrade path 3.x is essentially a
>new language, having to build a new userbase from scratch. Worse
>yet, 2.x will suffer as people have the perception "Python 2?
>That's a dead/abandoned language"

It's worse than that.  This perception has _already_ been created.  I already have heard a few folks looking for new languages to learn choose Ruby over Python and give Py3K as a reason.  "Isn't Python going to be totally different in a few years anyway?  I'll just wait until then, seems like a waste of time to learn it now."

Given Ruby's own checkered history of compatibility, I don't think this is an _accurate_ perception, but it is the nature of perception to be inaccurate.

If the plan is to provide a smooth transition, it would help a lot to have this plan of foward and backward compatibility documented somewhere very public.  It's hard to find information on Py3K right now, even if you know your way around the universe of PEPs.

FWIW, I also agree with James that Python 3 shouldn't even be released until the 2.x series has reached parity with its feature set.  However, if there's continuity in the version numbers instead of the release dates, I can at least explain to Twisted users that we will _pretend_ they are released in the order of their versions.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070112/2e57c8d4/attachment.html 

From glyph at divmod.com  Fri Jan 12 11:56:21 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 12 Jan 2007 10:56:21 -0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Message-ID: <20070112105621.17094.642619803.divmod.xquotient.1347@joule.divmod.com>

On 10:12 am, mwh at python.net wrote:

>For practical reasons (we have enough work to be getting on with) PyPy
>is more-or-less ignoring Python 2.5 at the moment.  After funding and
>so on, when there's less pressure, maybe it will seem worth it.  Not
>soon though.

I think I know what you mean from previous conversations, but the context of the question makes the answer ambiguous.  Are you ignoring 2.5 in favor of 2.4, or 3.0?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070112/a40df02b/attachment.htm 

From anthony at interlink.com.au  Fri Jan 12 12:22:51 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 12 Jan 2007 22:22:51 +1100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <20070112104225.17094.763678682.divmod.xquotient.1329@joule.divmod.com>
References: <20070112104225.17094.763678682.divmod.xquotient.1329@joule.divmod.com>
Message-ID: <200701122222.53885.anthony@interlink.com.au>

On Friday 12 January 2007 21:42, glyph at divmod.com wrote:
> If the plan is to provide a smooth transition, it would help a
> lot to have this plan of foward and backward compatibility
> documented somewhere very public.  It's hard to find information
> on Py3K right now, even if you know your way around the universe
> of PEPs.

I'd like to see this happen, too - however, there's no way I can 
even think about it until after LCA next week. First of all, of 
course, we need to get agreement on the preferred way forward.

> FWIW, I also agree with James that Python 3 shouldn't even be
> released until the 2.x series has reached parity with its feature
> set.  However, if there's continuity in the version numbers
> instead of the release dates, I can at least explain to Twisted
> users that we will _pretend_ they are released in the order of
> their versions.

I'm not sure what "parity with it's feature set" means. I think 
there's going to be some 3.0isms that just cannot be done sanely in 
2.x - for instance, the new I/O subsystem. But I do hope that it's 
_possible_ to work in a version of the language that works in both 
2.6+ and 3.0+, even if under the hood there are differences. For 
instance, if we did "except foo as bar" for 2.6, it might not 
auto-clean-up the exception object when it drops out of the except: 
block. 

I put up www.python.org/sf/1633807 a short time ago that deals with 
one of the big concerns I had - print vs print() (it was also as a 
learning exercise to figure out if it was possible, and how it 
might work). Something similar could probably be done for exec(). I 
suspect the problem cases are going to be things like the 
dictionary code - your idea (in another email) of trying to look up 
globals would probably cause a horrible performance issue, but it 
may be possible to do _something_ clever. 

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From glyph at divmod.com  Fri Jan 12 12:59:42 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 12 Jan 2007 11:59:42 -0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Message-ID: <20070112115942.17094.556946925.divmod.xquotient.1362@joule.divmod.com>

On 11:22 am, anthony at interlink.com.au wrote:

>> FWIW, I also agree with James that Python 3 shouldn't even be
>> released until the 2.x series has reached parity with its feature
>> set.  However, if there's continuity in the version numbers
>> instead of the release dates, I can at least explain to Twisted
>> users that we will _pretend_ they are released in the order of
>> their versions.
>
>I'm not sure what "parity with it's feature set" means.

I don't either!  "Parity with it is feature set"?  I can't even parse that!  ;-)

By "parity with *its* feature set", though, I meant what you said here:

>I do hope that it's _possible_ to work in a version of the language that works in both 2.6+ and 3.0+, even if under the hood there are differences.

In order to do this, everything that has been changed in 3.0 has to have some mechanism for working both ways in some 2.x release.  I phrased this as its "feature set" because I am not aware of any new functionality in 3.0 that simply isn't available 2.x - everything I've seen are "cleanups", which expose basically the same features as 2.5.

If there are some real new features, then I am in error.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070112/311faa55/attachment.htm 

From g.brandl at gmx.net  Fri Jan 12 15:17:25 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 12 Jan 2007 15:17:25 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <45A744A8.40707@v.loewis.de>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>	<45A57AE0.2060600@benjiyork.com>	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>	<eo63cp$1ak$1@sea.gmane.org>
	<45A744A8.40707@v.loewis.de>
Message-ID: <eo859h$fsa$1@sea.gmane.org>

Martin v. L?wis schrieb:
> Georg Brandl schrieb:
>>> If Python 3.0 was simply a release which removed deprecated features,  
>>> there would clearly be no issue. I would update my code in advance of  
>>> the 3.0 release to not use any of those features being removed, and  
>>> I'm all set. But that's not what I'm hearing. Python 3 is both adding  
>>> new ways to do things, and removing the older way, in the same  
>>> version, with no overlap. This makes me very anxious.
>> 
>> It has always been planned that in those cases that allow it, the new way to do
>> it will be introduced in a 2.x release too, and the old way removed only in 3.x.
> 
> What does that mean for the example James gave: if dict.items is going
> to be an iterator in 3.0, what 2.x version can make it return an
> iterator, when it currently returns a list?
> 
> There simply can't be a 2.x version that *introduces* the new way, as it
> is not merely a new API, but a changed API.

Well, that is one of the cases in which that won't be possible ;)

But e.g. moved methods, libraries or new syntax are areas where changes can
be introduced in 2.x (in case of new syntax with a __future__ statement,
of course).

Georg


From g.brandl at gmx.net  Fri Jan 12 16:10:24 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 12 Jan 2007 16:10:24 +0100
Subject: [Python-Dev] trunk changes not suited for 3.0
Message-ID: <eo88d0$shg$1@sea.gmane.org>

Hi,

currently, changes in the trunk are merged to the p3yk branch.
What about changes such as the discussed -3 warn option, or __future__
imports for stuff that'll be in 3k anyway?

Can/should these revisions just be excluded from the merge? Or would
it be better to apply them to a new branch?

Georg


From guido at python.org  Fri Jan 12 16:13:37 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 12 Jan 2007 07:13:37 -0800
Subject: [Python-Dev] trunk changes not suited for 3.0
In-Reply-To: <eo88d0$shg$1@sea.gmane.org>
References: <eo88d0$shg$1@sea.gmane.org>
Message-ID: <ca471dc20701120713n42ab905x78297a8b27150e6@mail.gmail.com>

AFAIK Thomas just excludes these from the merge, or occasionally (if
they make it through) they will be eradicated in the p3yk branch, and
then they'll be excluded from future merges of related changes (since
those would be flagged as conflict).

On 1/12/07, Georg Brandl <g.brandl at gmx.net> wrote:
> Hi,
>
> currently, changes in the trunk are merged to the p3yk branch.
> What about changes such as the discussed -3 warn option, or __future__
> imports for stuff that'll be in 3k anyway?
>
> Can/should these revisions just be excluded from the merge? Or would
> it be better to apply them to a new branch?
>
> Georg
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From barry at python.org  Fri Jan 12 16:14:57 2007
From: barry at python.org (Barry Warsaw)
Date: Fri, 12 Jan 2007 10:14:57 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <eo859h$fsa$1@sea.gmane.org>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>	<45A57AE0.2060600@benjiyork.com>	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>	<eo63cp$1ak$1@sea.gmane.org>
	<45A744A8.40707@v.loewis.de> <eo859h$fsa$1@sea.gmane.org>
Message-ID: <9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 12, 2007, at 9:17 AM, Georg Brandl wrote:

> Well, that is one of the cases in which that won't be possible ;)

I think there will be at least three areas that will make porting a  
challenge:

- - APIs where the semantics have changed instead of being moved,  
renamed, or extended
- - C API changes for extensions and embedded apps
- - Unicode/str/bytes incompatibilities

It's worth spending time thinking about how we can help ease the  
transition for each.  There may be more thing too.  I wonder if it  
doesn't make sense for all 3xxx PEPs to include a discussion section  
on porting.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRael9nEjvBPtnXfVAQIqRQP+PrYnz/XSnGsplGj1+/MtBFzkbKeZEEn7
tN1NyKNZ7Rt7w+K9JhO9D1XismB3Wwwd66d7+N8vY/y/ea/cMosUwYS7LbwTmvcV
MX3HeeqSfzjseuaTE2/3Mj+HHTWCE6dWASG5CHOguqs84FPeA5b+tbNNcp0Prp6I
HnkfO6d2V/I=
=TphK
-----END PGP SIGNATURE-----

From thomas at python.org  Fri Jan 12 16:54:03 2007
From: thomas at python.org (Thomas Wouters)
Date: Fri, 12 Jan 2007 07:54:03 -0800
Subject: [Python-Dev] trunk changes not suited for 3.0
In-Reply-To: <ca471dc20701120713n42ab905x78297a8b27150e6@mail.gmail.com>
References: <eo88d0$shg$1@sea.gmane.org>
	<ca471dc20701120713n42ab905x78297a8b27150e6@mail.gmail.com>
Message-ID: <9e804ac0701120754n3ba048dck8ff3087f0f687af6@mail.gmail.com>

On 1/12/07, Guido van Rossum <guido at python.org> wrote:
>
> AFAIK Thomas just excludes these from the merge,


Indeed I would.

or occasionally (if
> they make it through) they will be eradicated in the p3yk branch, and
> then they'll be excluded from future merges of related changes (since
> those would be flagged as conflict).
>
> On 1/12/07, Georg Brandl <g.brandl at gmx.net> wrote:
> > Hi,
> >
> > currently, changes in the trunk are merged to the p3yk branch.
> > What about changes such as the discussed -3 warn option, or __future__
> > imports for stuff that'll be in 3k anyway?
> >
> > Can/should these revisions just be excluded from the merge? Or would
> > it be better to apply them to a new branch?
> >
> > Georg
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/thomas%40python.org
>



-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070112/e0532844/attachment.html 

From facundo at taniquetil.com.ar  Fri Jan 12 16:42:38 2007
From: facundo at taniquetil.com.ar (Facundo Batista)
Date: Fri, 12 Jan 2007 15:42:38 +0000 (UTC)
Subject: [Python-Dev] Code working in both 2.x and 3.X
References: <fb6fbf560701110812p71b12567y6baae509c9747681@mail.gmail.com>
	<eo6c90$5ea$1@sea.gmane.org>
Message-ID: <eo8a9d$4a2$1@sea.gmane.org>

Ron Adam wrote:

> I thinking that the 3.0.X version be considered a try it out (alpha) release to 
> generate plenty of feed back, and the 3.1.X version be the first version meant 
> for actual development use.

+1 for this approach.

I think it's very clear, and everybody will understand it (ok, not
everybody, we know people...).

-- 
.   Facundo
.
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/



From amk at amk.ca  Fri Jan 12 17:59:32 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 12 Jan 2007 11:59:32 -0500
Subject: [Python-Dev] The bytes type (was Re: Warning for 2.6 and greater)
In-Reply-To: <9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
	<45A57AE0.2060600@benjiyork.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<eo63cp$1ak$1@sea.gmane.org> <45A744A8.40707@v.loewis.de>
	<eo859h$fsa$1@sea.gmane.org>
	<9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>
Message-ID: <20070112165932.GA11501@localhost.localdomain>

On Fri, Jan 12, 2007 at 10:14:57AM -0500, Barry Warsaw wrote:
> I think there will be at least three areas that will make porting a  
> challenge:
   ...
> - - Unicode/str/bytes incompatibilities

Something I've been meaning to bring up... do we know what shape the
Unicode/str/bytes resolution will take?  It would be nice to know so
that similar changes can be made to Jython.  For example, str in
Jython is a Unicode type, but things like the MD5 module should really
only be dealing with 8-bit strings.

Is PEP 358 still the current thinking on the bytes type?  If yes,
should we set about implementing it for Jython and Python 2.6?

--amk


From pje at telecommunity.com  Fri Jan 12 18:18:27 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 12 Jan 2007 12:18:27 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <20070112115942.17094.556946925.divmod.xquotient.1362@joule
	.divmod.com>
Message-ID: <5.1.1.6.0.20070112121751.0287ce38@sparrow.telecommunity.com>

At 11:59 AM 1/12/2007 +0000, glyph at divmod.com wrote:
>In order to do this, everything that has been changed in 3.0 has to have 
>some mechanism for working both ways in some 2.x release.  I phrased this 
>as its "feature set" because I am not aware of any new functionality in 
>3.0 that simply isn't available 2.x - everything I've seen are "cleanups", 
>which expose basically the same features as 2.5.
>
>If there are some real new features, then I am in error.

If I understand correctly, argument annotations, and exceptions having 
traceback attributes are new features.



From amk at amk.ca  Fri Jan 12 18:19:19 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 12 Jan 2007 12:19:19 -0500
Subject: [Python-Dev] file(file)
Message-ID: <20070112171919.GA14045@localhost.localdomain>

Forwarded for discussion from http://www.python.org/sf/1633665.

--amk

[forwarded from http://bugs.debian.org/327060]

Many types in Python are idempotent, so that int(1) works
as expected, float(2.34)==2.34, ''.join('hello')=='hello'
et cetera.

Why not file()? Currently, file(open(something, 'r')) fails
with "TypeError: coercing to Unicode: need string or buffer, file found."

Semantically, file(fd) should be equivalent to os.fdopen(fd.fileno())
or the proposed file.fromfd() (Jp Calderone, Python-dev, 2003).
You should get another independent
file object that accesses the same file.

What would be gained?

Primarily, it would allow you to derive classes from file more easily.
At present, if you want to derive like so, you're class can only work
when passed a file name or buffer.

class file_with_caching(file):
    def __init__(self, something):
        file.__init__(self, something)

def etcetera...

For instance, you have no way of creating a file_with_caching()
object from the file descriptors returned from os.fork().

Also, you have no way of taking a file that is already open,
and creating a file_with_caching() object from it. So,
you can't use classes derived from file() on the standard input
or standard output.

This breaks the nice Linux OS-level definition of a file descriptor.
At the Linux level, you have a nice uniform interface where all
file descriptors are equally good. At the python level, some
are better than others. It's a case where Python unnecessarily
restricts what you can do.

From guido at python.org  Fri Jan 12 18:35:51 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 12 Jan 2007 09:35:51 -0800
Subject: [Python-Dev] Code working in both 2.x and 3.X
In-Reply-To: <eo8a9d$4a2$1@sea.gmane.org>
References: <fb6fbf560701110812p71b12567y6baae509c9747681@mail.gmail.com>
	<eo6c90$5ea$1@sea.gmane.org> <eo8a9d$4a2$1@sea.gmane.org>
Message-ID: <ca471dc20701120935s73da4c89wa12135845e7729dd@mail.gmail.com>

On 1/12/07, Facundo Batista <facundo at taniquetil.com.ar> wrote:
> Ron Adam wrote:
>
> > I thinking that the 3.0.X version be considered a try it out (alpha) release to
> > generate plenty of feed back, and the 3.1.X version be the first version meant
> > for actual development use.
>
> +1 for this approach.
>
> I think it's very clear, and everybody will understand it (ok, not
> everybody, we know people...).

To the contrary, I think this is just a game with semantics -- if we
tell people not to use 3.0 because it is still alpha quality then we
won't get *actual* feedback until we release 3.1, and we'd still have
to do a 3.2 which takes the feedback from serious users into account.

IMO we should strive for 3.0 to be the best release we can make it --
on a par with 2.4.0, 2.5.0 and 2.6.0. I want about a year between
3.0a1 (to be released in the first half of this year) and 3.0final.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Fri Jan 12 18:51:25 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 12 Jan 2007 09:51:25 -0800
Subject: [Python-Dev] The bytes type (was Re: Warning for 2.6 and
	greater)
In-Reply-To: <20070112165932.GA11501@localhost.localdomain>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
	<45A57AE0.2060600@benjiyork.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<eo63cp$1ak$1@sea.gmane.org> <45A744A8.40707@v.loewis.de>
	<eo859h$fsa$1@sea.gmane.org>
	<9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>
	<20070112165932.GA11501@localhost.localdomain>
Message-ID: <ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>

On 1/12/07, A.M. Kuchling <amk at amk.ca> wrote:
> On Fri, Jan 12, 2007 at 10:14:57AM -0500, Barry Warsaw wrote:
> > I think there will be at least three areas that will make porting a
> > challenge:
>    ...
> > - - Unicode/str/bytes incompatibilities
>
> Something I've been meaning to bring up... do we know what shape the
> Unicode/str/bytes resolution will take?  It would be nice to know so
> that similar changes can be made to Jython.  For example, str in
> Jython is a Unicode type, but things like the MD5 module should really
> only be dealing with 8-bit strings.

This hasn't been sorted out in much detail, alas. I expect we'll have
some tough decisions to make about which APIs use bytes and which use
str. (For another example, considere base64.)

> Is PEP 358 still the current thinking on the bytes type?  If yes,
> should we set about implementing it for Jython and Python 2.6?

I'm afraid that PEP is not up to date; I don't think I used it as a
reference when I coded up the current bytes type in Py3k. Whenever the
PEP matches the implementation, we can be confident that we have the
right design. Where they differ, I'd be inclined to believe the code,
unless it's a matter of an as-yet-unimplemented feature, or a
temporary measure while we still have the 8-bit str type.

Some examples of differences:

- bytes(100) returns a bytes array of length 100 filled with zeros.

- repr() of a bytes array uses hex notation instead of decimal.

- .fromhex() and .hex() are not implemented.

- .append(), .insert(), .remove(), .pop(), .extend(), .index(),
.count() are not implemented.

- bytes has .decode() and .join() methods that are not in the PEP.

In general I believe the actual bytes object resembles strings more
than lists, and probably will grow more string-like behavior instead
of more list methods. (Case in point: 'in' implements a substring
check like strings do.)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Fri Jan 12 19:01:51 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 12 Jan 2007 10:01:51 -0800
Subject: [Python-Dev] file(file)
In-Reply-To: <20070112171919.GA14045@localhost.localdomain>
References: <20070112171919.GA14045@localhost.localdomain>
Message-ID: <ca471dc20701121001x2d1f1ea0h6266e8aa60ed56f2@mail.gmail.com>

On 1/12/07, A.M. Kuchling <amk at amk.ca> wrote:
> Forwarded for discussion from http://www.python.org/sf/1633665.
>
> --amk
>
> [forwarded from http://bugs.debian.org/327060]
>
> Many types in Python are idempotent, so that int(1) works
> as expected, float(2.34)==2.34, ''.join('hello')=='hello'
> et cetera.

The join(0 example seems out of place, since it is not a constructor call.

> Why not file()? Currently, file(open(something, 'r')) fails
> with "TypeError: coercing to Unicode: need string or buffer, file found."
>
> Semantically, file(fd) should be equivalent to os.fdopen(fd.fileno())
> or the proposed file.fromfd() (Jp Calderone, Python-dev, 2003).
> You should get another independent
> file object that accesses the same file.

That's rather Unix-centric semantics. I don't think the same thing
exists on Windows.

We should also consider the semantics in more detail. Should the seek
position be shared between the two objects? What about buffering?

I'm not sure I understand the use case; I don't believe I've ever felt
the need for this.

> What would be gained?
>
> Primarily, it would allow you to derive classes from file more easily.

Inheritance is overrated.

> At present, if you want to derive like so, you're class can only work
> when passed a file name or buffer.
>
> class file_with_caching(file):
>     def __init__(self, something):
>         file.__init__(self, something)
>
> def etcetera...

But the devil is in the details. What are the "caching" semantics you
want to implement here? Is it appropriate to subclass file for that
purpose, or is it better to use a wrapper (or proxy or delegate)?

> For instance, you have no way of creating a file_with_caching()
> object from the file descriptors returned from os.fork().

I didn't know fork() returned a file descriptor.

> Also, you have no way of taking a file that is already open,
> and creating a file_with_caching() object from it. So,
> you can't use classes derived from file() on the standard input
> or standard output.

More hints that you shouldn't be using inheritance.

> This breaks the nice Linux OS-level definition of a file descriptor.

But a Python file is not a Unix file descriptor. It's more like a C stdio file.

> At the Linux level, you have a nice uniform interface where all
> file descriptors are equally good. At the python level, some
> are better than others. It's a case where Python unnecessarily
> restricts what you can do.

That's what you say. I see a lot of murky thinking.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From amk at amk.ca  Fri Jan 12 19:22:48 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 12 Jan 2007 13:22:48 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
	<45A57AE0.2060600@benjiyork.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<eo63cp$1ak$1@sea.gmane.org> <45A744A8.40707@v.loewis.de>
	<eo859h$fsa$1@sea.gmane.org>
	<9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>
	<20070112165932.GA11501@localhost.localdomain>
	<ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>
Message-ID: <20070112182248.GA15187@localhost.localdomain>

On Fri, Jan 12, 2007 at 09:51:25AM -0800, Guido van Rossum wrote:
> I'm afraid that PEP is not up to date; I don't think I used it as a
> reference when I coded up the current bytes type in Py3k. Whenever the
> PEP matches the implementation, we can be confident that we have the
> right design. Where they differ, I'd be inclined to believe the code,
> unless it's a matter of an as-yet-unimplemented feature, or a
> temporary measure while we still have the 8-bit str type.

Is the Py3k bytes type considered 'finished', then, or at least
'largely correct', or is it still experimental?  I could backport the
type to the 2.x branch; it could be a new feature in 2.6 and then be
added to Jython.

2.6 wouldn't go changing existing APIs to begin requiring or returning
the bytes type[*], of course, but extensions and new modules might use
it.

--amk

[*] Anyone else keep wanting to write "byte type"?


From python at rcn.com  Fri Jan 12 19:49:13 2007
From: python at rcn.com (Raymond Hettinger)
Date: Fri, 12 Jan 2007 10:49:13 -0800
Subject: [Python-Dev] The bytes type
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com><79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com><45A57AE0.2060600@benjiyork.com><5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net><eo63cp$1ak$1@sea.gmane.org>
	<45A744A8.40707@v.loewis.de><eo859h$fsa$1@sea.gmane.org><9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org><20070112165932.GA11501@localhost.localdomain><ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>
	<20070112182248.GA15187@localhost.localdomain>
Message-ID: <00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>

[A.M. Kuchling]
> 2.6 wouldn't go changing existing APIs to begin requiring or returning
> the bytes type[*], of course, but extensions and new modules might use
> it.

The premise is dubious.

If I am currently maintaining a module, why would I switch to a bytes type
and forgo compatibility with Py2.5 and prior?  I might as well just convert
it to run on Py3.0 and leave my Py2.5 code as-is for people who want to
run 2.x.

If I'm writing a new module, what's the point of twisting myself into knots
to get it to run on both Py2.6 and Py3.0?  That just makes coding harder
(by limiting me to the intersection of the feature sets).

I think we should draw a line in the sand and resolve not to garbage-up Py2.6.
The whole Py3.0 project is about eliminating cruft and being free of the
bonds of backwards compatibility.  Adding non-essential cruft to Py2.6
goes against that philosophy.


Raymond

From guido at python.org  Fri Jan 12 20:00:28 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 12 Jan 2007 11:00:28 -0800
Subject: [Python-Dev] The bytes type
In-Reply-To: <00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<eo63cp$1ak$1@sea.gmane.org> <45A744A8.40707@v.loewis.de>
	<eo859h$fsa$1@sea.gmane.org>
	<9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>
	<20070112165932.GA11501@localhost.localdomain>
	<ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>
	<20070112182248.GA15187@localhost.localdomain>
	<00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
Message-ID: <ca471dc20701121100h741e8a6dr6e2121dc1873284f@mail.gmail.com>

On 1/12/07, Raymond Hettinger <python at rcn.com> wrote:
> [A.M. Kuchling]
> > 2.6 wouldn't go changing existing APIs to begin requiring or returning
> > the bytes type[*], of course, but extensions and new modules might use
> > it.
>
> The premise is dubious.
>
> If I am currently maintaining a module, why would I switch to a bytes type
> and forgo compatibility with Py2.5 and prior?  I might as well just convert
> it to run on Py3.0 and leave my Py2.5 code as-is for people who want to
> run 2.x.
>
> If I'm writing a new module, what's the point of twisting myself into knots
> to get it to run on both Py2.6 and Py3.0?  That just makes coding harder
> (by limiting me to the intersection of the feature sets).
>
> I think we should draw a line in the sand and resolve not to garbage-up Py2.6.
> The whole Py3.0 project is about eliminating cruft and being free of the
> bonds of backwards compatibility.  Adding non-essential cruft to Py2.6
> goes against that philosophy.

I'm not so sure, since 2.6 will likely be out and stable long before
3.0 gains much of a foothold. I believe the experiences with a similar
approach in the Zope world for the 2->3 transition was overall a
favorable one.

However, I'd be loathe to make any compromises in 3.0 in order to make
life easier for 2.6. The burden must be on 2.6 (and 2.7-2.9), and if
it's just impossible, that's too bad for them.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From amk at amk.ca  Fri Jan 12 20:00:10 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 12 Jan 2007 14:00:10 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
References: <20070112182248.GA15187@localhost.localdomain>
	<00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
Message-ID: <20070112190010.GA26052@localhost.localdomain>

On Fri, Jan 12, 2007 at 10:49:13AM -0800, Raymond Hettinger wrote:
> I think we should draw a line in the sand and resolve not to garbage-up 
> Py2.6.
> The whole Py3.0 project is about eliminating cruft and being free of the
> bonds of backwards compatibility.  Adding non-essential cruft to Py2.6
> goes against that philosophy.

We could add the bytes type to Jython only and leave it out of the 2.x
series.  That still requires nailing down the semantics of 'bytes'.

--amk


From steven.bethard at gmail.com  Fri Jan 12 20:06:51 2007
From: steven.bethard at gmail.com (Steven Bethard)
Date: Fri, 12 Jan 2007 12:06:51 -0700
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
	<45A57AE0.2060600@benjiyork.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<eo63cp$1ak$1@sea.gmane.org> <45A744A8.40707@v.loewis.de>
	<eo859h$fsa$1@sea.gmane.org>
	<9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>
Message-ID: <d11dcfba0701121106s6a19c7c2vc9ba2aa21164aaba@mail.gmail.com>

On 1/12/07, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Jan 12, 2007, at 9:17 AM, Georg Brandl wrote:
>
> > Well, that is one of the cases in which that won't be possible ;)
>
> I think there will be at least three areas that will make porting a
> challenge:
>
> - - APIs where the semantics have changed instead of being moved,
> renamed, or extended
> - - C API changes for extensions and embedded apps
> - - Unicode/str/bytes incompatibilities
>
> It's worth spending time thinking about how we can help ease the
> transition for each.  There may be more thing too.  I wonder if it
> doesn't make sense for all 3xxx PEPs to include a discussion section
> on porting.

Yeah, when this was talked about last time, I wrote PEP 3002 which
requests exactly this:

    http://www.python.org/dev/peps/pep-3002/

It basically proposes that all backwards-incompatible changes be
discussed in a PEP somewhere, and that code like Anthony's be added to
2.X to ease the transition.  Originally, it has proposed just a
utility like Guido's 2to3 but Guido himself suggested something like
Anthony's approach:

    http://mail.python.org/pipermail/python-3000/2006-April/001444.html

I'm planning to update the PEP with any new conclusions arrived at in
this thread.

Steve
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
        --- Bucky Katt, Get Fuzzy

From guido at python.org  Fri Jan 12 20:24:47 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 12 Jan 2007 11:24:47 -0800
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <d11dcfba0701121106s6a19c7c2vc9ba2aa21164aaba@mail.gmail.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>
	<45A57AE0.2060600@benjiyork.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<eo63cp$1ak$1@sea.gmane.org> <45A744A8.40707@v.loewis.de>
	<eo859h$fsa$1@sea.gmane.org>
	<9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>
	<d11dcfba0701121106s6a19c7c2vc9ba2aa21164aaba@mail.gmail.com>
Message-ID: <ca471dc20701121124q587994bbu19d24a0986f4cd5c@mail.gmail.com>

On 1/12/07, Steven Bethard <steven.bethard at gmail.com> wrote:
> Yeah, when this was talked about last time, I wrote PEP 3002 which
> requests exactly this:
>
>     http://www.python.org/dev/peps/pep-3002/
>
> It basically proposes that all backwards-incompatible changes be
> discussed in a PEP somewhere, and that code like Anthony's be added to
> 2.X to ease the transition.  Originally, it has proposed just a
> utility like Guido's 2to3 but Guido himself suggested something like
> Anthony's approach:
>
>     http://mail.python.org/pipermail/python-3000/2006-April/001444.html

To complement 2to3, not to replace it.

> I'm planning to update the PEP with any new conclusions arrived at in
> this thread.

I think that now the 2to3 tool is a reality, its limitations should
perhaps be taken into account.

For example, I don't believe it is realistic to expect it to do
semantic analysis, which means that for example it can't be expected
to know whether 'd' in 'd.keys()' is a dict or something else.

Also, in 'k = d.keys()' it can't be expected to analyze how k is used
later on; to be safe, it will translate all occurrences of d.keys()
into list(d.keys()) which is known to work (and won't be slower than
the original code either).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From raymond.hettinger at verizon.net  Fri Jan 12 20:33:53 2007
From: raymond.hettinger at verizon.net (Raymond Hettinger)
Date: Fri, 12 Jan 2007 11:33:53 -0800
Subject: [Python-Dev] Proposed 3.0 compatiblity module
Message-ID: <016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>

ISTM, we can decide to limit 3.0's impact to a single compatibility module, 
essentially as an alternate set of builtins.

import __new_builtins__
for name in 'bytes dict coerce'.split():      # list any new features you want 
to support
    setattr(__builtins__, name, getattr(__new_builtins__, name))

The bytes type would just be the new type.
The dict would have keys=iterkeys, no has_key, etc.
The coerce function would work but have a DeprecationWarning or somesuch.
If you want future division, we already have a mechanism for that.
If you want to eliminate backticks, just grep for them.
If you don't care about 3.0 compatibilty, just don't import the module.

Outside of a compatibility module, we can decide to leave Py2.6 unmolested by 
Py3.0, so I can stop being cranky.


Raymond 


From glyph at divmod.com  Fri Jan 12 21:22:27 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 12 Jan 2007 20:22:27 -0000
Subject: [Python-Dev] The bytes type
Message-ID: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>

On 06:49 pm, python at rcn.com wrote:

>I think we should draw a line in the sand and resolve not to garbage-up Py2.6.
>The whole Py3.0 project is about eliminating cruft and being free of the
>bonds of backwards compatibility.  Adding non-essential cruft to Py2.6
>goes against that philosophy.

Emotionally charged like "cruft" and "garbage" are obscuring the issue.

Let's replace them with equivalents charged in the opposite direction:

"I think we should draw a line in the sand and resolve not to compatibility-up Py2.6.  The whole Py3.0 project is about eliminating useful libraries and being free of the bonds of working software.  Adding non-essential forward-compatibility to Py2.6 goes against that philosophy."

The benefit (to me, and to many others) of 3.x over 2.x is the promise of more future maintenance, not the lack of cruft.  In fact, if I made a list of my current top ten problems with Python, "cruft" wouldn't even make it in.  There is lots of useful software that will not work in the 3.0 series, and without forward compatibility there is no way to get there from here.

As Guido said, if 3.0 is going to break compatibility, that burdens the 2.x series with the need to provide transitional functionality.  The upgrade path needs to be available in one version or the other, or 2.x needs to be maintained forever.  You can't have it both ways.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070112/71f2a506/attachment.html 

From mike.klaas at gmail.com  Fri Jan 12 21:24:58 2007
From: mike.klaas at gmail.com (Mike Klaas)
Date: Fri, 12 Jan 2007 12:24:58 -0800
Subject: [Python-Dev] The bytes type
In-Reply-To: <00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>
	<eo63cp$1ak$1@sea.gmane.org> <45A744A8.40707@v.loewis.de>
	<eo859h$fsa$1@sea.gmane.org>
	<9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>
	<20070112165932.GA11501@localhost.localdomain>
	<ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>
	<20070112182248.GA15187@localhost.localdomain>
	<00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
Message-ID: <3d2ce8cb0701121224m1c8a44d6g766a3efc1055a253@mail.gmail.com>

On 1/12/07, Raymond Hettinger <python at rcn.com> wrote:
> [A.M. Kuchling]
> > 2.6 wouldn't go changing existing APIs to begin requiring or returning
> > the bytes type[*], of course, but extensions and new modules might use
> > it.
>
> The premise is dubious.
>
> If I am currently maintaining a module, why would I switch to a bytes type
> and forgo compatibility with Py2.5 and prior?  I might as well just convert
> it to run on Py3.0 and leave my Py2.5 code as-is for people who want to
> run 2.x.

A mutables bytes type is a useful addition to 2.X aside of the
3.0-compatibility motivation.  Isn't that sufficient justification?

-Mike

From skip at pobox.com  Fri Jan 12 21:30:47 2007
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 12 Jan 2007 14:30:47 -0600
Subject: [Python-Dev] The bytes type
In-Reply-To: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
Message-ID: <17831.61431.792761.811711@montanaro.dyndns.org>


    glyph> "I think we should draw a line in the sand and resolve not to
    glyph> compatibility-up Py2.6.  The whole Py3.0 project is about
    glyph> eliminating useful libraries and being free of the bonds of
    glyph> working software.  Adding non-essential forward-compatibility to
    glyph> Py2.6 goes against that philosophy."

I think you meant "useless", right?  I'm not aware of any proposals to
eliminate the os, re or csv modules. ;-)

Skip

From guido at python.org  Fri Jan 12 21:39:59 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 12 Jan 2007 12:39:59 -0800
Subject: [Python-Dev] Proposed 3.0 compatiblity module
In-Reply-To: <016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
References: <016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
Message-ID: <ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>

On 1/12/07, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
> ISTM, we can decide to limit 3.0's impact to a single compatibility module,
> essentially as an alternate set of builtins.
>
> import __new_builtins__
> for name in 'bytes dict coerce'.split():      # list any new features you want
> to support
>     setattr(__builtins__, name, getattr(__new_builtins__, name))
>
> The bytes type would just be the new type.
> The dict would have keys=iterkeys, no has_key, etc.
> The coerce function would work but have a DeprecationWarning or somesuch.
> If you want future division, we already have a mechanism for that.
> If you want to eliminate backticks, just grep for them.
> If you don't care about 3.0 compatibilty, just don't import the module.

Most of these make sense, except for dict: dicts are passed around
between modules, and there's little hope that changing the dict type
in *your* module will be able to affect the behavior of dicts you
receive from *other* modules; and the dicts you create and pass out to
other modules will have behavior those other modules don't expect.

> Outside of a compatibility module, we can decide to leave Py2.6 unmolested by
> Py3.0, so I can stop being cranky.

If you're feeling cranky this early in the new year, maybe you should
look for reasons outside this list. :-)

I'm all for striving for minimal impact. I don't think we can attain
zero impact.

BTW, a possible alternative would be to strive for code that runs
under 2.x, and *after automatic transformation using the 2to3 tool*
runs under 3.x, *without mannual patch-up*. That's probably a much
lower bar, and it would still mean that developers wishing to maintain
code that works with both 2.x/3.x only need to maintain a single
version -- the other version could be automatically generated.

Since you need to do a lot of testing anyways to make sure you haven't
accidentally broken compatibility, the effort would be roughly
equivalent to the "identical source" approach that people are
advocating so far, and it would probably lift some restrictions.

For example, working around print becoming a function would mean no
use of print at all when using "identical source"; but that's a
transformation that the 2to3 tool can do trivially and perfectly, so
the "automatic conversion" approach would enable freely using print
statements.

A mixture of the two approaches would perhaps work best.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tcdelaney at optusnet.com.au  Fri Jan 12 22:04:59 2007
From: tcdelaney at optusnet.com.au (Tim Delaney)
Date: Sat, 13 Jan 2007 08:04:59 +1100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>	<79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com>	<45A57AE0.2060600@benjiyork.com>	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>	<eo63cp$1ak$1@sea.gmane.org><45A744A8.40707@v.loewis.de>
	<eo859h$fsa$1@sea.gmane.org>
Message-ID: <007b01c7368d$523656d0$f100a8c0@ryoko>

Georg Brandl wrote:

> Martin v. L?wis schrieb:
>>
>> What does that mean for the example James gave: if dict.items is
>> going to be an iterator in 3.0, what 2.x version can make it return
>> an iterator, when it currently returns a list?
>>
>> There simply can't be a 2.x version that *introduces* the new way,
>> as it is not merely a new API, but a changed API.
>
> Well, that is one of the cases in which that won't be possible ;)

Yes - but dict.items() *isn't* going to just return an iterator - it will 
return a view. For most uses of dict.items(), this means there will not need 
to be any code change.

I'm wondering if we might be going the wrong way about warning about 
compatibility between 2.x and 3.x. Perhaps it might be better if the 3.0 
alpha had a 2.x compatibility mode command-line flag, which is removed late 
in the beta cycle.

Tim Delaney 


From glyph at divmod.com  Fri Jan 12 22:19:56 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 12 Jan 2007 21:19:56 -0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Message-ID: <20070112211956.17094.484130944.divmod.xquotient.1906@joule.divmod.com>

On 09:04 pm, tcdelaney at optusnet.com.au wrote:

>I'm wondering if we might be going the wrong way about warning about
>compatibility between 2.x and 3.x. Perhaps it might be better if the 3.0
>alpha had a 2.x compatibility mode command-line flag, which is removed late
>in the beta cycle.

Please, no.

I don't think command-line flags are going to help much, because the problem is not the complexity of downloading and compiling py3k, it is the complexity of porting software that has modules from a variety of different sources.

More importantly however, I'm not even going to be looking into porting anything to py3k until a year or so after its release - and that's only if 2.6 has given me some tools to deal with the transition.  The reason I am posting to this thread is that I had *written off py3k entirely* before Anthony started making noises about forward compatibility.  The support cycle of Ubuntu Dapper makes it likely that Twisted will be supporting Python 2.4 until at least 2011.  I assume 2.5 will last a year after that, so assuming 2.6 has perfect forward compatibility, we'll be looking at a py3k port in early 2013.  Features available in the beta releases aren't going to help me there.

Plus, this defeats the whole purpose of py3k.  Why break compatibility if you're going to include a bunch of gross compatibility code?  Compatibility has already been broken in the py3k branch, there's no point in putting it back there.  Adding it to 2.x might make sense though.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070112/bc048187/attachment.htm 

From tjreedy at udel.edu  Fri Jan 12 22:40:08 2007
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 12 Jan 2007 16:40:08 -0500
Subject: [Python-Dev] The bytes type
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com><79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com><45A57AE0.2060600@benjiyork.com><5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net><eo63cp$1ak$1@sea.gmane.org>
	<45A744A8.40707@v.loewis.de><eo859h$fsa$1@sea.gmane.org><9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org><20070112165932.GA11501@localhost.localdomain><ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>
	<20070112182248.GA15187@localhost.localdomain>
Message-ID: <eo8v7l$shv$1@sea.gmane.org>


"A.M. Kuchling" <amk at amk.ca> wrote in message 
news:20070112182248.GA15187 at localhost.localdomain...
| [*] Anyone else keep wanting to write "byte type"?

All the other builtin types I can think of are singular.  So I think byte 
should be also. 




From fuzzyman at voidspace.org.uk  Fri Jan 12 22:49:12 2007
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 12 Jan 2007 21:49:12 +0000
Subject: [Python-Dev] The bytes type
In-Reply-To: <eo8v7l$shv$1@sea.gmane.org>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com><79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com><45A57AE0.2060600@benjiyork.com><5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net><eo63cp$1ak$1@sea.gmane.org>	<45A744A8.40707@v.loewis.de><eo859h$fsa$1@sea.gmane.org><9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org><20070112165932.GA11501@localhost.localdomain><ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>	<20070112182248.GA15187@localhost.localdomain>
	<eo8v7l$shv$1@sea.gmane.org>
Message-ID: <45A80258.6000509@voidspace.org.uk>

Terry Reedy wrote:
> "A.M. Kuchling" <amk at amk.ca> wrote in message 
> news:20070112182248.GA15187 at localhost.localdomain...
> | [*] Anyone else keep wanting to write "byte type"?
>
> All the other builtin types I can think of are singular.  So I think byte 
> should be also. 
>
>   
The string holds 'a string' (singular), the bytes datatype will hold 
'bytes' plural. Bytes is more accurate and I would stumble over byte.

Michael Foord

>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>   


From jcarlson at uci.edu  Fri Jan 12 23:57:18 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 12 Jan 2007 14:57:18 -0800
Subject: [Python-Dev] The bytes type
In-Reply-To: <eo8v7l$shv$1@sea.gmane.org>
References: <20070112182248.GA15187@localhost.localdomain>
	<eo8v7l$shv$1@sea.gmane.org>
Message-ID: <20070112144746.8DB7.JCARLSON@uci.edu>


"Terry Reedy" <tjreedy at udel.edu> wrote:
> "A.M. Kuchling" <amk at amk.ca> wrote in message 
> news:20070112182248.GA15187 at localhost.localdomain...
> | [*] Anyone else keep wanting to write "byte type"?
> 
> All the other builtin types I can think of are singular.  So I think byte 
> should be also. 

But a "byte" already has a standard definition in the realm of computer
science and/or programming, and that definition is singular.  No one
will think that a byte is a container or a sequence (which is more or
less what it is), they will think it is like a C 'char'.  Bytes, on the
other hand, at least lets people know from the first read that it is a
container or sequence of some kind.  Of what?  Bytes.

 - Josiah


From sluggoster at gmail.com  Sat Jan 13 00:13:32 2007
From: sluggoster at gmail.com (Mike Orr)
Date: Fri, 12 Jan 2007 15:13:32 -0800
Subject: [Python-Dev] The bytes type
In-Reply-To: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
Message-ID: <6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>

On 1/12/07, glyph at divmod.com <glyph at divmod.com> wrote:
> On 06:49 pm, python at rcn.com wrote:
>
> >I think we should draw a line in the sand and resolve not to garbage-up
> Py2.6.
> >The whole Py3.0 project is about eliminating cruft and being free of the
> >bonds of backwards compatibility.  Adding non-essential cruft to Py2.6
> >goes against that philosophy.
>
> Emotionally charged like "cruft" and "garbage" are obscuring the issue.
>
> Let's replace them with equivalents charged in the opposite direction:
>
> "I think we should draw a line in the sand and resolve not to
> compatibility-up Py2.6.  The whole Py3.0 project is about eliminating useful
> libraries and being free of the bonds of working software.  Adding
> non-essential forward-compatibility to Py2.6 goes against that philosophy."

Well put.  Not that I agree.

> The benefit (to me, and to many others) of 3.x over 2.x is the promise of
> more future maintenance, not the lack of cruft.

The benefit (to me, and to many others) of 3.x over 2.x is the promise
of getting rid of cruft.
If we're going to re-add cruft for the sake of temporary
compatibility, we may as well just stick with 2.x.  You have to take a
quantum leap sometimes or you end up working around the same old
mistakes.

For instance, getting rid of the str/unicode dichotomy would be a
wonderful thing. Then if I make a subclass of string I don't have to
think "str or unicode? or two parallel classes?"  It'll also eliminate
the possibility of a stray 8-bit string containing non-ASCII
characters from making your program go kablooie UnicodeDecodeError
when it's in a "%" or ".join()", or at least localize the problem to
ONE place (the string constructor) where it'll be in every
programmer's face and they'll all hopefully learn to handle it
consistently and not pass bad strings to other packages.

Getting rid of old-style classes would be nice too.

-- 
Mike Orr <sluggoster at gmail.com>

From Jack.Jansen at cwi.nl  Sat Jan 13 00:54:26 2007
From: Jack.Jansen at cwi.nl (Jack Jansen)
Date: Sat, 13 Jan 2007 00:54:26 +0100
Subject: [Python-Dev] file(file)
In-Reply-To: <ca471dc20701121001x2d1f1ea0h6266e8aa60ed56f2@mail.gmail.com>
References: <20070112171919.GA14045@localhost.localdomain>
	<ca471dc20701121001x2d1f1ea0h6266e8aa60ed56f2@mail.gmail.com>
Message-ID: <8F23FB57-3A79-4521-9462-BE8B10FE0F54@cwi.nl>


On  12-Jan-2007, at 19:01 , Guido van Rossum wrote:

> On 1/12/07, A.M. Kuchling <amk at amk.ca> wrote:
>> Many types in Python are idempotent, so that int(1) works
>> as expected, float(2.34)==2.34, ''.join('hello')=='hello'
>> et cetera.
>
> I'm not sure I understand the use case; I don't believe I've ever felt
> the need for this.

I have oodles of routines of the form
def foo(thefile):
	if type(thefile) == str: thefile = open(thefile)
or
	if not hasattr(thefile, 'read'): thefile = open(thefile)
or something similar.

> We should also consider the semantics in more detail. Should the seek
> position be shared between the two objects? What about buffering?

That's definitely the hard part. But it's somewhat similar to  
"normal" mutable objects which are (I think always, right?) shallow  
copied when used in a constructor.
--
Jack Jansen, <Jack.Jansen at cwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma  
Goldman


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070113/beebb5c5/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2255 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20070113/beebb5c5/attachment.bin 

From rrr at ronadam.com  Sat Jan 13 01:26:10 2007
From: rrr at ronadam.com (Ron Adam)
Date: Fri, 12 Jan 2007 18:26:10 -0600
Subject: [Python-Dev] The bytes type
In-Reply-To: <00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com><79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com><45A57AE0.2060600@benjiyork.com><5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net><eo63cp$1ak$1@sea.gmane.org>	<45A744A8.40707@v.loewis.de><eo859h$fsa$1@sea.gmane.org><9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org><20070112165932.GA11501@localhost.localdomain><ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>	<20070112182248.GA15187@localhost.localdomain>
	<00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
Message-ID: <45A82722.8060409@ronadam.com>

Raymond Hettinger wrote:
> [A.M. Kuchling]
>> 2.6 wouldn't go changing existing APIs to begin requiring or returning
>> the bytes type[*], of course, but extensions and new modules might use
>> it.
> 
> The premise is dubious.
> 
> If I am currently maintaining a module, why would I switch to a bytes type
> and forgo compatibility with Py2.5 and prior?  I might as well just convert
> it to run on Py3.0 and leave my Py2.5 code as-is for people who want to
> run 2.x.
> 
> If I'm writing a new module, what's the point of twisting myself into knots
> to get it to run on both Py2.6 and Py3.0?  That just makes coding harder
> (by limiting me to the intersection of the feature sets).
> 
> I think we should draw a line in the sand and resolve not to garbage-up Py2.6.
> The whole Py3.0 project is about eliminating cruft and being free of the
> bonds of backwards compatibility.  Adding non-essential cruft to Py2.6
> goes against that philosophy.
> 
> 
> Raymond

+1


For me, the thing that will make porting 2.x to 3.x code easy is to make python 
3.0 as clean and organized as you can with excellent documentation.  Half-way 
and duel-way approaches will probably not help me as much as this.

It also seems to me that instead of putting 3k warnings into 2.X, a external 3k 
warnings utility would work just as well.  It could generate a list of 
("filename", line #, "problem_string") tuples and even cross-reference an 
upgrade guide by using the problem_string for advice on each item.  The upgrade 
guide could contain examples and suggestions learned from experiences at 
upgrading pythons library.

Most of the difficulty I have in converting programs is in finding the 
information I need.  The actual editing is not a problem.

I think making external utilities such as these easy to use will go a long way.

Cheers,
    Ron


From brett at python.org  Sat Jan 13 01:37:26 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 12 Jan 2007 16:37:26 -0800
Subject: [Python-Dev] file(file)
In-Reply-To: <20070112171919.GA14045@localhost.localdomain>
References: <20070112171919.GA14045@localhost.localdomain>
Message-ID: <bbaeab100701121637i23820d49gfd8a99e10828da8e@mail.gmail.com>

On 1/12/07, A.M. Kuchling <amk at amk.ca> wrote:
> Forwarded for discussion from http://www.python.org/sf/1633665.
>
> --amk
>
> [forwarded from http://bugs.debian.org/327060]
>
> Many types in Python are idempotent, so that int(1) works
> as expected, float(2.34)==2.34, ''.join('hello')=='hello'
> et cetera.
>
> Why not file()? Currently, file(open(something, 'r')) fails
> with "TypeError: coercing to Unicode: need string or buffer, file found."
>
> Semantically, file(fd) should be equivalent to os.fdopen(fd.fileno())
> or the proposed file.fromfd() (Jp Calderone, Python-dev, 2003).
> You should get another independent
> file object that accesses the same file.
>

For security reasons I might be asking for file's constructor to be
removed from the type for Python source code at some point (it can be
relocated to an extension module if desired).  By forcing people to go
through open() to create a file object you can more easily control
read/write access to the file system (assuming the proper importation
of extension modules has been blocked).  Not removing the constructor
allows any code that has been explicitly given a file object but not
open() to just get the class and call the constructor to open a new
file.

Don't know when I am gonig to get around to asking for it, though
(still have to finish my security work to have something to present to
python-dev).

-Brett

From glyph at divmod.com  Sat Jan 13 01:54:58 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Sat, 13 Jan 2007 00:54:58 -0000
Subject: [Python-Dev] file(file)
Message-ID: <20070113005458.17094.1916256070.divmod.xquotient.1925@joule.divmod.com>

On 12:37 am, brett at python.org wrote:

>For security reasons I might be asking for file's constructor to be
>removed from the type for Python source code at some point (it can be
>relocated to an extension module if desired).  By forcing people to go
>through open() to create a file object you can more easily control
>read/write access to the file system (assuming the proper importation
>of extension modules has been blocked).  Not removing the constructor
>allows any code that has been explicitly given a file object but not
>open() to just get the class and call the constructor to open a new
>file.

This is a general problem with type access.  Secure versions of any type should not allow access to the type period.  It is hardly unique to files, and is not limited to constructors either.  How do you, e.g., allow a restricted piece of code write access to only a specified area of the filesystem?

More importantly, given the random behavior that open() will be growing (opening sockets?  dynamic dispatch on URL scheme???) file() will likely remain a popular way to be sure you are accessing the filesystem.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070113/50a2beac/attachment.htm 

From brett at python.org  Sat Jan 13 03:42:06 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 12 Jan 2007 18:42:06 -0800
Subject: [Python-Dev] file(file)
In-Reply-To: <20070113005458.17094.1916256070.divmod.xquotient.1925@joule.divmod.com>
References: <20070113005458.17094.1916256070.divmod.xquotient.1925@joule.divmod.com>
Message-ID: <bbaeab100701121842k6a065c0chd487c60b52f55bbd@mail.gmail.com>

On 1/12/07, glyph at divmod.com <glyph at divmod.com> wrote:
> On 12:37 am, brett at python.org wrote:
>
> >For security reasons I might be asking for file's constructor to be
> >removed from the type for Python source code at some point (it can be
> >relocated to an extension module if desired).  By forcing people to go
> >through open() to create a file object you can more easily control
> >read/write access to the file system (assuming the proper importation
> >of extension modules has been blocked).  Not removing the constructor
> >allows any code that has been explicitly given a file object but not
> >open() to just get the class and call the constructor to open a new
> >file.
>
> This is a general problem with type access.  Secure versions of any type
> should not allow access to the type period.  It is hardly unique to files,
> and is not limited to constructors either.  How do you, e.g., allow a
> restricted piece of code write access to only a specified area of the
> filesystem?
>

Wrapper around open() that does proper checking of its arguments.  I
will be discussing my security stuff at PyCon if you are attending and
are interested.

-Brett

From kbk at shore.net  Sat Jan 13 03:59:20 2007
From: kbk at shore.net (Kurt B. Kaiser)
Date: Fri, 12 Jan 2007 21:59:20 -0500 (EST)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200701130259.l0D2xK59005042@bayview.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  421 open ( +3) /  3530 closed ( +8) /  3951 total (+11)
Bugs    :  963 open ( +4) /  6426 closed (+21) /  7389 total (+25)
RFE     :  255 open ( +5) /   246 closed ( +1) /   501 total ( +6)

New / Reopened Patches
______________________

The Unicode "lazy strings" patches  (2007-01-06)
       http://python.org/sf/1629305  opened by  Larry Hastings

fast tuple[index] by inlining on BINARY_SUBSCR  (2007-01-07)
CLOSED http://python.org/sf/1629718  opened by  Hirokazu Yamamoto

Patch to add tempfile.SpooledTemporaryFile (for #415692)  (2007-01-07)
       http://python.org/sf/1630118  opened by  Dustin J. Mitchell

Implement named exception cleanup  (2007-01-07)
CLOSED http://python.org/sf/1630248  opened by  Collin Winter

Fix crash when replacing sys.stdout in sitecustomize  (2007-01-08)
       http://python.org/sf/1630975  opened by  Thomas Wouters

SyntaxWarning for backquotes  (2007-01-09)
       http://python.org/sf/1631035  opened by  Anthony Baxter

implement warnings module in C  (2007-01-08)
       http://python.org/sf/1631171  opened by  Neal Norwitz

sre module has misleading docs  (2007-01-09)
       http://python.org/sf/1631394  opened by  Tom Lynn

New exception syntax  (2007-01-09)
CLOSED http://python.org/sf/1631942  opened by  Collin Winter

from __future__ import print_function  (2007-01-12)
       http://python.org/sf/1633807  opened by  Anthony Baxter

Py3k: Fix pybench so it runs  (2007-01-13)
       http://python.org/sf/1634499  opened by  Larry Hastings

Patches Closed
______________

Win32: Add bytesobject.c to pythoncore.vcproj  (2007-01-04)
       http://python.org/sf/1628062  closed by  loewis

Win32: Fix build when you have TortoiseSVN but no .svn/*  (2007-01-04)
       http://python.org/sf/1628061  closed by  loewis

fast tuple[index] by inlining on BINARY_SUBSCR  (2007-01-07)
       http://python.org/sf/1629718  closed by  loewis

#1603424 subprocess.py wrongly claims 2.2 compatibility.  (2006-12-05)
       http://python.org/sf/1609282  closed by  astrand

Implement named exception cleanup  (2007-01-07)
       http://python.org/sf/1630248  closed by  collinwinter

(py3k) Remove the sets module  (2006-06-04)
       http://python.org/sf/1500611  closed by  gvanrossum

New exception syntax  (2007-01-09)
       http://python.org/sf/1631942  closed by  collinwinter

backticks will not be used at all  (2007-01-03)
       http://python.org/sf/1627052  closed by  gbrandl

New / Reopened Bugs
___________________

Python 2.5 64 bit compile fails on Solaris 10/gcc 4.1.1  (2007-01-05)
       http://python.org/sf/1628484  opened by  Bob Atkins

Pydoc sets choices for doc locations incorrectly  (2007-01-05)
       http://python.org/sf/1628895  opened by  Skip Montanaro

xml.dom.minidom parse bug  (2007-01-05)
CLOSED http://python.org/sf/1628902  opened by  Pierre Imbaud

clarify 80-char limit  (2007-01-05)
       http://python.org/sf/1628906  opened by  Jim Jewett

inspect trouble when source file changes  (2007-01-05)
       http://python.org/sf/1628987  opened by  phil

Incorrect type in PyDict_Next() example code  (2007-01-05)
       http://python.org/sf/1629125  opened by  Jason Evans

email._parseaddr AddrlistClass bug  (2007-01-06)
       http://python.org/sf/1629369  opened by  Tokio Kikuchi

array.array borks on deepcopy  (2006-08-24)
CLOSED http://python.org/sf/1545837  reopened by  nnorwitz

documentation of email.utils.parsedate is confusing  (2007-01-06)
       http://python.org/sf/1629566  opened by  Nicholas Riley

Backward incompatibility in logging.py   (2006-10-02)
CLOSED http://python.org/sf/1569622  reopened by  mklaas

doc error for re.sub  (2007-01-08)
CLOSED http://python.org/sf/1630511  opened by  Keith Briggs

doc misleading in re.compile  (2007-01-08)
       http://python.org/sf/1630515  opened by  Keith Briggs

Seg fault in readline call.  (2007-01-08)
       http://python.org/sf/1630794  opened by  gnovak

fnmatch.translate undocumented  (2007-01-08)
       http://python.org/sf/1630844  opened by  Gabriel Genellina

PyLong_AsLong doesn't check tp_as_number  (2007-01-08)
CLOSED http://python.org/sf/1630863  opened by  Roger Upole

Garbage output to file of specific size  (2007-01-08)
       http://python.org/sf/1630894  opened by  Michael Culbertson

email.Generators does not separates headers with "\r\n"  (2005-11-05)
       http://python.org/sf/1349106  reopened by  bwarsaw

Discrepancy between iterating empty and non-empty deques  (2007-01-09)
CLOSED http://python.org/sf/1631769  opened by  Christos Georgiou

logging.config.fileConfig doesn't clear logging._handlerList  (2007-01-10)
CLOSED http://python.org/sf/1632328  opened by  Stefan H. Holek

Hangs with 100% CPU load for certain regexes  (2007-01-11)
CLOSED http://python.org/sf/1633583  opened by  Matthias Klose

using locale does not display the intended behavior  (2007-01-11)
       http://python.org/sf/1633600  opened by  Matthias Klose

logging module / wrong bytecode?  (2007-01-12)
       http://python.org/sf/1633605  opened by  Matthias Klose

curses should reset curses.{COLS,LINES} when term. resized  (2007-01-12)
       http://python.org/sf/1633621  opened by  Matthias Klose

time.strftime() accepts format which time.strptime doesnt  (2007-01-12)
       http://python.org/sf/1633628  opened by  Matthias Klose

class derived from float evaporates under +=  (2007-01-12)
       http://python.org/sf/1633630  opened by  Matthias Klose

incomplete numerical comparisons  (2007-01-12)
       http://python.org/sf/1633648  opened by  Matthias Klose

file(file) should succeed  (2007-01-12)
CLOSED http://python.org/sf/1633665  opened by  Matthias Klose

mailbox.py _fromlinepattern regexp does not support positive  (2007-01-12)
       http://python.org/sf/1633678  opened by  Matthias Klose

AIX: configure ignores $CC; problems mit C++ comments  (2007-01-12)
       http://python.org/sf/1633863  opened by  Johannes Abt

for line in sys.stdin: doesn't notice EOF the first time   (2007-01-12)
       http://python.org/sf/1633941  opened by  Matthias Klose

re.compile("(.*$){1,4}", re.MULTILINE) fails  (2007-01-12)
       http://python.org/sf/1633953  opened by  Matthias Klose

configure problem for sem_init() on HP-UX  (2007-01-12)
CLOSED http://python.org/sf/1634033  opened by  Johannes Abt

AIX: wrong flags for ld when linking standard .so modules  (2007-01-12)
CLOSED http://python.org/sf/1634105  opened by  Johannes Abt

subprocess swallows empty arguments under win32  (2007-01-12)
       http://python.org/sf/1634343  opened by  Patrick M?zard

Bugs Closed
___________

documentation error for "startswith" string method  (2007-01-04)
       http://python.org/sf/1627956  closed by  akuchling

null bytes in docstrings  (2006-12-26)
       http://python.org/sf/1622533  closed by  akuchling

gzip.GzipFile is slow  (2003-11-25)
       http://python.org/sf/849046  closed by  akuchling

xml.dom.minidom parse bug  (2007-01-05)
       http://python.org/sf/1628902  closed by  loewis

frame->f_lasti not always correct  (2006-01-18)
       http://python.org/sf/1409443  closed by  rhettinger

Undocumented implicit strip() in split(None) string method  (2005-01-19)
       http://python.org/sf/1105286  closed by  rhettinger

split() description not fully accurate  (2005-12-14)
       http://python.org/sf/1380970  closed by  rhettinger

Underspecified behaviour of string methods split, rsplit  (2006-01-25)
       http://python.org/sf/1414673  closed by  rhettinger

32/64bit pickled Random incompatiblity  (2006-04-18)
       http://python.org/sf/1472695  closed by  rhettinger

array.array borks on deepcopy  (2006-08-24)
       http://python.org/sf/1545837  closed by  nnorwitz

subprocess.py (py2.5) wrongly claims py2.2 compatibility  (2006-11-27)
       http://python.org/sf/1603424  closed by  astrand

subprocess.Popen closes fds for sys.stdout or sys.stderr  (2006-11-28)
       http://python.org/sf/1604851  closed by  astrand

Backward incompatibility in logging.py   (2006-10-02)
       http://python.org/sf/1569622  closed by  vsajip

doc error for re.sub  (2007-01-08)
       http://python.org/sf/1630511  closed by  mwh

PyLong_AsLong doesn't check tp_as_number  (2007-01-08)
       http://python.org/sf/1630863  closed by  loewis

RotatingFileHandler cannot recover from failed doRollover()  (2007-01-04)
       http://python.org/sf/1627575  closed by  vsajip

Discrepancy between iterating empty and non-empty deques  (2007-01-09)
       http://python.org/sf/1631769  closed by  rhettinger

ctypes: Returning c_void_p from callback doesn\'t work  (2006-10-10)
       http://python.org/sf/1574593  closed by  theller

logging.config.fileConfig doesn't clear logging._handlerList  (2007-01-10)
       http://python.org/sf/1632328  closed by  vsajip

logging's fileConfig causes KeyError on shutdown  (2006-08-04)
       http://python.org/sf/1534765  closed by  vsajip

Hangs with 100% CPU load for certain regexes  (2007-01-11)
       http://python.org/sf/1633583  closed by  niemeyer

configure problem for sem_init() on HP-UX  (2007-01-12)
       http://python.org/sf/1634033  deleted by  jabt

AIX: wrong flags for ld when linking standard .so modules  (2007-01-12)
       http://python.org/sf/1634105  deleted by  jabt

New / Reopened RFE
__________________

Show "expected" token on syntax error  (2007-01-12)
       http://python.org/sf/1634034  opened by  Oliver Gramberg

RFE Closed
__________

file(file) should succeed  (2007-01-11)
       http://python.org/sf/1633665  closed by  akuchling


From glyph at divmod.com  Sat Jan 13 06:09:31 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Sat, 13 Jan 2007 05:09:31 -0000
Subject: [Python-Dev] file(file)
Message-ID: <20070113050931.17094.652453212.divmod.xquotient.1952@joule.divmod.com>




On 02:42 am, brett at python.org wrote:

>Wrapper around open() that does proper checking of its arguments.  I
>will be discussing my security stuff at PyCon if you are attending and
>are interested.

I am both, so I guess I'll see you there :).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070113/ae01c063/attachment.htm 

From tjreedy at udel.edu  Sat Jan 13 08:36:13 2007
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 13 Jan 2007 02:36:13 -0500
Subject: [Python-Dev] file(file)
References: <20070112171919.GA14045@localhost.localdomain><ca471dc20701121001x2d1f1ea0h6266e8aa60ed56f2@mail.gmail.com>
	<8F23FB57-3A79-4521-9462-BE8B10FE0F54@cwi.nl>
Message-ID: <eoa25d$a4p$1@sea.gmane.org>


"Jack Jansen" <Jack.Jansen at cwi.nl> wrote in message 
news:8F23FB57-3A79-4521-9462-BE8B10FE0F54 at cwi.nl...
|
| On  12-Jan-2007, at 19:01 , Guido van Rossum wrote:
|
| > On 1/12/07, A.M. Kuchling <amk at amk.ca> wrote:
| >> Many types in Python are idempotent, so that int(1) works
| >> as expected, float(2.34)==2.34, ''.join('hello')=='hello'
| >> et cetera.
| >
| > I'm not sure I understand the use case; I don't believe I've ever felt
| > the need for this.
|
| I have oodles of routines of the form
| def foo(thefile):
| if type(thefile) == str: thefile = open(thefile)
| or
| if not hasattr(thefile, 'read'): thefile = open(thefile)
| or something similar.

This strikes me as being pretty directly analogous with functions that take 
either an iterable or iterator as argument for a parameter, and which start 
with
'it = iter(iterable)'.  If we follow that analogy, file(fileob) should be 
the fileob itself, just as iter(iterator) == iterator.

| > We should also consider the semantics in more detail. Should the seek
| > position be shared between the two objects? What about buffering?
|
| That's definitely the hard part. But it's somewhat similar to
| "normal" mutable objects which are (I think always, right?) shallow
| copied when used in a constructor.

But why, in the normal case, do you want a copy?  It seems to me that is 
thefile is a file rather than a string, then that is what you want, and you 
are asking for
'thefile = file(thefile)' to work just as the iter statement does.

Terry Jan Reedy






From tjreedy at udel.edu  Sat Jan 13 08:38:57 2007
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 13 Jan 2007 02:38:57 -0500
Subject: [Python-Dev] The bytes type
References: <20070112182248.GA15187@localhost.localdomain><eo8v7l$shv$1@sea.gmane.org>
	<20070112144746.8DB7.JCARLSON@uci.edu>
Message-ID: <eoa2ai$ac2$1@sea.gmane.org>


"Josiah Carlson" <jcarlson at uci.edu> wrote in message 
news:20070112144746.8DB7.JCARLSON at uci.edu...
|
| "Terry Reedy" <tjreedy at udel.edu> wrote:
| > "A.M. Kuchling" <amk at amk.ca> wrote in message
| > news:20070112182248.GA15187 at localhost.localdomain...
| > | [*] Anyone else keep wanting to write "byte type"?
| >
| > All the other builtin types I can think of are singular.  So I think 
byte
| > should be also.
|
| But a "byte" already has a standard definition in the realm of computer
| science and/or programming, and that definition is singular.  No one
| will think that a byte is a container or a sequence (which is more or
| less what it is), they will think it is like a C 'char'.  Bytes, on the
| other hand, at least lets people know from the first read that it is a
| container or sequence of some kind.  Of what?  Bytes.

Okay, this pretty well convinces me.  Opinion withdrawn.

tjr




From g.brandl at gmx.net  Sat Jan 13 14:25:09 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 13 Jan 2007 13:25:09 +0000
Subject: [Python-Dev] file(file)
In-Reply-To: <eoa25d$a4p$1@sea.gmane.org>
References: <20070112171919.GA14045@localhost.localdomain><ca471dc20701121001x2d1f1ea0h6266e8aa60ed56f2@mail.gmail.com>	<8F23FB57-3A79-4521-9462-BE8B10FE0F54@cwi.nl>
	<eoa25d$a4p$1@sea.gmane.org>
Message-ID: <eoamjm$lgd$1@sea.gmane.org>

   Terry Reedy schrieb:

> | > We should also consider the semantics in more detail. Should the seek
> | > position be shared between the two objects? What about buffering?
> |
> | That's definitely the hard part. But it's somewhat similar to
> | "normal" mutable objects which are (I think always, right?) shallow
> | copied when used in a constructor.
> 
> But why, in the normal case, do you want a copy?  It seems to me that is 
> thefile is a file rather than a string, then that is what you want, and you 
> are asking for
> 'thefile = file(thefile)' to work just as the iter statement does.

Consider the use case presented before:

def foo(thefile):
     if type(thefile) == str:
         thefile = open(thefile)

Now, if thefile is already a file object, you won't have to close it
because it came from somewhere else. If thefile is a string, you'll have to
close it at the end.

Now, if file(fileobject) would return another file object created by
something like os.dup(), you can do

def foo(thefile):
     try:
         thefile = file(thefile)

         (...)
     finally:
         thefile.close()


Georg


From mwh at python.net  Sat Jan 13 14:56:18 2007
From: mwh at python.net (Michael Hudson)
Date: Sat, 13 Jan 2007 14:56:18 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <20070112105621.17094.642619803.divmod.xquotient.1347@joule.divmod.com>
	(glyph@divmod.com's message of "Fri, 12 Jan 2007 10:56:21 -0000")
References: <20070112105621.17094.642619803.divmod.xquotient.1347@joule.divmod.com>
Message-ID: <87odp3kp31.fsf@starship.python.net>

glyph at divmod.com writes:

> On 10:12 am, mwh at python.net wrote:
>
>>For practical reasons (we have enough work to be getting on with) PyPy
>>is more-or-less ignoring Python 2.5 at the moment.  After funding and
>>so on, when there's less pressure, maybe it will seem worth it.  Not
>>soon though.
>
> I think I know what you mean from previous conversations, but the
> context of the question makes the answer ambiguous.  Are you
> ignoring 2.5 in favor of 2.4, or 3.0?

In favour of 2.4 -- the one that exists now :) 

Cheers,
mwh

-- 
  > It might get my attention if you'd spin around in your chair,
  > spoke in tongues, and puked jets of green goblin goo.
  I can arrange for this.  ;-)            -- Barry Warsaw & Fred Drake

From martin at v.loewis.de  Sat Jan 13 16:47:45 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 13 Jan 2007 16:47:45 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <200701122003.56210.anthony@interlink.com.au>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>
	<eo63cp$1ak$1@sea.gmane.org> <45A744A8.40707@v.loewis.de>
	<200701122003.56210.anthony@interlink.com.au>
Message-ID: <45A8FF21.7010806@v.loewis.de>

Anthony Baxter schrieb:
> There's a couple of ways I see it - we could add a "-3" command line 
> flag to enable 3.x compat, or maybe a from __future__ statement. 
> Although the latter would be a global thing, which is different to 
> how all existing from __future__s work, so probably not good.

Neither does really work: if some package is rewritten to expect
.items() to be an iterator, one would have to turn on that -3 option
if using that package. OTOH, some other package used at the same
time might expect just the opposite.

> I don't see a path forward that doesn't involve something painful, 
> so long as 3.0 is going to be the clean break.

There would be if there was some version which already had .items
as an iterator, but still supported .iteritems as well. Let's call
that version 2.99. It would be compatible with 2.6, but also with
3.0.

Regards,
Martin


From martin at v.loewis.de  Sat Jan 13 16:50:56 2007
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 13 Jan 2007 16:50:56 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <20070112105144.17094.749039949.divmod.xquotient.1344@joule.divmod.com>
References: <20070112105144.17094.749039949.divmod.xquotient.1344@joule.divmod.com>
Message-ID: <45A8FFE0.5010605@v.loewis.de>

glyph at divmod.com schrieb:
>  It would certainly be possible to have:
> 
>    from __future__ import items_is_iter
> 
> be the same as:
> 
>    __py3k_compat_items_is_iter__ = True
> 
> and have the 2.x series' items() method check the globals() of the
> calling scope to identify the return value of items() in that particular
> context.

Why do you think that this would be that certainly possible?
I cannot imagine an efficient implementation.

Regards,
Martin

From martin at v.loewis.de  Sat Jan 13 16:58:32 2007
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 13 Jan 2007 16:58:32 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <20070112101236.17094.1688023152.divmod.xquotient.1279@joule.divmod.com>
References: <20070112101236.17094.1688023152.divmod.xquotient.1279@joule.divmod.com>
Message-ID: <45A901A8.605@v.loewis.de>

glyph at divmod.com schrieb:
>>There will certainly be demand for an asynchronous server in 3.0,
> 
> To flip the question around: there might be a demand for Twisted in 3.0,
> but will there be a demand for 3.0 in Twisted?  It might just be easier
> for everyone concerned to just continue maintaining 2.x forever.  I have
> yet to see a reason why, other than continued maintenance, 3.0 would be
> a preferable development platform.

Eventually, continued maintenance *will* be a concern to your users.
Maybe not at 3.0, but perhaps at 3.2.

Regards,
Martin

From rasky at develer.com  Sat Jan 13 20:52:22 2007
From: rasky at develer.com (Giovanni Bajo)
Date: Sat, 13 Jan 2007 20:52:22 +0100
Subject: [Python-Dev] file(file)
In-Reply-To: <bbaeab100701121637i23820d49gfd8a99e10828da8e@mail.gmail.com>
References: <20070112171919.GA14045@localhost.localdomain>
	<bbaeab100701121637i23820d49gfd8a99e10828da8e@mail.gmail.com>
Message-ID: <eobd9l$44m$1@sea.gmane.org>

On 13/01/2007 1.37, Brett Cannon wrote:

> For security reasons I might be asking for file's constructor to be
> removed from the type for Python source code at some point (it can be
> relocated to an extension module if desired).  

Isn't there the wrong list then? I can't see how this can ever happen in the 
2.x serie...
-- 
Giovanni Bajo


From nnorwitz at gmail.com  Sun Jan 14 01:09:56 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sat, 13 Jan 2007 16:09:56 -0800
Subject: [Python-Dev] updating Misc/NEWS
Message-ID: <ee2a432c0701131609j65346585h7abfa7c82a5f76fd@mail.gmail.com>

I've noticed a bunch of changes recently without corresponding items
added to Misc/NEWS.  Can everyone update NEWS especially when fixing
bugs or adding new features?    If you have made recent checkins, it
would be great if you could go back and update Misc/NEWS if you missed
it the first time.

Thanks,
n

From emerson.clarke at gmail.com  Wed Jan 10 06:20:38 2007
From: emerson.clarke at gmail.com (Emerson Clarke)
Date: Wed, 10 Jan 2007 05:20:38 +0000
Subject: [Python-Dev] C standard io library, flush/read/write problem
Message-ID: <80718dd60701092120q51376eebkfed997f2b75390a8@mail.gmail.com>

The other night i was watching a google techtalk about python 3000 and
Guido mentioned some problems with the C standard io library.

In particular he highlighted an issue with switching between reading
and writing without flushing and the fact that it caused serious
errors.  Not that i dont think its a good idea to write a new io
library, but I wondered if it was the same problem ive encounted.

It only happens on windows that i know off, but the fix is simple...

Assuming you have a hanlde to the file called "Handle" and a Flush()
method, the following logic for read and write will allow you to
detect and prevent the problem.

Add this to the Read() method before reading takes place:

if ( Handle && (Handle->_flag & _IORW) && (Handle->_flag & (_IOREAD |
_IOWRT)) == _IOWRT )
{
	Flush();
	Handle->_flag |= _IOREAD;
}

Add this to the Write() method before writing takes place:

if ( Handle && (Handle->_flag & _IORW) && (Handle->_flag & (_IOREAD |
_IOWRT)) == _IOREAD )
{
	Flush();
	Handle->_flag |= _IOWRT;
}



Emerson

From jafo at tummy.com  Thu Jan 11 00:30:28 2007
From: jafo at tummy.com (Sean Reifschneider)
Date: Wed, 10 Jan 2007 16:30:28 -0700
Subject: [Python-Dev] IOError errno=514 in time.sleep(1)
Message-ID: <20070110233028.GA10791@tummy.com>

I've been tracking down an issue with some code sporadically raising
IOError with errno=514 on "time.sleep(1)".  time.sleep() is implemented
with the select() system call, and on Linux this is sporadically returning
errno=514 (ERESTARTNOHAND).

This is on 2.6.9, though I've heard reports of it happening through 2.6.17,
and my review of the code indicates that 2.6.19.1 (the latest kernel) is
probably also going to act the same.

Now, the header file where ERESTARTNOHAND is defined says that user-space
should never see this errno.  Clearly, that's not as true as it could be.
I'm going to ask the LKML for more information about whether this should
make it to user-space, but thought I'd bring it up here as well.

Some options we have:

   Do nothing.  (I like this one :-)

   Catch and ignore 514.  (Only on Linux?  Far-reaching implications, I
   don't like this one).

   Use the libc sleep() call for integer time.sleep() values?  No guarantee
   that libc sleep() doesn't use select, though I've seen indications that
   it might use alarm.  I haven't looked at the libc source.

   Try to make time.sleep() smarter in the face of errnos.  Maybe try the
   sleep multiple times, check time.time() before and after the sleep to do
   multiple sleeps.  Could possibly even fail over to using /dev/rtc if
   select fails?

In general I'm reluctant to suggest a Python fix to this, but it's fairly
unexpected for time.sleep() to raise an exception...

Thoughts?

Sean
-- 
 I think the net needs some Viagra today.  It's just not performing...
                 -- Mike Loseke, 2000
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability


From glyph at divmod.com  Fri Jan 12 11:51:44 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 12 Jan 2007 10:51:44 -0000
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Message-ID: <20070112105144.17094.749039949.divmod.xquotient.1344@joule.divmod.com>

On 08:19 am, martin at v.loewis.de wrote:
>Georg Brandl schrieb:
>>> If Python 3.0 was simply a release which removed deprecated features,
>>> there would clearly be no issue. I would update my code in advance of
>>> the 3.0 release to not use any of those features being removed, and
>>> I'm all set. But that's not what I'm hearing. Python 3 is both adding
>>> new ways to do things, and removing the older way, in the same
>>> version, with no overlap. This makes me very anxious.
>>
>> It has always been planned that in those cases that allow it, the new way to do
>> it will be introduced in a 2.x release too, and the old way removed only in 3.x.
>
>What does that mean for the example James gave: if dict.items is going
>to be an iterator in 3.0, what 2.x version can make it return an
>iterator, when it currently returns a list?
>
>There simply can't be a 2.x version that *introduces* the new way, as it
>is not merely a new API, but a changed API.

The API isn't changed.  It's just dependent on its execution context in a confusing way.  That difference right now is being reflected as "2.x VM" vs. "3.0 VM" but there are other ways to reflect it more explicitly.  It would certainly be possible to have:

   from __future__ import items_is_iter

be the same as:

   __py3k_compat_items_is_iter__ = True

and have the 2.x series' items() method check the globals() of the calling scope to identify the return value of items() in that particular context.

If the actual data structures of module dictionaries and stack objects are too expensive there are other, similar things that could be done at the C level.  This implementation strategy is just the obvious thing that occurred to me after maybe 2 minutes of consideration.  I'm sure someone more familiar with the internals of Python could come up with something better.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070112/b132759e/attachment-0001.htm 

From martin at v.loewis.de  Sun Jan 14 02:22:05 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 14 Jan 2007 02:22:05 +0100
Subject: [Python-Dev] C standard io library, flush/read/write problem
In-Reply-To: <80718dd60701092120q51376eebkfed997f2b75390a8@mail.gmail.com>
References: <80718dd60701092120q51376eebkfed997f2b75390a8@mail.gmail.com>
Message-ID: <45A985BD.1040706@v.loewis.de>

Emerson Clarke schrieb:
> Add this to the Read() method before reading takes place:

If you don't submit it as a patch to sf.net/projects/python, it
is likely to get ignored.

Regards,
Martin

From martin at v.loewis.de  Sun Jan 14 02:24:15 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 14 Jan 2007 02:24:15 +0100
Subject: [Python-Dev] IOError errno=514 in time.sleep(1)
In-Reply-To: <20070110233028.GA10791@tummy.com>
References: <20070110233028.GA10791@tummy.com>
Message-ID: <45A9863F.8040106@v.loewis.de>

Sean Reifschneider schrieb:
> In general I'm reluctant to suggest a Python fix to this, but it's fairly
> unexpected for time.sleep() to raise an exception...
> 
> Thoughts?

Unless there is a reproducible test case, no action should be taken.
Once there is a test case, the specific action to take will become
clearer.

Regards,
Martin

From guido at python.org  Sun Jan 14 04:17:59 2007
From: guido at python.org (Guido van Rossum)
Date: Sat, 13 Jan 2007 19:17:59 -0800
Subject: [Python-Dev] buglet in long("123\0", 10)
Message-ID: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>

What's wrong with this session? :-)

Python 2.6a0 (trunk:53416, Jan 13 2007, 15:24:17)
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> int('123\0')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: null byte in argument for int()
>>> int('123\0', 10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '123\x00'
>>> long('123\0')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: null byte in argument for long()
>>> long('123\0', 10)
123L
>>>

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From ncoghlan at gmail.com  Sun Jan 14 08:06:45 2007
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Jan 2007 17:06:45 +1000
Subject: [Python-Dev] buglet in long("123\0", 10)
In-Reply-To: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>
References: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>
Message-ID: <45A9D685.3060604@gmail.com>

Guido van Rossum wrote:
>>> long('123\0', 10)
> 123L

Interesting - long_new goes through PyNumber_Long if no explicit base is 
provided. That does a pre-check for embedded NULLs in the input string. 
With an explicit base, however, PyLong_FromString is called directly. 
Since that API takes a char* string, it stops at the first embedded NULL:

 >>> long('123\0003', 10)
123L
 >>> long('123\00032', 10)
123L


So 'long_from_string' in abstract.c already has this problem solved - 
the helper function just needs to be moved into longobject.c so it can 
be used for explicit bases as well.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From lgautier at gmail.com  Sun Jan 14 10:20:59 2007
From: lgautier at gmail.com (Laurent Gautier)
Date: Sun, 14 Jan 2007 17:20:59 +0800
Subject: [Python-Dev] pydoc - revised
Message-ID: <27d1e6020701140120s753e3481w525db15f3330035b@mail.gmail.com>

Hi all,

This is a somehow an update to the thread on pydoc started by Ron.

Since the last entry we took the bull by the horns (so to speak), and
are seriously aiming at delivering something that can qualify as:
- a revision of the module pydoc
- something that will facilitate the inclusion of ideas explored in efforts
like the nice epydoc
- something that is in general easily extensible to answer most of the
need (both regarding the output that is returned, and what are the
capabilities regarding the exploration of the documentation).

We are probably setting the bar a little high, but we hope to get somewhere
at least with the first point thanks to a number of advices on this
list, some of which by pydoc's or ipython's original authors
(beside that, the bar could not be elsewhere since I am not a very
good limbo dancer ;-) ).

There is a sourceforge page for that project, since it lets us work with SVN
without risking to disturb a larger project, but we are ready to have that moved
to the sandbox in the python tree (although it would be neat to wait a
little until
there are more tests).
svn co https://pydoc-r.svn.sourceforge.net/svnroot/pydoc-r pydoc-r

Still, the thought process is not over and we were thinking of having
discussions
recorded either in the form of posts on python-dev, or say using the
mailing list
on the sourceforge page. An example of discussions we had are about
whether we should already name the module pydoc or an other name,
Posting on python-dev would give exposure to what is discussed, but at
the same could be perceived as off-topic until the module is in the
sandbox.
Posting on the sourceforge page would have pretty much opposite points
(no exposure, but no one would feel annoyed)

Any suggestion ?


Thanks,


Laurent

From skip at pobox.com  Sun Jan 14 14:36:55 2007
From: skip at pobox.com (skip at pobox.com)
Date: Sun, 14 Jan 2007 07:36:55 -0600
Subject: [Python-Dev] buglet in long("123\0", 10)
In-Reply-To: <45A9D685.3060604@gmail.com>
References: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>
	<45A9D685.3060604@gmail.com>
Message-ID: <17834.12791.420273.176115@montanaro.dyndns.org>


    Nick> With an explicit base, however, PyLong_FromString is called
    Nick> directly.  Since that API takes a char* string, it stops at the
    Nick> first embedded NULL:

    >>>> long('123\0003', 10)
    Nick> 123L
    >>>> long('123\00032', 10)
    Nick> 123L

    Nick> So 'long_from_string' in abstract.c already has this problem
    Nick> solved - the helper function just needs to be moved into
    Nick> longobject.c so it can be used for explicit bases as well.

That's a bug.  \0 is not a numeric character in any base, so the docs imply
that an exception should be raised:

  long([x[, radix]])
    Convert a string or number to a long integer. If the argument is a
    string, it must contain a possibly signed number of arbitrary size,
    possibly embedded in whitespace. The radix argument is interpreted in
    the same way as for int(), and may only be given when x is a
    string. Otherwise, the argument may be a plain or long integer or a
    floating point number, and a long integer with the same value is
    returned. Conversion of floating point numbers to integers truncates
    (towards zero). If no arguments are given, returns 0L.

The only nonnumeric characters which can occur are whitespace characters,
and they can only occur at the start or end of the string.  Unlike the
similar C functions atoi and atof conversion doesn't just continue until a
nonnumeric character is encountered.  There is no way to tell the caller
that part of the string wasn't munched.

Skip

From brett at python.org  Sun Jan 14 21:26:58 2007
From: brett at python.org (Brett Cannon)
Date: Sun, 14 Jan 2007 12:26:58 -0800
Subject: [Python-Dev] pydoc - revised
In-Reply-To: <27d1e6020701140120s753e3481w525db15f3330035b@mail.gmail.com>
References: <27d1e6020701140120s753e3481w525db15f3330035b@mail.gmail.com>
Message-ID: <bbaeab100701141226q568c6234kbb14e83b2a6b213f@mail.gmail.com>

On 1/14/07, Laurent Gautier <lgautier at gmail.com> wrote:
> Hi all,
>
> This is a somehow an update to the thread on pydoc started by Ron.
>
> Since the last entry we took the bull by the horns (so to speak), and
> are seriously aiming at delivering something that can qualify as:
> - a revision of the module pydoc
> - something that will facilitate the inclusion of ideas explored in efforts
> like the nice epydoc
> - something that is in general easily extensible to answer most of the
> need (both regarding the output that is returned, and what are the
> capabilities regarding the exploration of the documentation).
>
> We are probably setting the bar a little high, but we hope to get somewhere
> at least with the first point thanks to a number of advices on this
> list, some of which by pydoc's or ipython's original authors
> (beside that, the bar could not be elsewhere since I am not a very
> good limbo dancer ;-) ).
>
> There is a sourceforge page for that project, since it lets us work with SVN
> without risking to disturb a larger project, but we are ready to have that moved
> to the sandbox in the python tree (although it would be neat to wait a
> little until
> there are more tests).
> svn co https://pydoc-r.svn.sourceforge.net/svnroot/pydoc-r pydoc-r
>
> Still, the thought process is not over and we were thinking of having
> discussions
> recorded either in the form of posts on python-dev, or say using the
> mailing list
> on the sourceforge page. An example of discussions we had are about
> whether we should already name the module pydoc or an other name,
> Posting on python-dev would give exposure to what is discussed, but at
> the same could be perceived as off-topic until the module is in the
> sandbox.
> Posting on the sourceforge page would have pretty much opposite points
> (no exposure, but no one would feel annoyed)
>
> Any suggestion ?
>

Don't know how other list subscribers think, but if this is going to
take time it would be best to either use the SF one or get one on
mail.python.org until you are at a point that you are in the sandbox.
If we allowed you guys to talk here then everyone who wants to write
up a replacement for something in the stdlib would want to use
python-dev and that would be too much.

-Brett

From ironfroggy at gmail.com  Sun Jan 14 22:23:09 2007
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Sun, 14 Jan 2007 16:23:09 -0500
Subject: [Python-Dev] Rewrite of import in Python source (sans docs) is
	complete
In-Reply-To: <bbaeab100701051634g74a63c77lcc495f2ffc73bbd0@mail.gmail.com>
References: <bbaeab100701051634g74a63c77lcc495f2ffc73bbd0@mail.gmail.com>
Message-ID: <76fd5acf0701141323q481e3efar622db19e78b89bd1@mail.gmail.com>

I am really looking into get into hacking on CPython and I'm keenly
interested in your security work (my top reason for hoping i can make
PyCon. keeping fingers crossed!), so if you need help with this to
focus on other things, I'd be delighted to try my hand at the task. Do
you have some docs up anywhere of what directionyou hope this to go in
from here?

On 1/5/07, Brett Cannon <brett at python.org> wrote:
> Finally, after a few months worth of work, I have finally gotten far enough
> in my import rewrite that I am willing to stick my neck out and say it is
> semantically complete!  You can find it in the sandbox under import_in_py.
>
> So, details of this implementation.  I implemented PEP 302 importers/loaders
> for built-in, frozen, extension, .py, and .pyc files along with rewriting
> the steps __import__ goes through to do an import.  I also developed an API
> for .py/.pyc file handling so that there is a generic filesystem
> importer/loader and a separate handler for .py/.pyc files.  This should
> allow for (relatively) easy selective overriding of just how .py/.pyc files
> are stored ( e.g., introducing a database backend) or how variants on
> .py/.pyc files are handled (e.g., Quixote's .ptl format).
>
> This code has extensive tests and so I am fairly confident that it does what
> is expected of an import rewrite.  There are actually more lines in the test
> file than the implementation.  There is also a mock implementation used for
> testing.  Was interesting doing this in such a test-driven, XP style of only
> coding what I needed.
>
> I have run this code through the entire regression test suite and that is
> where you find out subtle differences between this implementation and the
> built-in import (you can see for yourself with the regrtest.sh shell
> script).  First test_pkg will fail because currently the new import adds a
> __loader__ attribute on all modules (that might change for security reasons)
> and test_pkg is an old, stdout comparing test.  Second, test_runpy fails
> because I have not implemented get_code on the filesystem loader which is
> required by runpy.  Both are shallow issues that can be dealt with.
>
> Third, and the hardest difference to deal with, is that you will get some
> warnings that print out that you normally don't see.  This is because
> warnings.warn and its stack_level argument don't have the effect people are
> used to when importing a deprecated module.  Before you could set
> stack_level to 2 and it would look like it came from the import statement.
> But now, with import written in Python and thus on the call stack compared
> to being in C and thus not showing up, two levels back is still in the
> import code.  I really don't know how this should be dealt with short of
> saying that the rule of thumb with 2 stack levels back for a warning does
> not work when done at the import level.
>
> It is not blazing fast at the moment.  Some things, like the built-in and
> frozen importers/loaders could be rewritten in C without huge issue.  I am
> also sure I have made some stupid design decisions at various points in the
> code.  But there is benchmarking code in the sandbox called importbench and
> it showed up a  10x speed slowdown on a Linux box I was using in mid to late
> December when doing a fresh import of certain types (I don't remember
> exactly which kind off the top of my head).
>
> Because of this current slowness I don't know if people want to rush into
> trying to make this the default import implementation quite yet or if this
> is not too big of a thing since the common case of pulling out of
> sys.modules is not that much slower.  I know I am currently not planning on
> devoting the time to bootstrap it in as I have my security work to finish
> first along with other Python stuff that seems more pressing to me.  And
> since (I think) I don't need to bootstrap it in order to finish my security
> work I can't justify spending work time on it.  But I can rearrange
> priorities if people really want to pursue this (especially if I can get
> some help with it).
>
> As with the module's name, it is currently named 'importer', but that is bad
> since it conflicts with the idea of importers from PEP 302.  I was thinking
> importlib, but I wanted to wait and see what other people thought.
>
> Don't know if you guys are okay with me checking this in without having it
> vetted by the community first like we prefer all new modules to do.  I have
> not done the LaTeX docs yet.
>
> I think that is all of the details that I can think of.  I am still working
> towards implementing the security needed so that an application that embeds
> Python can execute arbitrary code securely.  Giving a talk at PyCon on the
> topic for anyone interested.
>
>  Special thanks needs to go to Paul Moore who I talked to through most of
> the design of the code.  Nick Coghlan also provided some handy feedback.
> And Neal Norwitz for bugging about wanting something like this done.  Plus
> thanks to everyone who has shown support.
>
> -Brett
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
>
>
>


-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://ironfroggy-code.blogspot.com/

From steve at holdenweb.com  Sun Jan 14 22:38:53 2007
From: steve at holdenweb.com (Steve Holden)
Date: Sun, 14 Jan 2007 21:38:53 +0000
Subject: [Python-Dev] The bytes type
In-Reply-To: <ca471dc20701121100h741e8a6dr6e2121dc1873284f@mail.gmail.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com>	<5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net>	<eo63cp$1ak$1@sea.gmane.org>
	<45A744A8.40707@v.loewis.de>	<eo859h$fsa$1@sea.gmane.org>	<9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org>	<20070112165932.GA11501@localhost.localdomain>	<ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>	<20070112182248.GA15187@localhost.localdomain>	<00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
	<ca471dc20701121100h741e8a6dr6e2121dc1873284f@mail.gmail.com>
Message-ID: <45AAA2ED.6090601@holdenweb.com>

Guido van Rossum wrote:
> On 1/12/07, Raymond Hettinger <python at rcn.com> wrote:
>> [A.M. Kuchling]
>>> 2.6 wouldn't go changing existing APIs to begin requiring or returning
>>> the bytes type[*], of course, but extensions and new modules might use
>>> it.
>> The premise is dubious.
>>
>> If I am currently maintaining a module, why would I switch to a bytes type
>> and forgo compatibility with Py2.5 and prior?  I might as well just convert
>> it to run on Py3.0 and leave my Py2.5 code as-is for people who want to
>> run 2.x.
>>
>> If I'm writing a new module, what's the point of twisting myself into knots
>> to get it to run on both Py2.6 and Py3.0?  That just makes coding harder
>> (by limiting me to the intersection of the feature sets).
>>
>> I think we should draw a line in the sand and resolve not to garbage-up Py2.6.
>> The whole Py3.0 project is about eliminating cruft and being free of the
>> bonds of backwards compatibility.  Adding non-essential cruft to Py2.6
>> goes against that philosophy.
> 
> I'm not so sure, since 2.6 will likely be out and stable long before
> 3.0 gains much of a foothold. I believe the experiences with a similar
> approach in the Zope world for the 2->3 transition was overall a
> favorable one.
> 
> However, I'd be loathe to make any compromises in 3.0 in order to make
> life easier for 2.6. The burden must be on 2.6 (and 2.7-2.9), and if
> it's just impossible, that's too bad for them.
> 
This may have been discussed ob the Py3k list, and/or may be 
inappropriate for python-dev. I just wondered if you could indicate 
whether the developers of Jython, IronPython and PyPy have indicated any 
interest in and/or commitment to supporting Py3.0.

It's important that the development of 3.0 doesn't fragment the 
development community (not to mention the user community), and Jython is 
already aiming at a moving target in its attempts to catch up.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From nnorwitz at gmail.com  Mon Jan 15 00:14:25 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sun, 14 Jan 2007 15:14:25 -0800
Subject: [Python-Dev] buglet in long("123\0", 10)
In-Reply-To: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>
References: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>
Message-ID: <ee2a432c0701141514n391a707eh868c7e36a5c59fb@mail.gmail.com>

SVN rev 52305 resolved Bug #1545497: when given an explicit base,
int() did ignore NULs embedded in the string to convert.

However, the same fix wasn't applied for long().

n

On 1/13/07, Guido van Rossum <guido at python.org> wrote:
> What's wrong with this session? :-)
>
> Python 2.6a0 (trunk:53416, Jan 13 2007, 15:24:17)
> [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> int('123\0')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: null byte in argument for int()
> >>> int('123\0', 10)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: invalid literal for int() with base 10: '123\x00'
> >>> long('123\0')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: null byte in argument for long()
> >>> long('123\0', 10)
> 123L
> >>>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From brett at python.org  Mon Jan 15 01:24:19 2007
From: brett at python.org (Brett Cannon)
Date: Sun, 14 Jan 2007 16:24:19 -0800
Subject: [Python-Dev] Rewrite of import in Python source (sans docs) is
	complete
In-Reply-To: <76fd5acf0701141323q481e3efar622db19e78b89bd1@mail.gmail.com>
References: <bbaeab100701051634g74a63c77lcc495f2ffc73bbd0@mail.gmail.com>
	<76fd5acf0701141323q481e3efar622db19e78b89bd1@mail.gmail.com>
Message-ID: <bbaeab100701141624h41f6dac7g51273e433ac9364a@mail.gmail.com>

On 1/14/07, Calvin Spealman <ironfroggy at gmail.com> wrote:
> I am really looking into get into hacking on CPython

And doing the python-dev Summaries will definitely help with that.  =)

> and I'm keenly
> interested in your security work (my top reason for hoping i can make
> PyCon. keeping fingers crossed!),

I guess I better not screw up on the presentation.

> so if you need help with this to
> focus on other things, I'd be delighted to try my hand at the task. Do
> you have some docs up anywhere of what directionyou hope this to go in
> from here?
>

The progresss of the work is being kept in a Google Doc at
http://docs.google.com/View?docid=dg7fctr4_4d8tdbq .  As for the
overall design, you can read securing_python.txt in the bcannon-objcap
branch off the trunk.  It's a little outdated (mostly because it stems
back when I was trying to worry about supporting multiple interpreters
in a single process and thinking about how to expose all of it for
Python source code apps), but otherwise it is still accurate in terms
of how the security is being designed to work.

Overall the goal is to get it so that you can embed Python in your C
app but have it be secure so that if you have people use Python as a
plug-in or DSL you don't have to worry about them ruining your
machine.  I have ideas on how to extend the security to be more
general and such and possibly be used in Python source code apps, but
it looks like that will be outside of my Ph.D. thesis and thus I won't
be actively working on it (heck, this current work isn't either but I
have put so much time in at this point that my supervisor and I want
to get a published paper out of it eventually).

-Brett

> On 1/5/07, Brett Cannon <brett at python.org> wrote:
> > Finally, after a few months worth of work, I have finally gotten far enough
> > in my import rewrite that I am willing to stick my neck out and say it is
> > semantically complete!  You can find it in the sandbox under import_in_py.
> >
> > So, details of this implementation.  I implemented PEP 302 importers/loaders
> > for built-in, frozen, extension, .py, and .pyc files along with rewriting
> > the steps __import__ goes through to do an import.  I also developed an API
> > for .py/.pyc file handling so that there is a generic filesystem
> > importer/loader and a separate handler for .py/.pyc files.  This should
> > allow for (relatively) easy selective overriding of just how .py/.pyc files
> > are stored ( e.g., introducing a database backend) or how variants on
> > .py/.pyc files are handled (e.g., Quixote's .ptl format).
> >
> > This code has extensive tests and so I am fairly confident that it does what
> > is expected of an import rewrite.  There are actually more lines in the test
> > file than the implementation.  There is also a mock implementation used for
> > testing.  Was interesting doing this in such a test-driven, XP style of only
> > coding what I needed.
> >
> > I have run this code through the entire regression test suite and that is
> > where you find out subtle differences between this implementation and the
> > built-in import (you can see for yourself with the regrtest.sh shell
> > script).  First test_pkg will fail because currently the new import adds a
> > __loader__ attribute on all modules (that might change for security reasons)
> > and test_pkg is an old, stdout comparing test.  Second, test_runpy fails
> > because I have not implemented get_code on the filesystem loader which is
> > required by runpy.  Both are shallow issues that can be dealt with.
> >
> > Third, and the hardest difference to deal with, is that you will get some
> > warnings that print out that you normally don't see.  This is because
> > warnings.warn and its stack_level argument don't have the effect people are
> > used to when importing a deprecated module.  Before you could set
> > stack_level to 2 and it would look like it came from the import statement.
> > But now, with import written in Python and thus on the call stack compared
> > to being in C and thus not showing up, two levels back is still in the
> > import code.  I really don't know how this should be dealt with short of
> > saying that the rule of thumb with 2 stack levels back for a warning does
> > not work when done at the import level.
> >
> > It is not blazing fast at the moment.  Some things, like the built-in and
> > frozen importers/loaders could be rewritten in C without huge issue.  I am
> > also sure I have made some stupid design decisions at various points in the
> > code.  But there is benchmarking code in the sandbox called importbench and
> > it showed up a  10x speed slowdown on a Linux box I was using in mid to late
> > December when doing a fresh import of certain types (I don't remember
> > exactly which kind off the top of my head).
> >
> > Because of this current slowness I don't know if people want to rush into
> > trying to make this the default import implementation quite yet or if this
> > is not too big of a thing since the common case of pulling out of
> > sys.modules is not that much slower.  I know I am currently not planning on
> > devoting the time to bootstrap it in as I have my security work to finish
> > first along with other Python stuff that seems more pressing to me.  And
> > since (I think) I don't need to bootstrap it in order to finish my security
> > work I can't justify spending work time on it.  But I can rearrange
> > priorities if people really want to pursue this (especially if I can get
> > some help with it).
> >
> > As with the module's name, it is currently named 'importer', but that is bad
> > since it conflicts with the idea of importers from PEP 302.  I was thinking
> > importlib, but I wanted to wait and see what other people thought.
> >
> > Don't know if you guys are okay with me checking this in without having it
> > vetted by the community first like we prefer all new modules to do.  I have
> > not done the LaTeX docs yet.
> >
> > I think that is all of the details that I can think of.  I am still working
> > towards implementing the security needed so that an application that embeds
> > Python can execute arbitrary code securely.  Giving a talk at PyCon on the
> > topic for anyone interested.
> >
> >  Special thanks needs to go to Paul Moore who I talked to through most of
> > the design of the code.  Nick Coghlan also provided some handy feedback.
> > And Neal Norwitz for bugging about wanting something like this done.  Plus
> > thanks to everyone who has shown support.
> >
> > -Brett
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
> >
> >
> >
>
>
> --
> Read my blog! I depend on your acceptance of my opinion! I am interesting!
> http://ironfroggy-code.blogspot.com/
>

From ironfroggy at gmail.com  Mon Jan 15 04:30:08 2007
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Sun, 14 Jan 2007 22:30:08 -0500
Subject: [Python-Dev] buglet in long("123\0", 10)
In-Reply-To: <ee2a432c0701141514n391a707eh868c7e36a5c59fb@mail.gmail.com>
References: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>
	<ee2a432c0701141514n391a707eh868c7e36a5c59fb@mail.gmail.com>
Message-ID: <76fd5acf0701141930w6c619e76obd2ee28c20f3c9f2@mail.gmail.com>

Is it a more general problem that null-terminated strings are used
with data from strings we specifically allow to contain null bytes?
Perhaps a migration of *FromString() to *FromStringAndSize()
functions, or taking Python string object pointers, would be a more
general solution to set as a goal, so this sort of thing can't crop up
down the road, again.

I know I'm still very uninitiated in the internals of CPython, so
anyone please correct me if my thoughts here are against any on-going
policy or reasoning.

On 1/14/07, Neal Norwitz <nnorwitz at gmail.com> wrote:
> SVN rev 52305 resolved Bug #1545497: when given an explicit base,
> int() did ignore NULs embedded in the string to convert.
>
> However, the same fix wasn't applied for long().
>
> n
>
> On 1/13/07, Guido van Rossum <guido at python.org> wrote:
> > What's wrong with this session? :-)
> >
> > Python 2.6a0 (trunk:53416, Jan 13 2007, 15:24:17)
> > [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> int('123\0')
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > ValueError: null byte in argument for int()
> > >>> int('123\0', 10)
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > ValueError: invalid literal for int() with base 10: '123\x00'
> > >>> long('123\0')
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > ValueError: null byte in argument for long()
> > >>> long('123\0', 10)
> > 123L
> > >>>
> >
> > --
> > --Guido van Rossum (home page: http://www.python.org/~guido/)
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
> >
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
>


-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://ironfroggy-code.blogspot.com/

From guido at python.org  Mon Jan 15 06:16:52 2007
From: guido at python.org (Guido van Rossum)
Date: Sun, 14 Jan 2007 21:16:52 -0800
Subject: [Python-Dev] buglet in long("123\0", 10)
In-Reply-To: <76fd5acf0701141930w6c619e76obd2ee28c20f3c9f2@mail.gmail.com>
References: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>
	<ee2a432c0701141514n391a707eh868c7e36a5c59fb@mail.gmail.com>
	<76fd5acf0701141930w6c619e76obd2ee28c20f3c9f2@mail.gmail.com>
Message-ID: <ca471dc20701142116n695c4089y6f7f19fba7c9ce83@mail.gmail.com>

On 1/14/07, Calvin Spealman <ironfroggy at gmail.com> wrote:
> Is it a more general problem that null-terminated strings are used
> with data from strings we specifically allow to contain null bytes?
> Perhaps a migration of *FromString() to *FromStringAndSize()
> functions, or taking Python string object pointers, would be a more
> general solution to set as a goal, so this sort of thing can't crop up
> down the road, again.

Most of the time this is taken care of by the argument type codes
passed to PyArg_ParseTuple() and friends. If you use 's' then it
assumes the string is for consumption of C code that uses
null-termination, and it checks that there are no null bytes. Try
open("foo\0").

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From thomas at python.org  Mon Jan 15 14:02:03 2007
From: thomas at python.org (Thomas Wouters)
Date: Mon, 15 Jan 2007 05:02:03 -0800
Subject: [Python-Dev] The bytes type
In-Reply-To: <6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
Message-ID: <9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>

On 1/12/07, Mike Orr <sluggoster at gmail.com> wrote:
>
> On 1/12/07, glyph at divmod.com <glyph at divmod.com> wrote:
> > The benefit (to me, and to many others) of 3.x over 2.x is the promise
> of
> > more future maintenance, not the lack of cruft.
>
> The benefit (to me, and to many others) of 3.x over 2.x is the promise
> of getting rid of cruft.

If we're going to re-add cruft for the sake of temporary
> compatibility, we may as well just stick with 2.x.  You have to take a
> quantum leap sometimes or you end up working around the same old
> mistakes.


There seems to be rather a lot of confusion. No one is suggesting
Python 3.0be anything less for the sake of backward compatibility.
Instead, it has
been suggested Python 2.6 (and possibly 2.7) be something *more* in order to
provide for an easier upgrade path. No compromises in Python 3.0.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070115/725469b2/attachment.htm 

From thomas at python.org  Mon Jan 15 14:10:15 2007
From: thomas at python.org (Thomas Wouters)
Date: Mon, 15 Jan 2007 05:10:15 -0800
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <45A8FFE0.5010605@v.loewis.de>
References: <20070112105144.17094.749039949.divmod.xquotient.1344@joule.divmod.com>
	<45A8FFE0.5010605@v.loewis.de>
Message-ID: <9e804ac0701150510q2ab56b51t67f2931e00129e06@mail.gmail.com>

On 1/13/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>
> glyph at divmod.com schrieb:
> >  It would certainly be possible to have:
> >
> >    from __future__ import items_is_iter
> >
> > be the same as:
> >
> >    __py3k_compat_items_is_iter__ = True
> >
> > and have the 2.x series' items() method check the globals() of the
> > calling scope to identify the return value of items() in that particular
> > context.
>
> Why do you think that this would be that certainly possible?
> I cannot imagine an efficient implementation.


Ah, but can you imagine an inefficient one? If so, we're no longer arguing
about whether it's possible, but whether we want to pay the price. Given
that the dict.keys/values/items change hasn't been written and only broadly
designed, it's hard to tell *right now* what we can do to ease the
transition. I do believe we should be prepared to as much as we can, in
Python 2.6 and 2.7, to ease it (and the other transitions.) That means the
2to3 tool and warnings in appropriate places, and anything else we can do.
Including, if necessary, optionally futzing with the returned type of
dict.keys/values/items in Python 2.6, in order to make it a list-like type
that whines when it is being operated on in ways the 3.0 type won't allow.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070115/6965a6e5/attachment.html 

From martin at v.loewis.de  Mon Jan 15 23:08:21 2007
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 15 Jan 2007 23:08:21 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <9e804ac0701150510q2ab56b51t67f2931e00129e06@mail.gmail.com>
References: <20070112105144.17094.749039949.divmod.xquotient.1344@joule.divmod.com>	
	<45A8FFE0.5010605@v.loewis.de>
	<9e804ac0701150510q2ab56b51t67f2931e00129e06@mail.gmail.com>
Message-ID: <45ABFB55.2090708@v.loewis.de>

>     Why do you think that this would be that certainly possible?
>     I cannot imagine an efficient implementation.
> 
> 
> Ah, but can you imagine an inefficient one? 

I think so (although one can never know until it's implemented).

> If so, we're no longer
> arguing about whether it's possible, but whether we want to pay the
> price. Given that the dict.keys/values/items change hasn't been written
> and only broadly designed, it's hard to tell *right now* what we can do
> to ease the transition. I do believe we should be prepared to as much as
> we can, in Python 2.6 and 2.7, to ease it (and the other transitions.)
> That means the 2to3 tool and warnings in appropriate places, and
> anything else we can do.

I don't think we should do *everything* we can. If 2.6 performance will
suffer significantly to support 3.x warnings, the price would be too
high.

Regards,
Martin

From martin at v.loewis.de  Mon Jan 15 23:40:22 2007
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 15 Jan 2007 23:40:22 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <20070115182102.17094.456325909.divmod.xquotient.2056@joule.divmod.com>
References: <20070115182102.17094.456325909.divmod.xquotient.2056@joule.divmod.com>
Message-ID: <45AC02D6.7070900@v.loewis.de>

glyph at divmod.com schrieb:
> Also, the exact strategy I suggested could be implemented in various
> ways that might be efficient.  Here are a few ways it might be made more
> efficient than the straw many of one extra dict lookup per call to
> keys() et. al.:

I'm not saying that an efficient implementation is impossible, but I
still don't see it. The ones you suggest don't work that well:

>  * Don't make it a dictionary key; make it an attribute of the module at
> the C level.

Unfortunately, in a function, there is no reference to the module it
came from. There is __module__ (on the function, not on the code
object), but that is a string.

>  * Annotate the code object in some way that makes code compiled in that
> scope set a global variable to enable the compatibility; protect it with
> the GIL.

This I don't quite understand: what has the GIL to do with that? Are you
suggesting to hold the GIL while the entire function is running?
That would restrict concurrency even more than the GIL already restricts
it. Also, what about nested functions spanning multiple modules (with
different settings)?

>  * These methods are already function pointers in a data structure in C;
> actually have a py->py3k mode switch which swaps out the C function
> pointers to the appropriate implementation for the calling code.

That doesn't work: The items method is per dictionary (or actually per
type, i.e. for the dict type). The same dictionary, when accessed from
different modules, should have .items() behave differently. Wrt. that
property:

# a.py
from __future__ import items_is_iter

def f(d):
  return d.items

# b.oy
import a
d = {}
print a.f(d)()

What should that print?

I think I take back my earlier suggestion that I can imagine an
inefficient implementation. I cannot imagine any reasonable behaviour
of a per-module change of methods on dictionary objects.

> Do I just suffer from having an overactive imagination?  Are all of
> these implementation strategies impossible for some reason, and there
> are no others?

Yes, and yes.

Regards,
Martin

From foom at fuhm.net  Tue Jan 16 10:25:22 2007
From: foom at fuhm.net (James Y Knight)
Date: Tue, 16 Jan 2007 04:25:22 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <45A82722.8060409@ronadam.com>
References: <20070110214150.17094.1519280154.divmod.xquotient.1011@joule.divmod.com><79990c6b0701101510n5d0151bdqee45ff7e6888f034@mail.gmail.com><45A57AE0.2060600@benjiyork.com><5DB40860-80A0-4281-BBB7-E220F2DDB8E7@fuhm.net><eo63cp$1ak$1@sea.gmane.org>	<45A744A8.40707@v.loewis.de><eo859h$fsa$1@sea.gmane.org><9FE39FBA-759F-48B5-B6C9-FB694CFE1AED@python.org><20070112165932.GA11501@localhost.localdomain><ca471dc20701120951r5a22ec66s7099a63a9cc935c8@mail.gmail.com>	<20070112182248.GA15187@localhost.localdomain>
	<00fe01c7367a$5baccdb0$ea146b0a@RaymondLaptop1>
	<45A82722.8060409@ronadam.com>
Message-ID: <924C317C-D8CD-4DB2-87DE-305A4756CEE3@fuhm.net>

On Jan 12, 2007, at 7:26 PM, Ron Adam wrote:
> For me, the thing that will make porting 2.x to 3.x code easy is to  
> make python
> 3.0 as clean and organized as you can with excellent  
> documentation.  Half-way
> and duel-way approaches will probably not help me as much as this.
>
> Most of the difficulty I have in converting programs is in finding the
> information I need.  The actual editing is not a problem.

Yes, the actual editing is not a problem, as long as you only need to  
do it once. The problem is when you have a module/program/whatever  
that you need to provide a release of for 2.5 and would also like to  
provide a release of for 3.0. Then, the editing becomes a problem,  
because you have to do it again for every change you make.

James

From foom at fuhm.net  Tue Jan 16 10:37:11 2007
From: foom at fuhm.net (James Y Knight)
Date: Tue, 16 Jan 2007 04:37:11 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
Message-ID: <FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>

On Jan 15, 2007, at 8:02 AM, Thomas Wouters wrote:
>> The benefit (to me, and to many others) of 3.x over 2.x is the  
>> promise
>> of getting rid of cruft.
>> If we're going to re-add cruft for the sake of temporary
>> compatibility, we may as well just stick with 2.x.  You have to  
>> take a
>> quantum leap sometimes or you end up working around the same old
>> mistakes.
>>
> There seems to be rather a lot of confusion. No one is suggesting  
> Python 3.0 be anything less for the sake of backward compatibility.  
> Instead, it has been suggested Python 2.6 (and possibly 2.7) be  
> something *more* in order to provide for an easier upgrade path. No  
> compromises in Python 3.0.
>

True: nobody is suggesting python 3.0 be anything less. But, I am  
indeed suggesting that Python 3.0 be something *more*: I am  
suggesting that people keep in mind the ease of writing of a program  
which can run on both 2.5 and 3.0. And wherever possible, act so as  
to preserve that ease. That may indeed involve a "compromise" in 3.0.

Given that Python 3.0 is supposed to be released in ~ 1.5 years (I  
guess), around the same time as the 2.6 release, I do think 3.0 ought  
to have a compatibility story that doesn't depend on 2.6. My goal  
here is not to get Py3.0 to be backwards compatible, it's to get the  
intersection of 2.5 and 3.0 to be large enough that I can explicitly  
write code to that intersection without having to tear my hair out  
and make the code incredibly gross (or worse yet, not be able to do  
it at all). See previous iteritems() example.

I really don't see why there's such an urgency for removing things  
IMMEDIATELY in Python 3 that don't have a suitable replacement in an  
already released version. There's plenty to remove which already does  
have a suitable replacement. I think the major cruft all falls into  
this category (unicode strings already exist, newstyle classes  
already exist, non-string exceptions already exist. The 'bytes' type  
doesn't exist yet, but I expect can be introduced in such a way that  
it has a useful intersection of functionality with the current method  
of using 'str' to hold bytes.)

I'd like to remind everyone that 3.0 won't be the last version of  
python ever released (at least, I hope!). Where Py 3.0 introduces a  
brand-new way to do something, the old can be removed at a future  
date. Mark it as deprecated, so that nobody will use it in new code,  
and remove it a few more versions down the line. Getting rid of cruft  
is great...but you can't properly call something "cruft" when it's  
the only way.

Allowances for writing portable code could make a big difference in  
speed of adoption. I expect Python 3.0 will be a "quantum leap", even  
if people don't have to rewrite everything from scratch or fork their  
codebases to keep compatibility with both releases. There's plenty to  
look forward to, and I'd love to actually be able to actually use it.  
But I need to be able to run 3rd party libraries, and the 3rd party  
libraries must still remain compatible with earlier releases. If  
those requirements are incompatible with using Py 3.0, it'll be a shame.

James

From amk at amk.ca  Tue Jan 16 12:29:48 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Tue, 16 Jan 2007 06:29:48 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <45AC02D6.7070900@v.loewis.de>
References: <20070115182102.17094.456325909.divmod.xquotient.2056@joule.divmod.com>
	<45AC02D6.7070900@v.loewis.de>
Message-ID: <20070116112948.GA14364@andrew-kuchlings-computer.local>

On Mon, Jan 15, 2007 at 11:40:22PM +0100, "Martin v. L??wis" wrote:
> > Do I just suffer from having an overactive imagination?  Are all of
> > these implementation strategies impossible for some reason, and there
> > are no others?
> 
> Yes, and yes.

What about turning all references to obj.items into the equivalent
bytecode for this:

if isinstance(obj, dict): # XXX should this be 'type(obj) is dict'?
   if <2.x behaviour>:   _temp = obj.items
   elif <3.x behaviour>: _temp = obj.iteritems
else:
   _temp = obj.items

Ugly; very ugly.

--amk


From guido at python.org  Tue Jan 16 16:47:24 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Jan 2007 07:47:24 -0800
Subject: [Python-Dev] The bytes type
In-Reply-To: <FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
Message-ID: <ca471dc20701160747g4674f6fcj2e5811829e9f0eb8@mail.gmail.com>

On 1/16/07, James Y Knight <foom at fuhm.net> wrote:
> On Jan 15, 2007, at 8:02 AM, Thomas Wouters wrote:
> > There seems to be rather a lot of confusion. No one is suggesting
> > Python 3.0 be anything less for the sake of backward compatibility.
> > Instead, it has been suggested Python 2.6 (and possibly 2.7) be
> > something *more* in order to provide for an easier upgrade path. No
> > compromises in Python 3.0.
>
> True: nobody is suggesting python 3.0 be anything less. But, I am
> indeed suggesting that Python 3.0 be something *more*: I am
> suggesting that people keep in mind the ease of writing of a program
> which can run on both 2.5 and 3.0. And wherever possible, act so as
> to preserve that ease. That may indeed involve a "compromise" in 3.0.

I'm not keen on compromises in 3.0, but without specific proposals I
don't see why we're arguing. So, please, what specific thing(s) are
you proposing we do in 3.0? Please make a list of specifics rather
than attempting at specifying a general rule to match things that
could go into the list; you've tried the latter and I still don't know
what you want.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Tue Jan 16 18:06:12 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 16 Jan 2007 12:06:12 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <ca471dc20701160747g4674f6fcj2e5811829e9f0eb8@mail.gmail.co
 m>
References: <FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
Message-ID: <5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>

At 07:47 AM 1/16/2007 -0800, Guido van Rossum wrote:
>On 1/16/07, James Y Knight <foom at fuhm.net> wrote:
> > On Jan 15, 2007, at 8:02 AM, Thomas Wouters wrote:
> > > There seems to be rather a lot of confusion. No one is suggesting
> > > Python 3.0 be anything less for the sake of backward compatibility.
> > > Instead, it has been suggested Python 2.6 (and possibly 2.7) be
> > > something *more* in order to provide for an easier upgrade path. No
> > > compromises in Python 3.0.
> >
> > True: nobody is suggesting python 3.0 be anything less. But, I am
> > indeed suggesting that Python 3.0 be something *more*: I am
> > suggesting that people keep in mind the ease of writing of a program
> > which can run on both 2.5 and 3.0. And wherever possible, act so as
> > to preserve that ease. That may indeed involve a "compromise" in 3.0.
>
>I'm not keen on compromises in 3.0, but without specific proposals I
>don't see why we're arguing. So, please, what specific thing(s) are
>you proposing we do in 3.0? Please make a list of specifics rather
>than attempting at specifying a general rule to match things that
>could go into the list; you've tried the latter and I still don't know
>what you want.

I think what he's saying boils down to two things:

* Don't remove any feature for which an alternative doesn't already exist 
in 2.5

* Don't change APIs (e.g. items())

Now, I'm not sure these goals are achievable with respect to 2.5.  I think 
we'd be better off adding compatibility features in 2.6.

To be honest, the items() and keys() thing personally baffles me.  If 
they're supposed to be *views* on the underlying dictionary, wouldn't it 
make more sense for them to be *attributes* instead of methods?  I.e. 
dict.items and dict.keys.  Then, we could provide that feature in 2.6, and 
drop the availability of the callable forms in 3.0.

Then you could write code like:

     for k,v in somedict.items:
         ...

And have it work in 2.6 and 3.0.  Meanwhile, .items() would still return a 
list in 2.6 (but be warnable about with a -3 switch), but go away entirely 
in 3.0.

It's not a panacea, but at least would make it *possible* to write code 
that runs on both 3.0 and some 2.x version.

Without having at least *some* 2.x version that can run 3.x code, I think 
there is little chance of 3.0 becoming viable.  You've been comparing this 
to Zope 2/Zope 3, but in that world there is something called "Five" that 
lets you do Zope 3 things inside of Zope 2, so you have some chance of 
porting your code in an incremental fashion, without having to leap 
everything over in one go.

You've cited JoS on your decision not to do 3.0 as a ground-up rewrite, so 
perhaps you'll find this other JoS article relevant here:

"""It turns out that what was stopping people from switching to Excel was 
that everybody else they worked with was still using Lotus 123. They didn't 
want a product that would create spreadsheets that nobody else could read: 
a classic Chicken and Egg problem. When you're the lone Excel fan in a 
company where everyone else is using 123, even if you love Excel, you can't 
switch until you can participate in the 123 ecology."""

http://www.joelonsoftware.com/articles/fog0000000052.html

The analogy isn't perfect, because we are not so much trying to provide 
backward compatibility in "Excel" as to add *forward* compatibility to 
"123", but you get the idea.


From guido at python.org  Tue Jan 16 18:50:59 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Jan 2007 09:50:59 -0800
Subject: [Python-Dev] The bytes type
In-Reply-To: <5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
Message-ID: <ca471dc20701160950n46fd6bd2y3e243663487fc41@mail.gmail.com>

On 1/16/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 07:47 AM 1/16/2007 -0800, Guido van Rossum wrote:
> >On 1/16/07, James Y Knight <foom at fuhm.net> wrote:
> > > On Jan 15, 2007, at 8:02 AM, Thomas Wouters wrote:
> > > > There seems to be rather a lot of confusion. No one is suggesting
> > > > Python 3.0 be anything less for the sake of backward compatibility.
> > > > Instead, it has been suggested Python 2.6 (and possibly 2.7) be
> > > > something *more* in order to provide for an easier upgrade path. No
> > > > compromises in Python 3.0.
> > >
> > > True: nobody is suggesting python 3.0 be anything less. But, I am
> > > indeed suggesting that Python 3.0 be something *more*: I am
> > > suggesting that people keep in mind the ease of writing of a program
> > > which can run on both 2.5 and 3.0. And wherever possible, act so as
> > > to preserve that ease. That may indeed involve a "compromise" in 3.0.
> >
> >I'm not keen on compromises in 3.0, but without specific proposals I
> >don't see why we're arguing. So, please, what specific thing(s) are
> >you proposing we do in 3.0? Please make a list of specifics rather
> >than attempting at specifying a general rule to match things that
> >could go into the list; you've tried the latter and I still don't know
> >what you want.
>
> I think what he's saying boils down to two things:
>
> * Don't remove any feature for which an alternative doesn't already exist
> in 2.5

Without specific features named I don't understand what this means.

> * Don't change APIs (e.g. items())

That's a non-starter. "Don't change APIs" means essentially nothing
can be changed.

> Now, I'm not sure these goals are achievable with respect to 2.5.  I think
> we'd be better off adding compatibility features in 2.6.
>
> To be honest, the items() and keys() thing personally baffles me.  If
> they're supposed to be *views* on the underlying dictionary, wouldn't it
> make more sense for them to be *attributes* instead of methods?  I.e.
> dict.items and dict.keys.

Let's have that discussion on the py3k list. We went through that and
I definitely don't want them to be just attributes (it was proposed
several times before). I'd be happy to come up with a rationale, but
not here.

> Then, we could provide that feature in 2.6, and
> drop the availability of the callable forms in 3.0.

This is clearly an example of constraining 3.0 in order to provide
better compatibility, which is a contradiction of 3.0's very
existence.

[...]

> It's not a panacea, but at least would make it *possible* to write code
> that runs on both 3.0 and some 2.x version.

Actually it's very easy to write code using keys(), items() and
values() that works as well in 2.2 as it works in 3.0: never use the
iter* variants, and don't sweat the performance costs of creating a
list so much. If you can't afford to create a list, iterate over the
dict itself, which will give you the keys in both versions.

> Without having at least *some* 2.x version that can run 3.x code, I think
> there is little chance of 3.0 becoming viable.  You've been comparing this
> to Zope 2/Zope 3, but in that world there is something called "Five" that
> lets you do Zope 3 things inside of Zope 2, so you have some chance of
> porting your code in an incremental fashion, without having to leap
> everything over in one go.

I'm not aware of compromises to 3.0's architecture that were made in
order to ease the transition though -- all of the burden was placed on
Zope 2.x (though perhaps some optional compatibility packages were
also added to Zope 3.0?).

> You've cited JoS on your decision not to do 3.0 as a ground-up rewrite, so
> perhaps you'll find this other JoS article relevant here:
>
> """It turns out that what was stopping people from switching to Excel was
> that everybody else they worked with was still using Lotus 123. They didn't
> want a product that would create spreadsheets that nobody else could read:
> a classic Chicken and Egg problem. When you're the lone Excel fan in a
> company where everyone else is using 123, even if you love Excel, you can't
> switch until you can participate in the 123 ecology."""
>
> http://www.joelonsoftware.com/articles/fog0000000052.html
>
> The analogy isn't perfect, because we are not so much trying to provide
> backward compatibility in "Excel" as to add *forward* compatibility to
> "123", but you get the idea.

Such analogies are hard to get right; in this case he's talking about
file interchange formats. Perhaps the closest analogy is pickling.
AFAIK pickles are exchangeable between the two now; if at some point
they grow apart, it shouldn't be too hard to add a new default
pickling protocol to 3.0 and add support for it to 2.6.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tcdelaney at optusnet.com.au  Tue Jan 16 18:59:15 2007
From: tcdelaney at optusnet.com.au (Tim Delaney)
Date: Wed, 17 Jan 2007 04:59:15 +1100
Subject: [Python-Dev] dict.items as attributes [Was:  The bytes type]
References: <FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net><20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com><6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com><9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com><FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
Message-ID: <00c601c73998$0a2f1b20$f100a8c0@ryoko>

Phillip J. Eby wrote:

> To be honest, the items() and keys() thing personally baffles me.  If
> they're supposed to be *views* on the underlying dictionary, wouldn't
> it
> make more sense for them to be *attributes* instead of methods?  I.e.
> dict.items and dict.keys.  Then, we could provide that feature in
> 2.6, and
> drop the availability of the callable forms in 3.0.
>
> Then you could write code like:
>
>     for k,v in somedict.items:
>         ...
>
> And have it work in 2.6 and 3.0.  Meanwhile, .items() would still
> return a
> list in 2.6 (but be warnable about with a -3 switch), but go away
> entirely
> in 3.0.

I think this comes down to whether or not the views returned have any 
independent state. There's something that tells me that attributes (even 
properties) should not return different objects with independent state - 
working on two views obtained from the same dictionary property should 
either work identically to working on one view bound to two names, or they 
should not be obtained from a property.

But unless I'm mistaken, everything done to a view would pass through to the 
dict, or result in another object that has independent state (e.g. iter()) 
so the effect of working on two views of a dict *would* be identical to 
working on two names to the same view. The only case I can think of for 
which we might want to hold state in the view is for detecting concurrent 
modification - I know that iterators should throw exceptions in this case, 
but I can't remember what (if anything) was decided for views.

Tim Delaney 


From pje at telecommunity.com  Tue Jan 16 19:51:27 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 16 Jan 2007 13:51:27 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <ca471dc20701160950n46fd6bd2y3e243663487fc41@mail.gmail.com
 >
References: <5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
	<20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20070116134528.0288cb58@sparrow.telecommunity.com>

At 09:50 AM 1/16/2007 -0800, Guido van Rossum wrote:
>Actually it's very easy to write code using keys(), items() and
>values() that works as well in 2.2 as it works in 3.0: never use the
>iter* variants, and don't sweat the performance costs of creating a
>list so much. If you can't afford to create a list, iterate over the
>dict itself, which will give you the keys in both versions.

Actually, the main reason I use items() or keys() is because I want to 
mutate the dictionary, or because something else is going to mutate it 
while I'm looping over it.  But yeah, that discussion should be on Py3K.


>>Without having at least *some* 2.x version that can run 3.x code, I think
>>there is little chance of 3.0 becoming viable.  You've been comparing this
>>to Zope 2/Zope 3, but in that world there is something called "Five" that
>>lets you do Zope 3 things inside of Zope 2, so you have some chance of
>>porting your code in an incremental fashion, without having to leap
>>everything over in one go.
>
>I'm not aware of compromises to 3.0's architecture that were made in
>order to ease the transition though -- all of the burden was placed on
>Zope 2.x (though perhaps some optional compatibility packages were
>also added to Zope 3.0?).

Right, which is why I don't think we should worry about 2.5 as much as 2.6.


>>The analogy isn't perfect, because we are not so much trying to provide
>>backward compatibility in "Excel" as to add *forward* compatibility to
>>"123", but you get the idea.
>
>Such analogies are hard to get right; in this case he's talking about
>file interchange formats. Perhaps the closest analogy is pickling.

Actually, I meant the analogy for *code*.  If you can't write code that 
runs on some 2.x + 3.x combination, you have no way to cross the chasm.  If 
you're writing "spreadsheets for Excel" (code that runs on 3.x), you need 
to be able to "share them with 1-2-3 users" (run that code on 2.x).


From foom at fuhm.net  Tue Jan 16 19:52:12 2007
From: foom at fuhm.net (James Y Knight)
Date: Tue, 16 Jan 2007 13:52:12 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
References: <FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
Message-ID: <85A3FBBE-C077-4F7E-B95E-82D99CE87AB1@fuhm.net>

On Jan 16, 2007, at 12:06 PM, Phillip J. Eby wrote:
> At 07:47 AM 1/16/2007 -0800, Guido van Rossum wrote:
>> I'm not keen on compromises in 3.0, but without specific proposals I
>> don't see why we're arguing. So, please, what specific thing(s) are
>> you proposing we do in 3.0? Please make a list of specifics rather
>> than attempting at specifying a general rule to match things that
>> could go into the list; you've tried the latter and I still don't  
>> know
>> what you want.
>
> I think what he's saying boils down to two things:
>
> * Don't remove any feature for which an alternative doesn't already  
> exist in 2.5

Yes, this is it. As a refinement: if the New Way can easily be  
backported to 2.5, that's somewhat acceptable as well (but not *as*  
good, as it requires everyone wanting to write the portable code to  
include the backport or depend on an external package providing it).

> * Don't change APIs (e.g. items())

No: changing APIs is fine just so long as the API being changed can  
already be replaced by something better. E.g. changing d.items() is  
fine with me: it's trivially replaced in 2.5 with list(d.iteritems 
()). But changing/removing d.iteritems() is not okay, as it's not  
easily replaced by anything that will work well in both versions.

> Now, I'm not sure these goals are achievable with respect to 2.5.   
> I think we'd be better off adding compatibility features in 2.6.

I don't like the proposals to only have the ability to be compatible  
with 2.6, because that significantly delays the ability to port to  
3.0. If it's possible to be compatible between 2.5/3.0, I can start  
doing the work porting to 3.0 soon after it comes out, even while  
remaining compatible with a widely-deployed version of python. If the  
compatibility only works with 2.6/3.0, it will be another 2-3 years  
after 2.6 comes out before it's widely deployed enough to require it  
and thus that much longer before porting can start.

Really I'm interested in 2.4/3.0 portability, but I quite expect that  
if I can get 2.5/3.0, I can easily enough deal with the changes  
between 2.4/2.5 myself, so I'm not going to bother python-dev about  
that. :)


James

From ironfroggy at gmail.com  Tue Jan 16 19:57:18 2007
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Tue, 16 Jan 2007 13:57:18 -0500
Subject: [Python-Dev] buglet in long("123\0", 10)
In-Reply-To: <ee2a432c0701151129k67d09f74i135fb795d953ac08@mail.gmail.com>
References: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>
	<ee2a432c0701141514n391a707eh868c7e36a5c59fb@mail.gmail.com>
	<76fd5acf0701141930w6c619e76obd2ee28c20f3c9f2@mail.gmail.com>
	<ee2a432c0701151129k67d09f74i135fb795d953ac08@mail.gmail.com>
Message-ID: <76fd5acf0701161057m37b72af5w85aafd0edc672db4@mail.gmail.com>

Added a check in test_long.LongTest.test_misc() that long("123\0", 10)
fails properly and adapted the patch to int_new to long_new. I get
this weird feeling that if its impossible for the function
(PyLong_FromString) to know if its being given bad data, having know
way to know if the string is supposed to continue past the zero-byte,
then doesn't it make sense to say that the function by design is
broken? Anyway, here is the patch.

Index: Objects/longobject.c
===================================================================
--- Objects/longobject.c        (revision 53406)
+++ Objects/longobject.c        (working copy)
@@ -3287,10 +3287,27 @@
                return PyLong_FromLong(0L);
        if (base == -909)
                return PyNumber_Long(x);
-       else if (PyString_Check(x))
+       if (PyString_Check(x)) {
+               /* Since PyLong_FromString doesn't have a length parameter,
+                * check here for possible NULs in the string. */
+               char *string = PyString_AS_STRING(x);
+               if (strlen(string) != PyString_Size(x)) {
+                       /* create a repr() of the input string,
+                        * just like PyLong_FromString does. */
+                       PyObject *srepr;
+                       srepr = PyObject_Repr(x);
+                       if (srepr == NULL)
+                               return NULL;
+                       PyErr_Format(PyExc_ValueError,
+                            "Invalid literal for long() with base %d: %s",
+                            base, PyString_AS_STRING(srepr));
+                       Py_DECREF(srepr);
+                       return NULL;
+               }
                return PyLong_FromString(PyString_AS_STRING(x), NULL, base);
+       }
 #ifdef Py_USING_UNICODE
-       else if (PyUnicode_Check(x))
+       if (PyUnicode_Check(x))
                return PyLong_FromUnicode(PyUnicode_AS_UNICODE(x),
                                          PyUnicode_GET_SIZE(x),
                                          base);
@@ -3439,3 +3456,4 @@
        long_new,                               /* tp_new */
        PyObject_Del,                           /* tp_free */
 };
+
Index: Lib/test/test_long.py
===================================================================
--- Lib/test/test_long.py       (revision 53406)
+++ Lib/test/test_long.py       (working copy)
@@ -299,6 +299,8 @@
         slicemin, slicemax = X()[-2L**100:2L**100]
         self.assertEqual(X()[slicemin:slicemax], (slicemin, slicemax))

+        self.assertRaises(ValueError, long, "123\0", 10)
+
 # ----------------------------------- tests of auto int->long conversion

     def test_auto_overflow(self):


### END PATCH

On 1/15/07, Neal Norwitz <nnorwitz at gmail.com> wrote:
> Calvin,
>
> Was Guido's answer enough or did you want some more?  If you take a
> look at the svn rev I sent, I think it should be pretty easy to come
> up with a patch for long too.
>
> Getting the size won't help (exactly), since the problem is the diff
> between strlen() and the size.  You can see this in the int() fix.
>
> Let me know if you have any questions.  It would be great if you could
> produce a patch.
>
> Cheers,
> n
> --
> On 1/14/07, Calvin Spealman <ironfroggy at gmail.com> wrote:
> > Is it a more general problem that null-terminated strings are used
> > with data from strings we specifically allow to contain null bytes?
> > Perhaps a migration of *FromString() to *FromStringAndSize()
> > functions, or taking Python string object pointers, would be a more
> > general solution to set as a goal, so this sort of thing can't crop up
> > down the road, again.
> >
> > I know I'm still very uninitiated in the internals of CPython, so
> > anyone please correct me if my thoughts here are against any on-going
> > policy or reasoning.
> >
> > On 1/14/07, Neal Norwitz <nnorwitz at gmail.com> wrote:
> > > SVN rev 52305 resolved Bug #1545497: when given an explicit base,
> > > int() did ignore NULs embedded in the string to convert.
> > >
> > > However, the same fix wasn't applied for long().
> > >
> > > n
> > >
> > > On 1/13/07, Guido van Rossum <guido at python.org> wrote:
> > > > What's wrong with this session? :-)
> > > >
> > > > Python 2.6a0 (trunk:53416, Jan 13 2007, 15:24:17)
> > > > [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
> > > > Type "help", "copyright", "credits" or "license" for more information.
> > > > >>> int('123\0')
> > > > Traceback (most recent call last):
> > > >   File "<stdin>", line 1, in <module>
> > > > ValueError: null byte in argument for int()
> > > > >>> int('123\0', 10)
> > > > Traceback (most recent call last):
> > > >   File "<stdin>", line 1, in <module>
> > > > ValueError: invalid literal for int() with base 10: '123\x00'
> > > > >>> long('123\0')
> > > > Traceback (most recent call last):
> > > >   File "<stdin>", line 1, in <module>
> > > > ValueError: null byte in argument for long()
> > > > >>> long('123\0', 10)
> > > > 123L
> > > > >>>
> > > >
> > > > --
> > > > --Guido van Rossum (home page: http://www.python.org/~guido/)
> > > > _______________________________________________
> > > > Python-Dev mailing list
> > > > Python-Dev at python.org
> > > > http://mail.python.org/mailman/listinfo/python-dev
> > > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
> > > >
> > > _______________________________________________
> > > Python-Dev mailing list
> > > Python-Dev at python.org
> > > http://mail.python.org/mailman/listinfo/python-dev
> > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
> > >
> >
> >
> > --
> > Read my blog! I depend on your acceptance of my opinion! I am interesting!
> > http://ironfroggy-code.blogspot.com/
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
> >
>


-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://ironfroggy-code.blogspot.com/

From foom at fuhm.net  Tue Jan 16 20:03:32 2007
From: foom at fuhm.net (James Y Knight)
Date: Tue, 16 Jan 2007 14:03:32 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <ca471dc20701160747g4674f6fcj2e5811829e9f0eb8@mail.gmail.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<ca471dc20701160747g4674f6fcj2e5811829e9f0eb8@mail.gmail.com>
Message-ID: <E74ED1CD-BA77-4E4D-9866-48EB0B2E83E5@fuhm.net>

On Jan 16, 2007, at 10:47 AM, Guido van Rossum wrote:
> I'm not keen on compromises in 3.0, but without specific proposals I
> don't see why we're arguing. So, please, what specific thing(s) are
> you proposing we do in 3.0? Please make a list of specifics rather
> than attempting at specifying a general rule to match things that
> could go into the list; you've tried the latter and I still don't know
> what you want.

I can't come up with a full list, as I don't know everything that's  
changed or is being planned to change. But here's some stuff I do  
know about:

1) don't remove dict.iteritems, dict.itervalues, dict.iterkeys
2) For the bytes type, keep enough of an overlap with the current use  
of str-as-raw-bytes that common operations work with either one (so  
that I can, f.e. read from a socket or a file and process that data).  
Without seeing the details for how bytes is going to be implemented,  
it's hard to say exactly what this means.
3) Preferably continue to allow old syntax when adding new syntax  
(e.g. exception catching syntax). Alternatively, a completely  
reliable automated conversion script to convert the syntax would also  
be acceptable if not quite as convenient.

Mainly I'd just like to see "allowing the ability to write code which  
is portable between 2.5 and 3.0" as an explicit goal of the python  
3.0 release. I trust that if the developers agree upon that as being  
a goal, the right things would happen, whatever they may be for the  
specific change in question.

James

From thomas at python.org  Tue Jan 16 20:33:42 2007
From: thomas at python.org (Thomas Wouters)
Date: Tue, 16 Jan 2007 20:33:42 +0100
Subject: [Python-Dev] The bytes type
In-Reply-To: <E74ED1CD-BA77-4E4D-9866-48EB0B2E83E5@fuhm.net>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<ca471dc20701160747g4674f6fcj2e5811829e9f0eb8@mail.gmail.com>
	<E74ED1CD-BA77-4E4D-9866-48EB0B2E83E5@fuhm.net>
Message-ID: <9e804ac0701161133o326ba610o92de2faa57176e8b@mail.gmail.com>

On 1/16/07, James Y Knight <foom at fuhm.net> wrote:
>
> On Jan 16, 2007, at 10:47 AM, Guido van Rossum wrote:
> > I'm not keen on compromises in 3.0, but without specific proposals I
> > don't see why we're arguing. So, please, what specific thing(s) are
> > you proposing we do in 3.0? Please make a list of specifics rather
> > than attempting at specifying a general rule to match things that
> > could go into the list; you've tried the latter and I still don't know
> > what you want.
>
> I can't come up with a full list, as I don't know everything that's
> changed or is being planned to change. But here's some stuff I do
> know about:
>
> 1) don't remove dict.iteritems, dict.itervalues, dict.iterkeys
> 2) For the bytes type, keep enough of an overlap with the current use
> of str-as-raw-bytes that common operations work with either one (so
> that I can, f.e. read from a socket or a file and process that data).
> Without seeing the details for how bytes is going to be implemented,
> it's hard to say exactly what this means.
> 3) Preferably continue to allow old syntax when adding new syntax
> (e.g. exception catching syntax). Alternatively, a completely
> reliable automated conversion script to convert the syntax would also
> be acceptable if not quite as convenient.
>
> Mainly I'd just like to see "allowing the ability to write code which
> is portable between 2.5 and 3.0" as an explicit goal of the python
> 3.0 release.


Actually, you missed the part where that is explicitly *not* the goal of
Python 3.0. The main goal of Python 3.0 is explicitly to *break* the things
you don't want broken. If you really not want them broken, Python 3.0 is a
non-starter for you. You can basically stop worrying about it, and just
never switch any of your Python 2.5 projects to Python 3.0. If you want to
talk about an upgrade-path, you will have to accept that it means waiting
for Python 2.6, and also that it means not using some features that will be
removed, like the iter* methods of dicts. We (or at least some of us ;-)
will do our best to provide a viable upgrade path from 2.6 even for
complicated cases like the changing dict methods, but an upgrade path
directly from 2.5-or-earlier to 3.0 is just plain impossible; you would have
to change your code to avoid the things that will change in Python 3.0,
which may lead to inefficient or complicated code.

Python 3.0 is there to break backward compatibility, and do it cleanly. Any
work done to ease the upgrade path should be done in 2.x (and separate
tools), not 3.0. 3.0 is not a transitional release.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070116/cf962af4/attachment.htm 

From guido at python.org  Tue Jan 16 20:35:00 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Jan 2007 11:35:00 -0800
Subject: [Python-Dev] The bytes type
In-Reply-To: <E74ED1CD-BA77-4E4D-9866-48EB0B2E83E5@fuhm.net>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<ca471dc20701160747g4674f6fcj2e5811829e9f0eb8@mail.gmail.com>
	<E74ED1CD-BA77-4E4D-9866-48EB0B2E83E5@fuhm.net>
Message-ID: <ca471dc20701161135v5f3fdb3x4e490c5f057d8485@mail.gmail.com>

On 1/16/07, James Y Knight <foom at fuhm.net> wrote:
> On Jan 16, 2007, at 10:47 AM, Guido van Rossum wrote:
> > I'm not keen on compromises in 3.0, but without specific proposals I
> > don't see why we're arguing. So, please, what specific thing(s) are
> > you proposing we do in 3.0? Please make a list of specifics rather
> > than attempting at specifying a general rule to match things that
> > could go into the list; you've tried the latter and I still don't know
> > what you want.
>
> I can't come up with a full list, as I don't know everything that's
> changed or is being planned to change. But here's some stuff I do
> know about:
>
> 1) don't remove dict.iteritems, dict.itervalues, dict.iterkeys
> 2) For the bytes type, keep enough of an overlap with the current use
> of str-as-raw-bytes that common operations work with either one (so
> that I can, f.e. read from a socket or a file and process that data).
> Without seeing the details for how bytes is going to be implemented,
> it's hard to say exactly what this means.
> 3) Preferably continue to allow old syntax when adding new syntax
> (e.g. exception catching syntax). Alternatively, a completely
> reliable automated conversion script to convert the syntax would also
> be acceptable if not quite as convenient.
>
> Mainly I'd just like to see "allowing the ability to write code which
> is portable between 2.5 and 3.0" as an explicit goal of the python
> 3.0 release. I trust that if the developers agree upon that as being
> a goal, the right things would happen, whatever they may be for the
> specific change in question.

Well, there's the crux. That is an explicit NON-goal. It always has
been. It would add tons of constraints to Python 3000, and the net
effect would be that everybody would continue to code in the 2.5
dialect, making the intended cleanup of the language ineffective.

We are planning various other ways of providing conversion help. The
most promising at this point is to make the following an explicit goal
of Python 2.6 and Python 3000: make it possible to write code that
runs under 2.6, and that, after automatic conversion using the 2to3
refactoring tool (see sandbox/2to3), runs under 3.0, *without*
requiring manual patching up of the output of the refactoring tool.

The latter clause makes it easy (if not trivial) for developers to
maintain parallel versions. All you need is a sufficiently good test
suite, so that you can automatically verify that the conversion result
actually works as intended under 3.0. Since you'd have to do that
testing anyway, there is not much of an added effort except for
maintaining separate tarballs.

There are still certain things you can't do, because the conversion
tool doesn't handle them well (e.g. don't use variables named 'long'
and don't create classes with methods named 'keys'); but I expect that
the subset of 2.6 that you can use will be much larger using this
approach than what you would get using the identical-source approach
you are supporting.

Oh, and the same approach will also work (with only a few more
constraints) for 2.5 and even earlier versions.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From steve at holdenweb.com  Tue Jan 16 21:52:32 2007
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 16 Jan 2007 20:52:32 +0000
Subject: [Python-Dev] Proposed 3.0 compatiblity module
In-Reply-To: <ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
References: <016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
	<ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
Message-ID: <eojdue$dmr$1@sea.gmane.org>

Guido van Rossum wrote:
> On 1/12/07, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
[...]
>> Outside of a compatibility module, we can decide to leave Py2.6 unmolested by
>> Py3.0, so I can stop being cranky.
> 
> If you're feeling cranky this early in the new year, maybe you should
> look for reasons outside this list. :-)
> 
What, we're supposed to reset our crankometers on January 1? Nobody told 
me that.

> I'm all for striving for minimal impact. I don't think we can attain
> zero impact.
> 
> BTW, a possible alternative would be to strive for code that runs
> under 2.x, and *after automatic transformation using the 2to3 tool*
> runs under 3.x, *without mannual patch-up*. That's probably a much
> lower bar, and it would still mean that developers wishing to maintain
> code that works with both 2.x/3.x only need to maintain a single
> version -- the other version could be automatically generated.
> 
> Since you need to do a lot of testing anyways to make sure you haven't
> accidentally broken compatibility, the effort would be roughly
> equivalent to the "identical source" approach that people are
> advocating so far, and it would probably lift some restrictions.
> 
> For example, working around print becoming a function would mean no
> use of print at all when using "identical source"; but that's a
> transformation that the 2to3 tool can do trivially and perfectly, so
> the "automatic conversion" approach would enable freely using print
> statements.
> 
> A mixture of the two approaches would perhaps work best.
> 
I foresee that many people would be happy restricting their 2.X source 
slightly to ensure perfect translation into (working, no necessarily 
optimal) 3.0. Under those circumstances the 2to3 tool wouldn't 
necessarily have to translate all valid 2.X to 3.0.

cranki-ly y'rs  - steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From jimjjewett at gmail.com  Tue Jan 16 23:04:53 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Tue, 16 Jan 2007 17:04:53 -0500
Subject: [Python-Dev] [Python-3000] dict.items as attributes [Was: The
	bytes type]
In-Reply-To: <00d001c739af$36c15b50$f100a8c0@ryoko>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
	<00c601c73998$0a2f1b20$f100a8c0@ryoko>
	<ca471dc20701161019g5658d346gd295e0b65724dbf0@mail.gmail.com>
	<00d001c739af$36c15b50$f100a8c0@ryoko>
Message-ID: <fb6fbf560701161404v1986fffcv7eca0e0772850bd1@mail.gmail.com>

Other than dict.items (and .keys and .values) returning a non-list,
are there any other cases where the Py3K idiom can't already be used
in (or at least backported to) Py 2.x?

Is there a chance that this special case could be handled by an import
hook in py 2.6?

For example 2.6 could have an additional LOAD_ATTR_PY3 opcode that
just handles the special cases before delegating to the normal
LOAD_ATTR.

It would be generated instead of LOAD_ATTR whenever

    >>> from __future__ import py3

was in effect.

Supporting 2.5 (and 2.2) would be a bit harder, but I think the new
exception syntax would already require a py3->py2 translator, and
these three names still seem like a fairly simple transform.  So long
as the "real" source is kept in py3 format, autogenerating python 2
code shouldn't be much worse than autogenerating pyc files.

-jJ

From pje at telecommunity.com  Tue Jan 16 23:07:26 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 16 Jan 2007 17:07:26 -0500
Subject: [Python-Dev] Proposed 3.0 compatiblity module
In-Reply-To: <eojdue$dmr$1@sea.gmane.org>
References: <ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
	<016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
	<ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
Message-ID: <5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>

At 08:52 PM 1/16/2007 +0000, Steve Holden wrote:
>I foresee that many people would be happy restricting their 2.X source
>slightly to ensure perfect translation into (working, no necessarily
>optimal) 3.0. Under those circumstances the 2to3 tool wouldn't
>necessarily have to translate all valid 2.X to 3.0.

Actually, it would be several times more preferable to either have that 
restricted subset of code run on 3.0 without translation, or for translated 
code to still be usable in 2.X.

I have often been in the habit of running test suites back-to-back on 
multiple versions of Python while doing test-driven development, so having 
a repeated translation step would interfere with that.  Ideally, a 
translation should be necessary one time only -- in which case requiring 
manual cleanup steps isn't as big of a problem.

The idea here being that, once 2.6 is widely-enough deployed that it can be 
assumed as a base for one's users, you can simply run the translator once, 
do any cleanup, and then have 3.0-clean code that also still runs for your 
installed base.

That way, there's no chasm to leap; just a code cleanup.


From foom at fuhm.net  Tue Jan 16 23:07:08 2007
From: foom at fuhm.net (James Y Knight)
Date: Tue, 16 Jan 2007 17:07:08 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <ca471dc20701161135v5f3fdb3x4e490c5f057d8485@mail.gmail.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<ca471dc20701160747g4674f6fcj2e5811829e9f0eb8@mail.gmail.com>
	<E74ED1CD-BA77-4E4D-9866-48EB0B2E83E5@fuhm.net>
	<ca471dc20701161135v5f3fdb3x4e490c5f057d8485@mail.gmail.com>
Message-ID: <1C852710-575A-4855-8460-F70E74D57192@fuhm.net>


On Jan 16, 2007, at 2:35 PM, Guido van Rossum wrote:
>> Mainly I'd just like to see "allowing the ability to write code which
>> is portable between 2.5 and 3.0" as an explicit goal of the python
>> 3.0 release. I trust that if the developers agree upon that as being
>> a goal, the right things would happen, whatever they may be for the
>> specific change in question.
>
> Well, there's the crux. That is an explicit NON-goal. It always has
> been. It would add tons of constraints to Python 3000, and the net
> effect would be that everybody would continue to code in the 2.5
> dialect, making the intended cleanup of the language ineffective.

Okay. I think this is a mistake. Everyone will continue to code in a  
2.5-compatible dialect for the immediate future, because their code  
needs to continue to function in 2.5. As I keep pointing out, the  
language cleanup doesn't have to be all at once: remove all the  
already-replaced features now, and deprecate a bunch more things for  
removal a few releases later. That'd allow transition to 3.0 to occur  
during the time when people have to write in a 2.5-compatible  
dialect, anyhow. Otherwise, they'll be writing to 2.5-ONLY that won't  
run on 3.0 at all.

> We are planning various other ways of providing conversion help. The
> most promising at this point is to make the following an explicit goal
> of Python 2.6 and Python 3000: make it possible to write code that
> runs under 2.6, and that, after automatic conversion using the 2to3
> refactoring tool (see sandbox/2to3), runs under 3.0, *without*
> requiring manual patching up of the output of the refactoring tool.

Given that my ideal is not going to happen, I guess I'll have to be  
content with this. It WILL allow code to be converted incrementally  
to be compatible with Python 3.X (which is very important!), but,  
unfortunately, not for many years after Py3.0's release. (since the  
code can't be made portable until Python 2.6 has been out long enough  
for that to become a reasonable minimum requirement).

But, a delayed migration strategy is certainly much much better than  
*no* migration strategy, so it's not all doom and gloom.

But, anyhow, that's enough from me. My opinion has been made known,  
and disagreed with by (I think) all the prominent python developers.  
So now, I'll let it drop and go get some work done.

Thanks for listening,
James

From steve at holdenweb.com  Tue Jan 16 23:23:04 2007
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 16 Jan 2007 22:23:04 +0000
Subject: [Python-Dev] Proposed 3.0 compatiblity module
In-Reply-To: <5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
References: <ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
	<016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
	<ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
	<5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
Message-ID: <45AD5048.3050103@holdenweb.com>

Phillip J. Eby wrote:
> At 08:52 PM 1/16/2007 +0000, Steve Holden wrote:
>> I foresee that many people would be happy restricting their 2.X source
>> slightly to ensure perfect translation into (working, no necessarily
>> optimal) 3.0. Under those circumstances the 2to3 tool wouldn't
>> necessarily have to translate all valid 2.X to 3.0.
> 
> Actually, it would be several times more preferable to either have that 
> restricted subset of code run on 3.0 without translation, or for 
> translated code to still be usable in 2.X.
> 
> I have often been in the habit of running test suites back-to-back on 
> multiple versions of Python while doing test-driven development, so 
> having a repeated translation step would interfere with that.  Ideally, 
> a translation should be necessary one time only -- in which case 
> requiring manual cleanup steps isn't as big of a problem.
> 
> The idea here being that, once 2.6 is widely-enough deployed that it can 
> be assumed as a base for one's users, you can simply run the translator 
> once, do any cleanup, and then have 3.0-clean code that also still runs 
> for your installed base.
> 
> That way, there's no chasm to leap; just a code cleanup.
> 
That assumes that once you convert to 3.0 you are no longer interested 
in back-porting further functionality changes to your 2.X code, surely, 
or am I missing something?

It seems to me preferable to allow repeated translation of 2.X code to 
working no-fixes-required 3.0. Then the cleanup step is required only 
when withdrawing support (or development) for 2.X and making the final 
translation. At that point it would be a one-time task.

Or are you suggesting that users make the change to 3.0 and then 
backport changes to 2.X where necessary using the older semantics? In 
either case you end up with two different code bases.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com

From barry at python.org  Tue Jan 16 23:47:18 2007
From: barry at python.org (Barry Warsaw)
Date: Tue, 16 Jan 2007 17:47:18 -0500
Subject: [Python-Dev] [Python-3000] dict.items as attributes [Was: The
	bytes type]
In-Reply-To: <fb6fbf560701161404v1986fffcv7eca0e0772850bd1@mail.gmail.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
	<00c601c73998$0a2f1b20$f100a8c0@ryoko>
	<ca471dc20701161019g5658d346gd295e0b65724dbf0@mail.gmail.com>
	<00d001c739af$36c15b50$f100a8c0@ryoko>
	<fb6fbf560701161404v1986fffcv7eca0e0772850bd1@mail.gmail.com>
Message-ID: <F3974591-324D-41D6-8C33-DECBDCB7CAFF@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 16, 2007, at 5:04 PM, Jim Jewett wrote:

> Other than dict.items (and .keys and .values) returning a non-list,
> are there any other cases where the Py3K idiom can't already be used
> in (or at least backported to) Py 2.x?

I know Guido is against attribute syntax for dict.items and friends,  
and I agree with him for reasons I can't quite put my finger on.   
But, would it be possible to support both the py3k way and the Python  
2 way if you accepted attribute syntax for returning the view  
thingie?  Couldn't that view thingie have an __call__ that returned  
the backward compatible list object?

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRa1V9nEjvBPtnXfVAQIODQQAlCI4iykl93zKJeIW/3QuuOm4jCRrhffV
BOkuu+mM4N/1Rqc2TCBHUIi995TkYo/6Wuzb7LrXweG5kLm2cdWTNm+aSgZq35HL
XxPVP9qwCElKbZjl84udx/dbLdl9AfVasT2iVUiEVZN3Kq5j+UbyK+oaQdVjBc+5
j51iDW+vhcc=
=Qq7c
-----END PGP SIGNATURE-----

From martin at v.loewis.de  Wed Jan 17 00:08:56 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 17 Jan 2007 00:08:56 +0100
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <20070116112948.GA14364@andrew-kuchlings-computer.local>
References: <20070115182102.17094.456325909.divmod.xquotient.2056@joule.divmod.com>	<45AC02D6.7070900@v.loewis.de>
	<20070116112948.GA14364@andrew-kuchlings-computer.local>
Message-ID: <45AD5B08.7080805@v.loewis.de>

A.M. Kuchling schrieb:
> What about turning all references to obj.items into the equivalent
> bytecode for this:
> 
> if isinstance(obj, dict): # XXX should this be 'type(obj) is dict'?
>    if <2.x behaviour>:   _temp = obj.items
>    elif <3.x behaviour>: _temp = obj.iteritems
> else:
>    _temp = obj.items
> 
> Ugly; very ugly.

This would "work", I think, although I'd rather add a tp_getattr3 slot
to types, as somebody else proposed: getattr3 would default to getattr
if not defined, and would look up iteritems when asked for iter on dict
objects.

It would still "suffer" from the cross-module issue:

# a.py
from __future__ import items_is_iterator
def f(d):
  return d.items
# b.py
import a
d = d{}
print a.f(d)()

For compatibility with 2.x, a.f should really return a bound
method that returns lists; for compatibility with 3.x, it will return
a method that produces an iterator.

Of course, one might say "don't do that, then".

Regards,
Martin

From guido at python.org  Wed Jan 17 00:18:31 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Jan 2007 15:18:31 -0800
Subject: [Python-Dev] Proposed 3.0 compatiblity module
In-Reply-To: <5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
References: <016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
	<ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
	<eojdue$dmr$1@sea.gmane.org>
	<5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
Message-ID: <ca471dc20701161518xb7ac622ueac7adfe56d275cd@mail.gmail.com>

On 1/16/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 08:52 PM 1/16/2007 +0000, Steve Holden wrote:
> >I foresee that many people would be happy restricting their 2.X source
> >slightly to ensure perfect translation into (working, no necessarily
> >optimal) 3.0. Under those circumstances the 2to3 tool wouldn't
> >necessarily have to translate all valid 2.X to 3.0.
>
> Actually, it would be several times more preferable to either have that
> restricted subset of code run on 3.0 without translation, or for translated
> code to still be usable in 2.X.
>
> I have often been in the habit of running test suites back-to-back on
> multiple versions of Python while doing test-driven development, so having
> a repeated translation step would interfere with that.  Ideally, a
> translation should be necessary one time only -- in which case requiring
> manual cleanup steps isn't as big of a problem.
>
> The idea here being that, once 2.6 is widely-enough deployed that it can be
> assumed as a base for one's users, you can simply run the translator once,
> do any cleanup, and then have 3.0-clean code that also still runs for your
> installed base.
>
> That way, there's no chasm to leap; just a code cleanup.

I understand; I would rather have that too, everything else being the
same. But everything else wouldn't be the same -- it would place many
more restrictions on 3.0, and the common subset would still be much
smaller. For me personally, the weight of the added restrictions to
3.0 is the killer.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Wed Jan 17 00:45:22 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 16 Jan 2007 18:45:22 -0500
Subject: [Python-Dev] Proposed 3.0 compatiblity module
In-Reply-To: <ca471dc20701161518xb7ac622ueac7adfe56d275cd@mail.gmail.com
 >
References: <5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
	<016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
	<ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
	<eojdue$dmr$1@sea.gmane.org>
	<5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20070116184433.028e86a8@sparrow.telecommunity.com>

At 03:18 PM 1/16/2007 -0800, Guido van Rossum wrote:
>On 1/16/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> > The idea here being that, once 2.6 is widely-enough deployed that it can be
> > assumed as a base for one's users, you can simply run the translator once,
> > do any cleanup, and then have 3.0-clean code that also still runs for your
> > installed base.
> >
> > That way, there's no chasm to leap; just a code cleanup.
>
>I understand; I would rather have that too, everything else being the
>same. But everything else wouldn't be the same -- it would place many
>more restrictions on 3.0, and the common subset would still be much
>smaller. For me personally, the weight of the added restrictions to
>3.0 is the killer.

I don't understand; how would adding features to 2.6 restrict what you 
could add to 3.0?


From guido at python.org  Wed Jan 17 00:47:07 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Jan 2007 15:47:07 -0800
Subject: [Python-Dev] Proposed 3.0 compatiblity module
In-Reply-To: <5.1.1.6.0.20070116184433.028e86a8@sparrow.telecommunity.com>
References: <016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
	<ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
	<eojdue$dmr$1@sea.gmane.org>
	<5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
	<5.1.1.6.0.20070116184433.028e86a8@sparrow.telecommunity.com>
Message-ID: <ca471dc20701161547s6a976b3cxb469e016953a53dc@mail.gmail.com>

On 1/16/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 03:18 PM 1/16/2007 -0800, Guido van Rossum wrote:
> >On 1/16/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> > > The idea here being that, once 2.6 is widely-enough deployed that it can be
> > > assumed as a base for one's users, you can simply run the translator once,
> > > do any cleanup, and then have 3.0-clean code that also still runs for your
> > > installed base.
> > >
> > > That way, there's no chasm to leap; just a code cleanup.
> >
> >I understand; I would rather have that too, everything else being the
> >same. But everything else wouldn't be the same -- it would place many
> >more restrictions on 3.0, and the common subset would still be much
> >smaller. For me personally, the weight of the added restrictions to
> >3.0 is the killer.
>
> I don't understand; how would adding features to 2.6 restrict what you
> could add to 3.0?

Oh, as long as we're talking adding features to 2.6 I'm fine. I
thought you were proposing changes to the plans for 3.0, as in your
proposal" regarding the dict view API.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Wed Jan 17 00:50:07 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 16 Jan 2007 18:50:07 -0500
Subject: [Python-Dev] Proposed 3.0 compatiblity module
In-Reply-To: <45AD5048.3050103@holdenweb.com>
References: <5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
	<ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
	<016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
	<ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
	<5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20070116184534.028e87f0@sparrow.telecommunity.com>

At 10:23 PM 1/16/2007 +0000, Steve Holden wrote:
>Phillip J. Eby wrote:
>>At 08:52 PM 1/16/2007 +0000, Steve Holden wrote:
>>>I foresee that many people would be happy restricting their 2.X source
>>>slightly to ensure perfect translation into (working, no necessarily
>>>optimal) 3.0. Under those circumstances the 2to3 tool wouldn't
>>>necessarily have to translate all valid 2.X to 3.0.
>>Actually, it would be several times more preferable to either have that 
>>restricted subset of code run on 3.0 without translation, or for 
>>translated code to still be usable in 2.X.
>>I have often been in the habit of running test suites back-to-back on 
>>multiple versions of Python while doing test-driven development, so 
>>having a repeated translation step would interfere with that.  Ideally, a 
>>translation should be necessary one time only -- in which case requiring 
>>manual cleanup steps isn't as big of a problem.
>>The idea here being that, once 2.6 is widely-enough deployed that it can 
>>be assumed as a base for one's users, you can simply run the translator 
>>once, do any cleanup, and then have 3.0-clean code that also still runs 
>>for your installed base.
>>That way, there's no chasm to leap; just a code cleanup.
>That assumes that once you convert to 3.0 you are no longer interested in 
>back-porting further functionality changes to your 2.X code, surely, or am 
>I missing something?

I'm saying that the 3.0 code would have to be usable in 2.x.  That is, any 
new features added to 3.0 need to be backported to a 2.x release.

For example, a "from __future__ import print" would need to be added to 2.6 
to allow the use of print-as-function syntax.  Otherwise, there is no way 
to write code that works in 2.x *and* 3.x, because print is a reserved word 
in 2.x, and can't be a statement in 3.x.


>It seems to me preferable to allow repeated translation of 2.X code to 
>working no-fixes-required 3.0. Then the cleanup step is required only when 
>withdrawing support (or development) for 2.X and making the final 
>translation. At that point it would be a one-time task.
>
>Or are you suggesting that users make the change to 3.0 and then backport 
>changes to 2.X where necessary using the older semantics? In either case 
>you end up with two different code bases.

No, I'm saying one code base that runs on both.  The main difference 
between me and James Knight on this is that I'm saying 2.6 or 2.7, he 
wanted 2.5.  I agree with Guido that a 2.5 backward target is unreasonably 
restrictive for 3.0.


From scott+python-dev at scottdial.com  Wed Jan 17 00:42:35 2007
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Tue, 16 Jan 2007 18:42:35 -0500
Subject: [Python-Dev] [Python-3000] Warning for 2.6 and greater
In-Reply-To: <45AD5B08.7080805@v.loewis.de>
References: <20070115182102.17094.456325909.divmod.xquotient.2056@joule.divmod.com>	<45AC02D6.7070900@v.loewis.de>	<20070116112948.GA14364@andrew-kuchlings-computer.local>
	<45AD5B08.7080805@v.loewis.de>
Message-ID: <45AD62EB.9000806@scottdial.com>

Martin v. L?wis wrote:
> It would still "suffer" from the cross-module issue:
> 
> # a.py
> from __future__ import items_is_iterator
> def f(d):
>   return d.items
> # b.py
> import a
> d = {}
> print a.f(d)
> 
> For compatibility with 2.x, a.f should really return a bound
> method that returns lists; for compatibility with 3.x, it will return
> a method that produces an iterator.
> 
> Of course, one might say "don't do that, then".
> 

This seems like a perfectly reasonable thing to say. If I am writing 
something for 2.X and want to use a 3.X feature switch, then I am 
responsible for maintaining the external API, be it the 2.X version of 
d.items if that is what is expected by the user.

Your complaint only works if a.f was pre-defined to return a list, which 
the developer of "a" would know about when he decided to flip on 
items_is_iterator. As a part of that refactoring, he would patch up the 
return to be list(d.items) to maintain the API he has already defined. 
It is not clear by your example that "b" wasn't in fact expecting to get 
an iterator, which is now a great deal simpler thanks to the future 
statement.

My point is simply that this argument doesn't work unless you assume the 
developer of "a" has failed to do his job (which is to maintain the 
external API by also testing the external API).

BTW, I patched your code because both versions you have presented didn't 
actually work.

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu

From pje at telecommunity.com  Wed Jan 17 01:11:14 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 16 Jan 2007 19:11:14 -0500
Subject: [Python-Dev] Proposed 3.0 compatiblity module
In-Reply-To: <ca471dc20701161547s6a976b3cxb469e016953a53dc@mail.gmail.co
 m>
References: <5.1.1.6.0.20070116184433.028e86a8@sparrow.telecommunity.com>
	<016f01c73680$9a80e200$ea146b0a@RaymondLaptop1>
	<ca471dc20701121239u78326ac0k5ceba8b7c00bae23@mail.gmail.com>
	<eojdue$dmr$1@sea.gmane.org>
	<5.1.1.6.0.20070116165933.028e8550@sparrow.telecommunity.com>
	<5.1.1.6.0.20070116184433.028e86a8@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20070116190847.028fd090@sparrow.telecommunity.com>

At 03:47 PM 1/16/2007 -0800, Guido van Rossum wrote:
>Oh, as long as we're talking adding features to 2.6 I'm fine. I
>thought you were proposing changes to the plans for 3.0, as in your
>proposal" regarding the dict view API.

Nah, that was a side tangent, combined with a partial lack of understanding 
of how the view thing was supposed to work.  Having to wrap 'em with list() 
calls is annoying but doable.  I was just throwing out an idea I didn't 
realize had already been shot down.


From nnorwitz at gmail.com  Wed Jan 17 07:34:43 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 16 Jan 2007 22:34:43 -0800
Subject: [Python-Dev] adding _Py prefix to names in 2.5.1?
Message-ID: <ee2a432c0701162234p7a294d5aj3fe54ff9b55f479@mail.gmail.com>

http://python.org/sf/1637022 points out a problem caused by the lack
of a _Py prefix on Ellipsis.  Most (all?) of the new AST names are not
prefixed.  These are all meant to be internal names.  Are there any
issues with changing this?  If we do so, it means that any module
built with 2.5 that is using these names will fail to work in 2.5.1.
No code outside the core *should* be using these names.

Assuming there is no dissent, does some budding Python developer want
to take on a patch?  This should be pretty straightforward.  Feel free
to mail me off list if you prefer.

Cheers,
n

From tjreedy at udel.edu  Wed Jan 17 09:03:29 2007
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 17 Jan 2007 03:03:29 -0500
Subject: [Python-Dev] The bytes type
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com><6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com><9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com><FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net><ca471dc20701160747g4674f6fcj2e5811829e9f0eb8@mail.gmail.com><E74ED1CD-BA77-4E4D-9866-48EB0B2E83E5@fuhm.net><ca471dc20701161135v5f3fdb3x4e490c5f057d8485@mail.gmail.com>
	<1C852710-575A-4855-8460-F70E74D57192@fuhm.net>
Message-ID: <eokl8h$alc$1@sea.gmane.org>


"James Y Knight" <foom at fuhm.net> wrote in message 
news:1C852710-575A-4855-8460-F70E74D57192 at fuhm.net...

| Everyone will continue to code in a
| 2.5-compatible dialect for the immediate future, because their code
| needs to continue to function in 2.5.

??  But once 2.6 and 3.0 come out, there will be some to many who don't 
have that need.  Anyone who needs their code to function in x.y will have 
to restrict their code to x.y and not use future additions.  And people who 
don't, may use new features.  Nothing new here.

I expect that if Guido survives and continues with the Py3 series, then in 
10 years, at least half of Python programmers will be essentially Py3 only, 
either by switching and not looking back or by having learned on Py3.

| As I keep pointing out, the
| language cleanup doesn't have to be all at once:

Of course, and gradual, continual cleanup was the original policy: 
deprecate, warn, and then remove.  But several years ago, enough people 
complained that Guido said "Ok, we will put non-trivial removals off for 
several years and then do it all at once in a new 3.0 version."  Under the 
old policy, apply(), old int division (1/2 ==0 instead of .5), and probably 
some others would have been long gone.  Now several years is about here.

Terry Jan Reedy




From thomas at python.org  Wed Jan 17 13:19:06 2007
From: thomas at python.org (Thomas Wouters)
Date: Wed, 17 Jan 2007 04:19:06 -0800
Subject: [Python-Dev] [Python-3000] dict.items as attributes [Was: The
	bytes type]
In-Reply-To: <fb6fbf560701161404v1986fffcv7eca0e0772850bd1@mail.gmail.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
	<00c601c73998$0a2f1b20$f100a8c0@ryoko>
	<ca471dc20701161019g5658d346gd295e0b65724dbf0@mail.gmail.com>
	<00d001c739af$36c15b50$f100a8c0@ryoko>
	<fb6fbf560701161404v1986fffcv7eca0e0772850bd1@mail.gmail.com>
Message-ID: <9e804ac0701170419v1d802b5aqd49456231a75f1cc@mail.gmail.com>

On 1/16/07, Jim Jewett <jimjjewett at gmail.com> wrote:
>
> Other than dict.items (and .keys and .values) returning a non-list,
> are there any other cases where the Py3K idiom can't already be used
> in (or at least backported to) Py 2.x?


Well, there is that bit where strings are all unicode, including all string
literals and attribute names and all that. And the bit where open() will be
different, although how different hasn't been decided yet. (Or rather, it
hasn't been written down in code yet, and to me that means it hasn't been
decided yet :)

What we need to do now is do the work on 3.0, and *then* (and a bit
'during', too) think about backward compatibility. As far as I can see, all
the existing changes in the p3yk branch are upgrade-pathable: the int/long
unification has long been done on the Python-visible side, 'except as' will
make it into 2.6, print/exec as functions can be imported from the future,
sys.intern/compile/etc can be made available in 2.6, classic slices are
already on the way out, et cetera ad nauseam. The aim for 2.6 should be to
have all the new features that 3.0 has, as well as full backward
compatibility (albeit potentially with a few extra warnings, as any release
might, but only for code that can be written sanely for 2.5-and-earlier as
well.) The biggest hurdle will be the unicode/string unification and the I/O
system (much bigger than the dict keys/values/items change), but since those
haven't been implemented yet, it's rather pointless to talk about how to
handle them in 2.6.

Is there a chance that this special case could be handled by an import
> hook in py 2.6?


I expect you don't mean an actual import hook, but just a future import?
Then sure, there is a chance. We'll have to see.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070117/94cc8fbb/attachment.htm 

From thomas at python.org  Wed Jan 17 13:21:57 2007
From: thomas at python.org (Thomas Wouters)
Date: Wed, 17 Jan 2007 04:21:57 -0800
Subject: [Python-Dev] adding _Py prefix to names in 2.5.1?
In-Reply-To: <ee2a432c0701162234p7a294d5aj3fe54ff9b55f479@mail.gmail.com>
References: <ee2a432c0701162234p7a294d5aj3fe54ff9b55f479@mail.gmail.com>
Message-ID: <9e804ac0701170421t16b9ac5eoc2efd666b1c1e5d4@mail.gmail.com>

On 1/16/07, Neal Norwitz <nnorwitz at gmail.com> wrote:
>
> http://python.org/sf/1637022 points out a problem caused by the lack
> of a _Py prefix on Ellipsis.  Most (all?) of the new AST names are not
> prefixed.  These are all meant to be internal names.  Are there any
> issues with changing this?  If we do so, it means that any module
> built with 2.5 that is using these names will fail to work in 2.5.1.
> No code outside the core *should* be using these names.


It makes me terribly uncomfortable, as removing symbols is absolutely not
something that should happen in a bugfix release, but I think that, in this
case, it is the right thing to do. Well, the least-wrong thing to do. If
only C had more visibility control, sigh ;-P

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070117/7bd49343/attachment.html 

From jimjjewett at gmail.com  Wed Jan 17 18:05:36 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 17 Jan 2007 12:05:36 -0500
Subject: [Python-Dev] [Python-3000] dict.items as attributes [Was: The
	bytes type]
In-Reply-To: <9e804ac0701170419v1d802b5aqd49456231a75f1cc@mail.gmail.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>
	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>
	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>
	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
	<00c601c73998$0a2f1b20$f100a8c0@ryoko>
	<ca471dc20701161019g5658d346gd295e0b65724dbf0@mail.gmail.com>
	<00d001c739af$36c15b50$f100a8c0@ryoko>
	<fb6fbf560701161404v1986fffcv7eca0e0772850bd1@mail.gmail.com>
	<9e804ac0701170419v1d802b5aqd49456231a75f1cc@mail.gmail.com>
Message-ID: <fb6fbf560701170905l6cbfe29qf92fd87352ed3290@mail.gmail.com>

On 1/17/07, Thomas Wouters <thomas at python.org> wrote:
> On 1/16/07, Jim Jewett <jimjjewett at gmail.com> wrote:
> > Other than dict.items (and .keys and .values) returning a non-list,
> > are there any other cases where the Py3K idiom can't already be used
> > in (or at least backported to) Py 2.x?

> The aim for 2.6 should be to have all the new features that 3.0 has,
> as well as full backward compatibility ...

And I'm asking if there are any real barriers to this.  Some people
have suggested that they'll have to maintain separate code bases.  So
far, the closest I've seen[1] to something that can't use
common-subset is wanting an iterator over a mapping.

> Well, there is that bit where strings are all unicode, including all string
> literals and attribute names and all that.

(But note that attribute names will still be limited to ASCII.)

I don't see any problems for python code there, unless you're going
out of your way to care which stringlike type you get.

I do see that C extensions could be trickier, though I haven't yet see
anything that couldn't be handled with some relatively simple
conditionally-defined macros.

> And the bit where open() will be different,

This I couldn't find, unless it was the security abilities.  These
wouldn't bother conservative code, and could probably be backported to
2.6, or at least 2.7.

> The biggest hurdle will be the unicode/string unification and the I/O
> system (much bigger than the dict keys/values/items change), but since those
> haven't been implemented yet, it's rather pointless to talk about how to
> handle them in 2.6.

Fair enough.


[1]  I did miss some introspection names, but that can be handled in
python, and anyone using, rather than reinventing, inspect should be
fine.

-jJ

From theller at ctypes.org  Wed Jan 17 21:08:12 2007
From: theller at ctypes.org (Thomas Heller)
Date: Wed, 17 Jan 2007 21:08:12 +0100
Subject: [Python-Dev] Pythn 2.5 buildbots do not build anymore?
Message-ID: <eolvnc$e63$1@sea.gmane.org>

Why don't the Python 2.5 buildbots build anymore?

http://www.python.org/dev/buildbot/2.5/

Thanks,
Thomas


From amk at amk.ca  Wed Jan 17 22:20:48 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Wed, 17 Jan 2007 16:20:48 -0500
Subject: [Python-Dev] Testers wanted: mailbox.py bugfix
Message-ID: <20070117212048.GA17844@localhost.localdomain>

http://www.python.org/sf/1599254 is a bug in the mailbox.py module
that affects single-file mailbox formats (mbox, Babyl, MMDF)
and can cause data loss.

A proposed fix, mailbox-unified-patch, is attached to the bug.  If you
use mailbox.py for modifying mailboxes in the above formats, can you
please test the patch and report if there are any problems?  The bug
is pretty complicated, and such bugs in the mailbox.py (and the
necessary fixes) make me nervous.  Some more testing would be very
reassuring.

--amk

From python at discworld.dyndns.org  Wed Jan 17 22:36:58 2007
From: python at discworld.dyndns.org (Charles Cazabon)
Date: Wed, 17 Jan 2007 15:36:58 -0600
Subject: [Python-Dev] Testers wanted: mailbox.py bugfix
In-Reply-To: <20070117212048.GA17844@localhost.localdomain>
References: <20070117212048.GA17844@localhost.localdomain>
Message-ID: <20070117213658.GA31510@discworld.dyndns.org>

A.M. Kuchling <amk at amk.ca> wrote:
> http://www.python.org/sf/1599254 is a bug in the mailbox.py module
> that affects single-file mailbox formats (mbox, Babyl, MMDF)
> and can cause data loss.

I can't speak for MMDF or Babyl, but it's a well-known requirement that all
programs on a system must use the same type of locking when mbox files are
used; the administrator picks his favourite (fcntl, dotlocking, whatever) and
then has to ensure all mbox-aware programs are compiled to use that locking
style.

> A proposed fix, mailbox-unified-patch, is attached to the bug.

I'm not sure that this actually fixes the problem, but more to the point --
I'm not sure it should try.  Any mail admin that deploys multiple mbox-aware
programs with different locking methods is shooting themselves in the foot,
regardless of whether Python tries to turn it into "just a flesh wound"...

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                           <python at discworld.dyndns.org>
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------

From anthony at interlink.com.au  Thu Jan 18 00:22:14 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu, 18 Jan 2007 10:22:14 +1100
Subject: [Python-Dev] The bytes type
In-Reply-To: <85A3FBBE-C077-4F7E-B95E-82D99CE87AB1@fuhm.net>
References: <FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
	<85A3FBBE-C077-4F7E-B95E-82D99CE87AB1@fuhm.net>
Message-ID: <200701181022.16962.anthony@interlink.com.au>

On Wednesday 17 January 2007 05:52, James Y Knight wrote:
> Yes, this is it. As a refinement: if the New Way can easily be
> backported to 2.5, 

Um - 2.5 is _done_. Released. In maintenance mode. New features will 
not be getting backported to a 2.5.x release. 

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From foom at fuhm.net  Thu Jan 18 01:49:47 2007
From: foom at fuhm.net (James Y Knight)
Date: Wed, 17 Jan 2007 19:49:47 -0500
Subject: [Python-Dev] The bytes type
In-Reply-To: <200701181022.16962.anthony@interlink.com.au>
References: <FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>
	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>
	<85A3FBBE-C077-4F7E-B95E-82D99CE87AB1@fuhm.net>
	<200701181022.16962.anthony@interlink.com.au>
Message-ID: <174163D7-DED2-47A7-9D3B-B3BA2B1E5183@fuhm.net>

On Jan 17, 2007, at 6:22 PM, Anthony Baxter wrote:

> On Wednesday 17 January 2007 05:52, James Y Knight wrote:
>> Yes, this is it. As a refinement: if the New Way can easily be
>> backported to 2.5,
>
> Um - 2.5 is _done_. Released. In maintenance mode. New features will
> not be getting backported to a 2.5.x release.

Sorry if I was unclear there. I was not suggesting doing feature  
changes to 2.5, of course.

I meant backported such that they will _run_ on 2.5. And distributed  
as a private module to go along with my code. This was doable for  
many advancements in the 2.X series. For example, I could update my  
code to use heapq and UserDict.DictMixin (which were added in py  
2.3), even though it still ran on 2.2, by including a copy of those  
which were used if you're running 2.2. So code could be written to  
use a couple new features even though it runs on the old python. This  
is, of course, only useful if the things you want to backport are  
very small. Otherwise, you might as well just include a copy of all  
of Python. :)

But this is moot, anyhow, as it has been decreed that it will not be  
a goal to enable people to write code that runs on both 2.5 and 3.0.

James

From larry at hastings.org  Thu Jan 18 01:57:54 2007
From: larry at hastings.org (Larry Hastings)
Date: Wed, 17 Jan 2007 16:57:54 -0800
Subject: [Python-Dev] Deletion order when leaving a scope?
Message-ID: <45AEC612.1030001@hastings.org>



I just ran a quickie experiment and determined: when leaving a scope, 
variables are deleted FIFO, aka in the same order they were created.  
This surprised me; I'd expected them to be deleted LIFO, aka last 
first.  Why is it thus?  Is this behavior an important feature or an 
irrelevant side-effect?

Cheers,


/larry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070117/4b9c5cc3/attachment.html 

From jcarlson at uci.edu  Thu Jan 18 03:38:41 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Wed, 17 Jan 2007 18:38:41 -0800
Subject: [Python-Dev] Deletion order when leaving a scope?
In-Reply-To: <45AEC612.1030001@hastings.org>
References: <45AEC612.1030001@hastings.org>
Message-ID: <20070117183446.596B.JCARLSON@uci.edu>


Larry Hastings <larry at hastings.org> wrote:
> I just ran a quickie experiment and determined: when leaving a scope, 
> variables are deleted FIFO, aka in the same order they were created.  
> This surprised me; I'd expected them to be deleted LIFO, aka last 
> first.  Why is it thus?  Is this behavior an important feature or an 
> irrelevant side-effect?

If you are talking about the locals in the scope of a function, it is an
artifact of how the locals array is created.  That is to say, the locals
of a function are a flat array, and are decref'd in-order.  That
ordering is the original assignment ordering in the function, which is
an artifact of how the compiler goes from local name -> FAST_LOCALS
index.  I don't believe it is an "important feature", but to decref in
any other order would be silly and/or unintuitive.


 - Josiah


From nnorwitz at gmail.com  Thu Jan 18 08:21:11 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 17 Jan 2007 23:21:11 -0800
Subject: [Python-Dev] Pythn 2.5 buildbots do not build anymore?
In-Reply-To: <eolvnc$e63$1@sea.gmane.org>
References: <eolvnc$e63$1@sea.gmane.org>
Message-ID: <ee2a432c0701172321x5dcd2c02re361db50362237cc@mail.gmail.com>

On 1/17/07, Thomas Heller <theller at ctypes.org> wrote:
> Why don't the Python 2.5 buildbots build anymore?

It looks like there were no checkins that caused a build.  Changing
doc or Misc/NEWS is excluded.  I'm guessing that there is a bug in
buildbot that causes it to miss any changes from svnmerge.  Those
should have been caught ISTM.

I just checked in a change and the build looks like it's going.

http://www.python.org/dev/buildbot/all/changes/521

Thomas, you might want to look at that, cause it's something I don't
understand in cytpes. :-)

In other buildbot news, I fixed the failures on windows through some
lucky guessing.  I'm not sure the fixes are optimal, so someone with
more Windows knowledge should take a look at them.

I also got the alpha tru64 buildbot running again.  We still have a
few down, but they generally look in decent shape.

n

From thomas at python.org  Thu Jan 18 10:55:33 2007
From: thomas at python.org (Thomas Wouters)
Date: Thu, 18 Jan 2007 10:55:33 +0100
Subject: [Python-Dev] Deletion order when leaving a scope?
In-Reply-To: <45AEC612.1030001@hastings.org>
References: <45AEC612.1030001@hastings.org>
Message-ID: <9e804ac0701180155m7c054980k21a2f95db9783d88@mail.gmail.com>

On 1/18/07, Larry Hastings <larry at hastings.org> wrote:
>
>
>
> I just ran a quickie experiment and determined: when leaving a scope,
> variables are deleted FIFO, aka in the same order they were created.  This
> surprised me; I'd expected them to be deleted LIFO, aka last first.  Why is
> it thus?  Is this behavior an important feature or an irrelevant
> side-effect?
>

Please regard it as an irrelevant side-effect. If you want objects to be
cleaned up in a particular order, you should enforce it by having one of
them refer to the other. A great many details can affect the order in which
variables are cleaned up, and that only decreases refcounts of the actual
objects -- a great many other details can then affect the order in which any
objects left with a 0 refcount are actually cleaned up. Even not counting
the more complicated stuff like GC and funky __del__ methods, just having
'import *' or a bare 'exec' in your function can change the order of
DECREFs.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070118/56e22b46/attachment.html 

From ncoghlan at gmail.com  Thu Jan 18 13:02:49 2007
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Jan 2007 22:02:49 +1000
Subject: [Python-Dev] [Python-3000] dict.items as attributes [Was: The
 bytes type]
In-Reply-To: <fb6fbf560701170905l6cbfe29qf92fd87352ed3290@mail.gmail.com>
References: <20070112202227.17094.1740638192.divmod.xquotient.1876@joule.divmod.com>	<6e9196d20701121513i41ca78c6gc348bd52145ba8d4@mail.gmail.com>	<9e804ac0701150502i2e65430dtbe766f71eddfdc2b@mail.gmail.com>	<FAB32448-D96D-4DA6-AC7E-3BE2A7341120@fuhm.net>	<5.1.1.6.0.20070116114140.04e8b8f0@sparrow.telecommunity.com>	<00c601c73998$0a2f1b20$f100a8c0@ryoko>	<ca471dc20701161019g5658d346gd295e0b65724dbf0@mail.gmail.com>	<00d001c739af$36c15b50$f100a8c0@ryoko>	<fb6fbf560701161404v1986fffcv7eca0e0772850bd1@mail.gmail.com>	<9e804ac0701170419v1d802b5aqd49456231a75f1cc@mail.gmail.com>
	<fb6fbf560701170905l6cbfe29qf92fd87352ed3290@mail.gmail.com>
Message-ID: <45AF61E9.9090600@gmail.com>

Jim Jewett wrote:
> On 1/17/07, Thomas Wouters <thomas at python.org> wrote:
>> On 1/16/07, Jim Jewett <jimjjewett at gmail.com> wrote:
>>> Other than dict.items (and .keys and .values) returning a non-list,
>>> are there any other cases where the Py3K idiom can't already be used
>>> in (or at least backported to) Py 2.x?
> 
>> The aim for 2.6 should be to have all the new features that 3.0 has,
>> as well as full backward compatibility ...
> 
> And I'm asking if there are any real barriers to this.  Some people
> have suggested that they'll have to maintain separate code bases.  So
> far, the closest I've seen[1] to something that can't use
> common-subset is wanting an iterator over a mapping.

To help with the transition code, could we have a 'py3migration' module 
that looked something like:

   if sys.version_info >= (3, 0, 0):
       from _py2compat import *
   else:
       from _py3compat import *

By providing functions like dkeys(), ditems(), dvalues() and maybe a few 
others in that module, this could allow code that needs to work in both 
environments to be written, without causing significant pain to pure 
Python 2 code or to pure Python 3 code.

OK, so it would mean there's still some backward compatibility cruft in 
Py3k, but one module seems a small price to pay for a single obvious way 
to handle semantic changes in the builtin APIs.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Thu Jan 18 13:19:55 2007
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Jan 2007 22:19:55 +1000
Subject: [Python-Dev] buglet in long("123\0", 10)
In-Reply-To: <76fd5acf0701161057m37b72af5w85aafd0edc672db4@mail.gmail.com>
References: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>	<ee2a432c0701141514n391a707eh868c7e36a5c59fb@mail.gmail.com>	<76fd5acf0701141930w6c619e76obd2ee28c20f3c9f2@mail.gmail.com>	<ee2a432c0701151129k67d09f74i135fb795d953ac08@mail.gmail.com>
	<76fd5acf0701161057m37b72af5w85aafd0edc672db4@mail.gmail.com>
Message-ID: <45AF65EB.7060002@gmail.com>

Calvin Spealman wrote:
> Added a check in test_long.LongTest.test_misc() that long("123\0", 10)
> fails properly and adapted the patch to int_new to long_new. I get
> this weird feeling that if its impossible for the function
> (PyLong_FromString) to know if its being given bad data, having know
> way to know if the string is supposed to continue past the zero-byte,
> then doesn't it make sense to say that the function by design is
> broken?

It makes sense to say the function is being misused in this case - it's 
designed to convert *C* strings to PyLong objects, so the assumption 
that there are no embedded NULs is a valid one. That said, I think a 
better patch for 2.6 would be to provide a separate PyLong_FromPyString 
function which did the embedded NULL check (and update abstract.c to use 
that instead of its own internal function).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From amk at amk.ca  Thu Jan 18 14:07:02 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 18 Jan 2007 08:07:02 -0500
Subject: [Python-Dev] Testers wanted: mailbox.py bugfix
In-Reply-To: <20070117213658.GA31510@discworld.dyndns.org>
References: <20070117212048.GA17844@localhost.localdomain>
	<20070117213658.GA31510@discworld.dyndns.org>
Message-ID: <20070118130702.GA5248@localhost.localdomain>

On Wed, Jan 17, 2007 at 03:36:58PM -0600, Charles Cazabon wrote:
> I can't speak for MMDF or Babyl, but it's a well-known requirement that all
> programs on a system must use the same type of locking when mbox files are
> used; the administrator picks his favourite (fcntl, dotlocking, whatever) and
> then has to ensure all mbox-aware programs are compiled to use that locking
> style.

That's not actually the bug,  The problem is that code that does:

    <open mailbox>
    <read mailbox>
    ... wait around a bit ...
    <lock mailbox>
    <modify mailbox>
    <unlock mailbox>

can corrupt the mailbox if something else comes in and modifies the
mailbox during the "wait around a bit" phase.  This happens because an
table of contents is generated when the mailbox is read and then never
updated, missing changes made by other processes after the ToC is
read.

--amk

From ntoronto at cs.byu.edu  Thu Jan 18 18:04:42 2007
From: ntoronto at cs.byu.edu (Neil Toronto)
Date: Thu, 18 Jan 2007 10:04:42 -0700
Subject: [Python-Dev] Deletion order when leaving a scope?
In-Reply-To: <9e804ac0701180155m7c054980k21a2f95db9783d88@mail.gmail.com>
References: <45AEC612.1030001@hastings.org>
	<9e804ac0701180155m7c054980k21a2f95db9783d88@mail.gmail.com>
Message-ID: <45AFA8AA.50601@cs.byu.edu>

Thomas Wouters wrote:
> On 1/18/07, *Larry Hastings* <larry at hastings.org 
> <mailto:larry at hastings.org>> wrote:
>
>
>
>     I just ran a quickie experiment and determined: when leaving a
>     scope, variables are deleted FIFO, aka in the same order they were
>     created.  This surprised me; I'd expected them to be deleted LIFO,
>     aka last first.  Why is it thus?  Is this behavior an important
>     feature or an irrelevant side-effect?
>
>
> Please regard it as an irrelevant side-effect. If you want objects to 
> be cleaned up in a particular order, you should enforce it by having 
> one of them refer to the other. A great many details can affect the 
> order in which variables are cleaned up, and that only decreases 
> refcounts of the actual objects -- a great many other details can then 
> affect the order in which any objects left with a 0 refcount are 
> actually cleaned up. Even not counting the more complicated stuff like 
> GC and funky __del__ methods, just having 'import *' or a bare 'exec' 
> in your function can change the order of DECREFs.

I imagine this would be important to someone expecting system resources 
to be cleaned up, closed, deallocated, or returned inside of __del__ 
methods. Someone coming from C++ might expect LIFO behavior because 
common idioms like RAII (Resource Allocation Is Instantiation) don't 
work otherwise. A Java programmer wouldn't care, being used to cleaning 
up resources manually with a try...catch...finally.

I'm just putting a possible motivation on the concern. It happens that 
the Pythonic Way is also the Java Way in this area: don't expect any 
specific deletion order (don't even expect a guaranteed call to 
__del__), and manually clean up after yourself. As a warning, this has 
been the subject of a great many flame wars between C++ and Java 
programmers...

Neil


From ironfroggy at gmail.com  Thu Jan 18 19:35:59 2007
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Thu, 18 Jan 2007 13:35:59 -0500
Subject: [Python-Dev] buglet in long("123\0", 10)
In-Reply-To: <45AF65EB.7060002@gmail.com>
References: <ca471dc20701131917o62362adfub24bee25ab14d7e6@mail.gmail.com>
	<ee2a432c0701141514n391a707eh868c7e36a5c59fb@mail.gmail.com>
	<76fd5acf0701141930w6c619e76obd2ee28c20f3c9f2@mail.gmail.com>
	<ee2a432c0701151129k67d09f74i135fb795d953ac08@mail.gmail.com>
	<76fd5acf0701161057m37b72af5w85aafd0edc672db4@mail.gmail.com>
	<45AF65EB.7060002@gmail.com>
Message-ID: <76fd5acf0701181035y477a1603i28b7dd50d772645c@mail.gmail.com>

On 1/18/07, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Calvin Spealman wrote:
> > Added a check in test_long.LongTest.test_misc() that long("123\0", 10)
> > fails properly and adapted the patch to int_new to long_new. I get
> > this weird feeling that if its impossible for the function
> > (PyLong_FromString) to know if its being given bad data, having know
> > way to know if the string is supposed to continue past the zero-byte,
> > then doesn't it make sense to say that the function by design is
> > broken?
>
> It makes sense to say the function is being misused in this case - it's
> designed to convert *C* strings to PyLong objects, so the assumption
> that there are no embedded NULs is a valid one. That said, I think a
> better patch for 2.6 would be to provide a separate PyLong_FromPyString
> function which did the embedded NULL check (and update abstract.c to use
> that instead of its own internal function).

Thats more or less the route that would make sense to me! If they
would be accepted I would gladly provide patches for both int and long
types (and float? anything else?).

Is there often a usecase that anyone wants to use PyLong_FromString()
where the c-string is not the contents of PyString? I suppose when
used from extensions creating them.

> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>              http://www.boredomandlaziness.org
>


-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://ironfroggy-code.blogspot.com/

From brett at python.org  Thu Jan 18 20:53:54 2007
From: brett at python.org (Brett Cannon)
Date: Thu, 18 Jan 2007 11:53:54 -0800
Subject: [Python-Dev] Problem between deallocation of modules and
	func_globals
Message-ID: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>

I have discovered an issue relating to func_globals for functions and
the deallocation of the module it is contained within.  Let's say you
store a reference to the function encodings.search_function from the
'encodings' module (this came up in C code, but I don't see why it
couldn't happen in Python code).  Then you delete the one reference to
the module that is stored in sys.modules, leading to its deallocation.
 That triggers the setting of None to every value in
encodings.__dict__.

Oops, now the global namespace for that module has everything valued
at None.  The dict doesn't get deallocated since a reference is held
by encodings.search_function.func_globals and there is still a
reference to that (technically held in the interpreter's
codec_search_path field).  So the function can still execute, but
throws exceptions like AttributeError because a module variable that
once held a dict now has None and thus doesn't have the 'get' method.

My question is whether this is at all worth trying to rectify.  Since
Google didn't turn anything up I am going to guess this is not exactly
a common thing.  =)  That would lead me to believe some (probably
most) of you will say, "just leave it alone and work around it".

The other option I can think of is to store a reference to the module
instead of just to its __dict__ in the function.  The problem with
that is we end up with a circular dependency of the functions in
modules having a reference to the module but then the module having a
reference to the functions.  I tried not having the values in the
module's __dict__ set to None if the reference count was above 1 and
that solved this issue, but that leads to dangling references on
anything in that dict that does not have a reference stored away
somewhere else like encodings.search_function.

Anybody have any ideas on how to deal with this short of rewriting
some codecs stuff so that they don't depend on global state in the
module or just telling me to just live with it?

-Brett

From jcarlson at uci.edu  Fri Jan 19 01:37:42 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu, 18 Jan 2007 16:37:42 -0800
Subject: [Python-Dev] Problem between deallocation of modules and
	func_globals
In-Reply-To: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
Message-ID: <20070118163044.5989.JCARLSON@uci.edu>


"Brett Cannon" <brett at python.org> wrote:
> I have discovered an issue relating to func_globals for functions and
> the deallocation of the module it is contained within.  Let's say you
> store a reference to the function encodings.search_function from the
> 'encodings' module (this came up in C code, but I don't see why it
> couldn't happen in Python code).  Then you delete the one reference to
> the module that is stored in sys.modules, leading to its deallocation.
>  That triggers the setting of None to every value in
> encodings.__dict__.
[snip]
> Anybody have any ideas on how to deal with this short of rewriting
> some codecs stuff so that they don't depend on global state in the
> module or just telling me to just live with it?

I would have presumed that keeping a reference to a function should have
kept the module "alive".  Why?  If a function keeps a reference to a
module's globals, then even if the module is deleted, the module's
dictionary should still persist, because there exists a reference to it,
through the reference to the function.

Seems to me like a bug, but the bug could be fixed if the module's
dictionary kept a (circular) reference to the module object.  Who else
has been waiting for a __module__ attribute?


 - Josiah


From martin at v.loewis.de  Fri Jan 19 03:59:10 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 19 Jan 2007 03:59:10 +0100
Subject: [Python-Dev] adding _Py prefix to names in 2.5.1?
In-Reply-To: <ee2a432c0701162234p7a294d5aj3fe54ff9b55f479@mail.gmail.com>
References: <ee2a432c0701162234p7a294d5aj3fe54ff9b55f479@mail.gmail.com>
Message-ID: <45B033FE.5070908@v.loewis.de>

Neal Norwitz schrieb:
> http://python.org/sf/1637022 points out a problem caused by the lack
> of a _Py prefix on Ellipsis.  Most (all?) of the new AST names are not
> prefixed.  These are all meant to be internal names.  Are there any
> issues with changing this?  If we do so, it means that any module
> built with 2.5 that is using these names will fail to work in 2.5.1.
> No code outside the core *should* be using these names.

I'll look into this. I will create macros in the header file for them,
so that existing source code will continue to compile.

Regards,
Martin

From martin at v.loewis.de  Fri Jan 19 04:10:23 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 19 Jan 2007 04:10:23 +0100
Subject: [Python-Dev] Deletion order when leaving a scope?
In-Reply-To: <45AEC612.1030001@hastings.org>
References: <45AEC612.1030001@hastings.org>
Message-ID: <45B0369F.2040206@v.loewis.de>

Larry Hastings schrieb:
> I just ran a quickie experiment and determined: when leaving a scope,
> variables are deleted FIFO, aka in the same order they were created. 

Your experiment was apparently incomplete: Variables are *not* deleted
in the same order in which they are created:

py> class A:
...   def __init__(self, n):self.n = n
...   def __del__(self): print "Deleting", self.n
...
py> def f(x):
...   if x:
...     a = A("a")
...     b = A("b")
...   else:
...     b = A("b")
...     a = A("a")
...
py> f(0)
Deleting a
Deleting b

Here, it creates b first, then a (it's the else case), yet
deletes them in reverse order.

As others have pointed out, the deletion order is the one
indicated by the locals array:

py> f.func_code.co_varnames
('x', 'a', 'b')

Regards,
Martin

From martin at v.loewis.de  Fri Jan 19 04:18:02 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 19 Jan 2007 04:18:02 +0100
Subject: [Python-Dev] Problem between deallocation of modules
	and	func_globals
In-Reply-To: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
Message-ID: <45B0386A.4010908@v.loewis.de>

Brett Cannon schrieb:
> Anybody have any ideas on how to deal with this short of rewriting
> some codecs stuff so that they don't depend on global state in the
> module or just telling me to just live with it?

There is an old patch by Armin Rigo ( python.org/sf/812369 ), which
attempts to implement shutdown based on gc, rather than the explicit
clearing of modules. It would be good if that could be put to work;
I don't know what undesirable side effects doing so would cause.

Short of that, I don't think Python needs to support explicit deletion
of the encodings module from sys.modules when somebody still has a
reference to the search function. Don't do that, then.

Regards,
Martin

From martin at v.loewis.de  Fri Jan 19 04:20:04 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 19 Jan 2007 04:20:04 +0100
Subject: [Python-Dev] Problem between deallocation of modules
	and	func_globals
In-Reply-To: <20070118163044.5989.JCARLSON@uci.edu>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
	<20070118163044.5989.JCARLSON@uci.edu>
Message-ID: <45B038E4.90707@v.loewis.de>

Josiah Carlson schrieb:
> Seems to me like a bug, but the bug could be fixed if the module's
> dictionary kept a (circular) reference to the module object.  Who else
> has been waiting for a __module__ attribute?

This is the time machine at work:

py> import encodings
py> encodings.search_function.__module__
'encodings'

It's a string, rather than the module object, precisely to avoid cyclic
references.

Regards,
Martin

From draghuram at gmail.com  Fri Jan 19 04:28:56 2007
From: draghuram at gmail.com (Raghuram Devarakonda)
Date: Thu, 18 Jan 2007 22:28:56 -0500
Subject: [Python-Dev] adding _Py prefix to names in 2.5.1?
In-Reply-To: <45B033FE.5070908@v.loewis.de>
References: <ee2a432c0701162234p7a294d5aj3fe54ff9b55f479@mail.gmail.com>
	<45B033FE.5070908@v.loewis.de>
Message-ID: <2c51ecee0701181928v652aae87r8ea2d85d41a9b616@mail.gmail.com>

On 1/18/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Neal Norwitz schrieb:
> > http://python.org/sf/1637022 points out a problem caused by the lack
> > of a _Py prefix on Ellipsis.  Most (all?) of the new AST names are not
> > prefixed.  These are all meant to be internal names.  Are there any
> > issues with changing this?  If we do so, it means that any module
> > built with 2.5 that is using these names will fail to work in 2.5.1.
> > No code outside the core *should* be using these names.
>
> I'll look into this. I will create macros in the header file for them,
> so that existing source code will continue to compile.
>
> Regards,
> Martin

Hi,

I started to look into this based on Neal's suggestions (in offline
conversation). Please let me know if you want me to continue.

Thanks,
Raghu.

From mal at egenix.com  Fri Jan 19 10:25:58 2007
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 19 Jan 2007 10:25:58 +0100
Subject: [Python-Dev] Problem between deallocation of modules and
	func_globals
In-Reply-To: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
Message-ID: <45B08EA6.3040703@egenix.com>

On 2007-01-18 20:53, Brett Cannon wrote:
> I have discovered an issue relating to func_globals for functions and
> the deallocation of the module it is contained within.  Let's say you
> store a reference to the function encodings.search_function from the
> 'encodings' module (this came up in C code, but I don't see why it
> couldn't happen in Python code).  Then you delete the one reference to
> the module that is stored in sys.modules, leading to its deallocation.
>  That triggers the setting of None to every value in
> encodings.__dict__.
> 
> Oops, now the global namespace for that module has everything valued
> at None.  The dict doesn't get deallocated since a reference is held
> by encodings.search_function.func_globals and there is still a
> reference to that (technically held in the interpreter's
> codec_search_path field).  So the function can still execute, but
> throws exceptions like AttributeError because a module variable that
> once held a dict now has None and thus doesn't have the 'get' method.

That's a typical error situation you get in __del__ methods at
the time the interpreter is shut down.

The main reason for setting everything to None first is to
break circular references and make sure that at least some
of the object destructors can run.

> My question is whether this is at all worth trying to rectify.  Since
> Google didn't turn anything up I am going to guess this is not exactly
> a common thing.  =)  That would lead me to believe some (probably
> most) of you will say, "just leave it alone and work around it".

If you can come up with a better way, sure :-)

> The other option I can think of is to store a reference to the module
> instead of just to its __dict__ in the function.  The problem with
> that is we end up with a circular dependency of the functions in
> modules having a reference to the module but then the module having a
> reference to the functions.  I tried not having the values in the
> module's __dict__ set to None if the reference count was above 1 and
> that solved this issue, but that leads to dangling references on
> anything in that dict that does not have a reference stored away
> somewhere else like encodings.search_function.
> 
> Anybody have any ideas on how to deal with this short of rewriting
> some codecs stuff so that they don't depend on global state in the
> module or just telling me to just live with it?

I'm not exactly sure which global state you are referring to. The
aliase map, the cache used by the search function ?

Note that the search function registry is a global managed
in the thread state (it's not stored in any module).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 19 2007)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From ncoghlan at gmail.com  Fri Jan 19 14:20:55 2007
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Jan 2007 23:20:55 +1000
Subject: [Python-Dev] Deletion order when leaving a scope?
In-Reply-To: <45AFA8AA.50601@cs.byu.edu>
References: <45AEC612.1030001@hastings.org>	<9e804ac0701180155m7c054980k21a2f95db9783d88@mail.gmail.com>
	<45AFA8AA.50601@cs.byu.edu>
Message-ID: <45B0C5B7.2000701@gmail.com>

Neil Toronto wrote:
> I imagine this would be important to someone expecting system resources 
> to be cleaned up, closed, deallocated, or returned inside of __del__ 
> methods. Someone coming from C++ might expect LIFO behavior because 
> common idioms like RAII (Resource Allocation Is Instantiation) don't 
> work otherwise. A Java programmer wouldn't care, being used to cleaning 
> up resources manually with a try...catch...finally.
> 
> I'm just putting a possible motivation on the concern. It happens that 
> the Pythonic Way is also the Java Way in this area: don't expect any 
> specific deletion order (don't even expect a guaranteed call to 
> __del__), and manually clean up after yourself. As a warning, this has 
> been the subject of a great many flame wars between C++ and Java 
> programmers...

We know. Python 2.5 added a new statement (with) and a new standard 
library module (contextlib) to allow resource deallocation to be dealt 
with cleanly without requiring assumptions about the interpreter's 
memory model.

While RAII isn't mentioned explicitly in the corresponding PEP (PEP 
343), it was certainly a factor in the python-dev discussions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From itsanushshetty at gmail.com  Fri Jan 19 12:12:50 2007
From: itsanushshetty at gmail.com (Anush Shetty)
Date: Fri, 19 Jan 2007 16:42:50 +0530
Subject: [Python-Dev] BangPycon 2007 - First Indian Python Conference : Call
	for Participation
Message-ID: <636fe28f0701190312p5c1aa594ya716cedae333d00c@mail.gmail.com>

Hi,
The BangPypers (Bangalore Python Users Group) invite you to submit a
proposal to present a technical paper or tutorial,
or to host a meeting or BOF session at the first Indian Python conference
here at St Josephs PG College Auditorium in Bangalore,India on March
3rd,2007.

There are no fixed guidelines except for the below.

1) The talk has to be on a Pythonic Topic
2) It will be restricted to 45 mts with buffer for Q& A -10 mts
3) Though not mandatory, preference will be given to relevance of the topic
with the agenda of the conference being "to promote Python"

Though currently the plans of having a separate tutorial section is not on,
we still want to hear from you, and we can see whether something can be
fitted in...

There might be just 10 talks split into 2 tracks. So please hurry up and
send in your proposals.

Your Proposal should be mailed conference AT bangpypers DOT org


BangPycon
First Indian Python Conference,
March 3rd 2007,
St Josephs PG College, Bangalore, India.
Email: conference at bangpypers.org
 http://groups.yahoo.com/group/bangpypers
http://bangpypers.org
-- 
Cheers.
Anush Shetty
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070119/83918ca5/attachment.html 

From mwh at python.net  Fri Jan 19 20:28:16 2007
From: mwh at python.net (Michael Hudson)
Date: Fri, 19 Jan 2007 20:28:16 +0100
Subject: [Python-Dev] Deletion order when leaving a scope?
In-Reply-To: <45B0C5B7.2000701@gmail.com> (Nick Coghlan's message of "Fri,
	19 Jan 2007 23:20:55 +1000")
References: <45AEC612.1030001@hastings.org>
	<9e804ac0701180155m7c054980k21a2f95db9783d88@mail.gmail.com>
	<45AFA8AA.50601@cs.byu.edu> <45B0C5B7.2000701@gmail.com>
Message-ID: <87k5zi3jfz.fsf@starship.python.net>

Nick Coghlan <ncoghlan at gmail.com> writes:

> Neil Toronto wrote:
>> I imagine this would be important to someone expecting system resources 
>> to be cleaned up, closed, deallocated, or returned inside of __del__ 
>> methods. Someone coming from C++ might expect LIFO behavior because 
>> common idioms like RAII (Resource Allocation Is Instantiation) don't 
>> work otherwise. A Java programmer wouldn't care, being used to cleaning 
>> up resources manually with a try...catch...finally.
>> 
>> I'm just putting a possible motivation on the concern. It happens that 
>> the Pythonic Way is also the Java Way in this area: don't expect any 
>> specific deletion order (don't even expect a guaranteed call to 
>> __del__), and manually clean up after yourself. As a warning, this has 
>> been the subject of a great many flame wars between C++ and Java 
>> programmers...
>
> We know. Python 2.5 added a new statement (with) and a new standard 
> library module (contextlib) to allow resource deallocation to be dealt 
> with cleanly without requiring assumptions about the interpreter's 
> memory model.
>
> While RAII isn't mentioned explicitly in the corresponding PEP (PEP 
> 343), it was certainly a factor in the python-dev discussions.

It's mentioned in PEP 310, the predecessor to PEP 343.

Cheers,
mwh

-- 
 "I lacked the courage to investigate the weaknesses of the wicked,
  because I discovered they are the same as the weaknesses of the
  saintly."                       -- The Name Of The Rose, Umberto Eco

From jcarlson at uci.edu  Fri Jan 19 21:09:48 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 19 Jan 2007 12:09:48 -0800
Subject: [Python-Dev] Problem between deallocation of modules
	and	func_globals
In-Reply-To: <45B038E4.90707@v.loewis.de>
References: <20070118163044.5989.JCARLSON@uci.edu> <45B038E4.90707@v.loewis.de>
Message-ID: <20070119120334.599D.JCARLSON@uci.edu>


"Martin v. L?wis" <martin at v.loewis.de> wrote:
> 
> Josiah Carlson schrieb:
> > Seems to me like a bug, but the bug could be fixed if the module's
> > dictionary kept a (circular) reference to the module object.  Who else
> > has been waiting for a __module__ attribute?
> 
> This is the time machine at work:
> 
> py> import encodings
> py> encodings.search_function.__module__
> 'encodings'
> 
> It's a string, rather than the module object, precisely to avoid cyclic
> references.

I was saying that it would be nice if the following were true:

    >>> encodings.__module__
    <module 'encodings' from 'C:\python25\lib\encodings\__init__.pyc'>

That would make it easier for functions inside a module to pass around
references to the module namespace (I've had the need to do so before,
and have ended up using sys.modules[__name__], but that doesn't always
work).

So what if it is a circular reference (module references dict which
references module), we've got a GC which handles cycles just fine (when
users try not to be too smart).

 - Josiah


From brett at python.org  Fri Jan 19 22:33:35 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 19 Jan 2007 13:33:35 -0800
Subject: [Python-Dev] Problem between deallocation of modules and
	func_globals
In-Reply-To: <45B08EA6.3040703@egenix.com>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
	<45B08EA6.3040703@egenix.com>
Message-ID: <bbaeab100701191333y22a56852q18dc62070d5bd982@mail.gmail.com>

On 1/19/07, M.-A. Lemburg <mal at egenix.com> wrote:
> On 2007-01-18 20:53, Brett Cannon wrote:
> > I have discovered an issue relating to func_globals for functions and
> > the deallocation of the module it is contained within.  Let's say you
> > store a reference to the function encodings.search_function from the
> > 'encodings' module (this came up in C code, but I don't see why it
> > couldn't happen in Python code).  Then you delete the one reference to
> > the module that is stored in sys.modules, leading to its deallocation.
> >  That triggers the setting of None to every value in
> > encodings.__dict__.
> >
> > Oops, now the global namespace for that module has everything valued
> > at None.  The dict doesn't get deallocated since a reference is held
> > by encodings.search_function.func_globals and there is still a
> > reference to that (technically held in the interpreter's
> > codec_search_path field).  So the function can still execute, but
> > throws exceptions like AttributeError because a module variable that
> > once held a dict now has None and thus doesn't have the 'get' method.
>
> That's a typical error situation you get in __del__ methods at
> the time the interpreter is shut down.
>

Yeah, but in this case this is at the end of Py_Initialize() for the
stuff I am doing to the interpreter.  =)

> The main reason for setting everything to None first is to
> break circular references and make sure that at least some
> of the object destructors can run.
>

I know the reason, it just happens to occur at a bad time for me.

> > My question is whether this is at all worth trying to rectify.  Since
> > Google didn't turn anything up I am going to guess this is not exactly
> > a common thing.  =)  That would lead me to believe some (probably
> > most) of you will say, "just leave it alone and work around it".
>
> If you can come up with a better way, sure :-)
>
> > The other option I can think of is to store a reference to the module
> > instead of just to its __dict__ in the function.  The problem with
> > that is we end up with a circular dependency of the functions in
> > modules having a reference to the module but then the module having a
> > reference to the functions.  I tried not having the values in the
> > module's __dict__ set to None if the reference count was above 1 and
> > that solved this issue, but that leads to dangling references on
> > anything in that dict that does not have a reference stored away
> > somewhere else like encodings.search_function.
> >
> > Anybody have any ideas on how to deal with this short of rewriting
> > some codecs stuff so that they don't depend on global state in the
> > module or just telling me to just live with it?
>
> I'm not exactly sure which global state you are referring to. The
> aliase map, the cache used by the search function ?
>

encodings._cache .

> Note that the search function registry is a global managed
> in the thread state (it's not stored in any module).
>

Right, but that is not the issue.  If you have deleted the reference
to the encodings module from sys.modules it then sets encodings._cache
to None.  After the deletion, if you try to encode/decode a unicode
string you can an AttributeError about how encodings._cache does not
have a 'get' method since it is now None instead of a dict.  The
function is fine and still runs, it's just that the global state it
depends on is no longer the way it assume it should be.

-Brett

> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, Jan 19 2007)
> >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
> >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
>
> ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
>

From brett at python.org  Fri Jan 19 22:37:06 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 19 Jan 2007 13:37:06 -0800
Subject: [Python-Dev] Problem between deallocation of modules and
	func_globals
In-Reply-To: <45B0386A.4010908@v.loewis.de>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
	<45B0386A.4010908@v.loewis.de>
Message-ID: <bbaeab100701191337raeffb8ck1e1fdfee4892d9b5@mail.gmail.com>

On 1/18/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Brett Cannon schrieb:
> > Anybody have any ideas on how to deal with this short of rewriting
> > some codecs stuff so that they don't depend on global state in the
> > module or just telling me to just live with it?
>
> There is an old patch by Armin Rigo ( python.org/sf/812369 ), which
> attempts to implement shutdown based on gc, rather than the explicit
> clearing of modules. It would be good if that could be put to work;
> I don't know what undesirable side effects doing so would cause.
>

I will have a look.

> Short of that, I don't think Python needs to support explicit deletion
> of the encodings module from sys.modules when somebody still has a
> reference to the search function. Don't do that, then.

=)  Yeah.  As of this moment I am leaving __builtin__, exceptions,
encodings, codecs, encodings.utf_8, warnings, and sys.  I am deleting
all other modules after Py_Initialize finishes its thing.  I need to
do a security audit on all of those modules before I permanently let
them stick around (which is what I was hoping to avoid).  I am also
hoping make the sys module not be required to stay since it is only
there because of the amount of stuff that is put into the module
before its __dict__ is cached by the import machinery.

-Brett

From mal at egenix.com  Fri Jan 19 23:42:53 2007
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 19 Jan 2007 23:42:53 +0100
Subject: [Python-Dev] Problem between deallocation of modules
	and	func_globals
In-Reply-To: <bbaeab100701191333y22a56852q18dc62070d5bd982@mail.gmail.com>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>	<45B08EA6.3040703@egenix.com>
	<bbaeab100701191333y22a56852q18dc62070d5bd982@mail.gmail.com>
Message-ID: <45B1496D.4040403@egenix.com>

On 2007-01-19 22:33, Brett Cannon wrote:
>> That's a typical error situation you get in __del__ methods at
>> the time the interpreter is shut down.
>>
> 
> Yeah, but in this case this is at the end of Py_Initialize() for the
> stuff I am doing to the interpreter.  =)

Is that in some error branch of Py_Initialize() ? Otherwise
I don't see how the modules could get garbage-collected.

>> I'm not exactly sure which global state you are referring to. The
>> aliase map, the cache used by the search function ?
>>
> 
> encodings._cache .
> 
>> Note that the search function registry is a global managed
>> in the thread state (it's not stored in any module).
>>
> 
> Right, but that is not the issue.  If you have deleted the reference
> to the encodings module from sys.modules it then sets encodings._cache
> to None.  After the deletion, if you try to encode/decode a unicode
> string you can an AttributeError about how encodings._cache does not
> have a 'get' method since it is now None instead of a dict.  The
> function is fine and still runs, it's just that the global state it
> depends on is no longer the way it assume it should be.

While I could add some tricks to have the cache dictionary stay
alive even after the globals were set to None, I doubt that this
will really fix the problem.

The encoding package relies on the import mechanism, the codecs
module and the _codecs builtin module. Any of these could fail
to work depending on the order in which the modules get
GCed.

There's a reason why things in Py_Finalize() are as carefully
ordered :-) Perhaps we need to apply some reordering to the
steps in Py_Initialize() ?!

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 19 2007)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From brett at python.org  Sat Jan 20 00:01:42 2007
From: brett at python.org (Brett Cannon)
Date: Fri, 19 Jan 2007 15:01:42 -0800
Subject: [Python-Dev] Problem between deallocation of modules and
	func_globals
In-Reply-To: <45B1496D.4040403@egenix.com>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
	<45B08EA6.3040703@egenix.com>
	<bbaeab100701191333y22a56852q18dc62070d5bd982@mail.gmail.com>
	<45B1496D.4040403@egenix.com>
Message-ID: <bbaeab100701191501ueb33b48na4110b8548e237c1@mail.gmail.com>

On 1/19/07, M.-A. Lemburg <mal at egenix.com> wrote:
> On 2007-01-19 22:33, Brett Cannon wrote:
> >> That's a typical error situation you get in __del__ methods at
> >> the time the interpreter is shut down.
> >>
> >
> > Yeah, but in this case this is at the end of Py_Initialize() for the
> > stuff I am doing to the interpreter.  =)
>
> Is that in some error branch of Py_Initialize() ? Otherwise
> I don't see how the modules could get garbage-collected.
>

Nope, it's code I am adding to clean out sys.modules of stuff the user
didn't import themselves; it's for security reasons.

> >> I'm not exactly sure which global state you are referring to. The
> >> aliase map, the cache used by the search function ?
> >>
> >
> > encodings._cache .
> >
> >> Note that the search function registry is a global managed
> >> in the thread state (it's not stored in any module).
> >>
> >
> > Right, but that is not the issue.  If you have deleted the reference
> > to the encodings module from sys.modules it then sets encodings._cache
> > to None.  After the deletion, if you try to encode/decode a unicode
> > string you can an AttributeError about how encodings._cache does not
> > have a 'get' method since it is now None instead of a dict.  The
> > function is fine and still runs, it's just that the global state it
> > depends on is no longer the way it assume it should be.
>
> While I could add some tricks to have the cache dictionary stay
> alive even after the globals were set to None, I doubt that this
> will really fix the problem.
>
> The encoding package relies on the import mechanism, the codecs
> module and the _codecs builtin module. Any of these could fail
> to work depending on the order in which the modules get
> GCed.
>
> There's a reason why things in Py_Finalize() are as carefully
> ordered :-) Perhaps we need to apply some reordering to the
> steps in Py_Initialize() ?!
>

Nah, I just  need to not delete the modules.  =)

-Brett

From martin at v.loewis.de  Sat Jan 20 00:55:07 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 20 Jan 2007 00:55:07 +0100
Subject: [Python-Dev] Problem between deallocation of modules
	and	func_globals
In-Reply-To: <20070119120334.599D.JCARLSON@uci.edu>
References: <20070118163044.5989.JCARLSON@uci.edu> <45B038E4.90707@v.loewis.de>
	<20070119120334.599D.JCARLSON@uci.edu>
Message-ID: <45B15A5B.1020207@v.loewis.de>

Josiah Carlson schrieb:
> I was saying that it would be nice if the following were true:
> 
>     >>> encodings.__module__
>     <module 'encodings' from 'C:\python25\lib\encodings\__init__.pyc'>

Ah, ok. It would be somewhat confusing, though, that __module__ is
sometimes a module object, and sometimes a string (it certainly confused
me).

> So what if it is a circular reference (module references dict which
> references module), we've got a GC which handles cycles just fine (when
> users try not to be too smart).

That remains to be seen in practice. Currently, modules are explicitly
cleared at shutdown. I think any cycle with an object implementing
__del__ will keep loads of modules alive, noncollectable for GC.

Regards,
Martin

From raymond.hettinger at verizon.net  Sat Jan 20 01:28:17 2007
From: raymond.hettinger at verizon.net (Raymond Hettinger)
Date: Fri, 19 Jan 2007 16:28:17 -0800
Subject: [Python-Dev] Floor division
Message-ID: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>

I bumped into an oddity today:

    6.0 // 0.001 != math.floor(6.0 / 0.001)

In looking at Objects/floatobject.c, I was surprised to find that 
float_floor_division() is implemented in terms of float_divmod().  Does anyone 
know why it takes such a circuitous path?  I had expected something simpler and 
faster:

     return PyFloat_FromDouble(floor(a/b));


Raymond 


From guido at python.org  Sat Jan 20 02:32:27 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 19 Jan 2007 17:32:27 -0800
Subject: [Python-Dev] Floor division
In-Reply-To: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
Message-ID: <ca471dc20701191732q4d66e4ffue5c4ea394a857021@mail.gmail.com>

Probably because I tend not to know what I'm doing when numerics are
concerned. :-(

On 1/19/07, Raymond Hettinger <raymond.hettinger at verizon.net> wrote:
> I bumped into an oddity today:
>
>     6.0 // 0.001 != math.floor(6.0 / 0.001)
>
> In looking at Objects/floatobject.c, I was surprised to find that
> float_floor_division() is implemented in terms of float_divmod().  Does anyone
> know why it takes such a circuitous path?  I had expected something simpler and
> faster:
>
>      return PyFloat_FromDouble(floor(a/b));
>
>
> Raymond
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.peters at gmail.com  Sat Jan 20 02:33:23 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 19 Jan 2007 20:33:23 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
Message-ID: <1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>

[Raymond Hettinger]
> I bumped into an oddity today:
>
>     6.0 // 0.001 != math.floor(6.0 / 0.001)
>
> In looking at Objects/floatobject.c, I was surprised to find that
> float_floor_division() is implemented in terms of float_divmod().  Does anyone
> know why it takes such a circuitous path?  I had expected something simpler and
> faster:
>
>      return PyFloat_FromDouble(floor(a/b));

To preserve, so far as is possible with floats, that

(*)    a == (a//b)*b + a%b

In this case the binary double closest to decimal 0.001 is

0.001000000000000000020816681711721685132943093776702880859375

which is a little bit larger than 1/1000.  Therefore the mathematical
value of a/b is a little bit smaller than 6/(1/1000) = 6000, and the
true floor of the mathematically correct result is 5999.

a % b is always computed exactly (mathematical result == machine
result) under Python's definition whenever a and b have the same sign
(under C99's and the `decimal` module's definition it's always exact
regardless of signs), and getting the exact value for a%b implies the
exact value for a//b must also be returned (when possible) in order to
preserve identity (*) above.

IOW, since

>>> 6.0 % 0.001
0.00099999999999987512

it would be inconsistent to return 6000 for 6.0 // 0.001:

>>> 6.0 - 6000 * 0.001   # this isn't close to the value of a%b
0.0
>>> 6.0 - 5999 * 0.001   # but this is close
0.00099999999999944578

Note that two rounding errors occur in computing a - N*b via binary
doubles.  If there were no rounding errors, we'd have

    6 % b ==  6.0 - 5999 * b

exactly where

b =
0.001000000000000000020816681711721685132943093776702880859375

is the value actually stored in the machine for 0.001:

>>> import decimal
>>> decimal.getcontext().prec = 1000 # way more than needed
>>> import math
>>> # Set b to exact decimal value of binary double closest to 0.001.
>>> m, e = math.frexp(0.001)
>>> b = decimal.Decimal(int(m*2**53)) / decimal.Decimal(1 << (53-e))
>>> # Then 6%b is exactly equal to 6 - 5999*b
>>> 6 % b == 6 - 5999*b
True
>>> # Confirm that all decimal calculations were exact.
>>> decimal.getcontext().flags[decimal.Inexact]
False
>>> # Confirm that floor(6/b) is 5999.
>>> int(6/b)
5999
>>> print 6/b
5999.99999999999987509990972966989180234686226...

All that said, (*) doesn't make a lot of sense for floats to begin
with (for that matter, Python's definition of mod alone doesn't make
much sense for floats when the signs differ -- e.g.,

>>> -1 % 1e100
1e+100
>>> decimal.Decimal(-1) % decimal.Decimal("1e100")
Decimal("-1")

).

From kbk at shore.net  Sat Jan 20 04:15:13 2007
From: kbk at shore.net (Kurt B. Kaiser)
Date: Fri, 19 Jan 2007 22:15:13 -0500 (EST)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200701200315.l0K3FD59009106@bayview.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  423 open ( +2) /  3539 closed ( +9) /  3962 total (+11)
Bugs    :  960 open ( -3) /  6446 closed (+20) /  7406 total (+17)
RFE     :  258 open ( +3) /   249 closed ( +3) /   507 total ( +6)

New / Reopened Patches
______________________

Add aliases for latin7/9/10 charsets  (2007-01-13)
       http://python.org/sf/1634778  opened by  Christoph Zwerschke

htonl et al accept negative ints  (2007-01-14)
CLOSED http://python.org/sf/1635058  opened by  Mark Roberts

Logging Module - followfile patch  (2006-11-17)
CLOSED http://python.org/sf/1598415  reopened by  cjschr

CSV DictWriter Errors  (2007-01-14)
       http://python.org/sf/1635454  opened by  Mark Roberts

strptime %F and %T directives  (2007-01-14)
CLOSED http://python.org/sf/1635473  opened by  Mark Roberts

File Read/Write Flushing Patch  (2007-01-16)
       http://python.org/sf/1636874  opened by  jurojin

urllib: change email.Utils -> email.utils  (2007-01-16)
       http://python.org/sf/1637157  opened by  Russell Owen

urllib2: email.Utils->email.utils  (2007-01-16)
       http://python.org/sf/1637159  opened by  Russell Owen

smtplib email renames  (2007-01-16)
       http://python.org/sf/1637162  opened by  Russell Owen

Add httponly to Cookie module  (2007-01-17)
       http://python.org/sf/1638033  opened by  Arvin Schnell

compiler.pycodegen causes crashes when compiling 'with'  (2007-01-17)
       http://python.org/sf/1638243  opened by  kirat

Fix to the long("123\0", 10) problem  (2007-01-18)
       http://python.org/sf/1638879  opened by  Calvin Spealman

email.utils.parsedate documentation  (2007-01-19)
       http://python.org/sf/1639973  opened by  Mark Roberts

Patches Closed
______________

platform.py support for IronPython  (2006-09-23)
       http://python.org/sf/1563842  closed by  lemburg

Small upgrades to platform.platform()  (2005-11-10)
       http://python.org/sf/1352731  closed by  lemburg

pybench support for IronPython  (2006-09-23)
       http://python.org/sf/1563844  closed by  lemburg

Py3k: Fix pybench so it runs  (2007-01-12)
       http://python.org/sf/1634499  closed by  gvanrossum

Bug fixes for int unification branch  (2006-12-20)
       http://python.org/sf/1619846  closed by  gvanrossum

htonl et al accept negative ints  (2007-01-14)
       http://python.org/sf/1635058  closed by  gvanrossum

Logging Module - followfile patch  (2006-11-17)
       http://python.org/sf/1598415  closed by  vsajip

strptime %F and %T directives  (2007-01-14)
       http://python.org/sf/1635473  closed by  bcannon

BSD version of ctypes.util.find_library   (2006-12-07)
       http://python.org/sf/1610795  closed by  theller

New / Reopened Bugs
___________________

Problem running a subprocess  (2007-01-13)
       http://python.org/sf/1634739  opened by  Florent Rougon

locale 1251 does not convert to upper case properly  (2007-01-13)
       http://python.org/sf/1634774  opened by  Ivan Dobrokotov

Little mistake in docs  (2007-01-14)
       http://python.org/sf/1635217  opened by  anatoly techtonik

Add registry functions to windows postinstall  (2007-01-14)
       http://python.org/sf/1635335  opened by  anatoly techtonik

expanduser tests in test_posixpath fail if $HOME ends in a /  (2007-01-14)
CLOSED http://python.org/sf/1635353  opened by  Marien Zwart

Add command line help to windows unistall binary  (2007-01-14)
       http://python.org/sf/1635363  opened by  anatoly techtonik

ConfigParser does not quote %  (2007-01-15)
       http://python.org/sf/1635639  opened by  Mark Roberts

Interpreter seems to leak references after finalization  (2007-01-15)
       http://python.org/sf/1635741  opened by  B Sizer

description of the beta distribution is incorrect  (2007-01-15)
CLOSED http://python.org/sf/1635892  opened by  elgordo

Newline skipped in "for line in file"  (2007-01-16)
       http://python.org/sf/1636950  opened by  Andy Monthei

Python-2.5 segfault with tktreectrl  (2007-01-16)
CLOSED http://python.org/sf/1637022  opened by  klappnase

Python 2.5 fails to build on AIX 5.3 (xlc_r compiler)  (2007-01-16)
       http://python.org/sf/1637120  opened by  Orlando Irrazabal

mailbox.py uses old email names  (2007-01-16)
       http://python.org/sf/1637167  opened by  Russell Owen

make_table in difflib does not work with unicode  (2007-01-18)
       http://python.org/sf/1637850  opened by  y-unno

Problem packaging wx application with py2exe.  (2007-01-17)
CLOSED http://python.org/sf/1637943  opened by  Indy

typo    http://www.python.org/doc/current/tut/node10.html  (2007-01-17)
CLOSED http://python.org/sf/1637952  opened by  jim pruett

langref: missing item in numeric op list  (2007-01-17)
CLOSED http://python.org/sf/1637967  opened by  paul rubin

Incorrect documentation for random.betavariate()  (2007-01-18)
CLOSED http://python.org/sf/1638627  opened by  Troels Walsted Hansen

Bugs Closed
___________

class derived from float evaporates under +=  (2007-01-11)
       http://python.org/sf/1633630  closed by  gbrandl

file.encoding doesn't apply to file.write  (2004-06-07)
       http://python.org/sf/967986  closed by  lemburg

platform.libc_ver() fails on Cygwin  (2004-04-02)
       http://python.org/sf/928297  closed by  lemburg

subprocess swallows empty arguments under win32  (2007-01-12)
       http://python.org/sf/1634343  closed by  astrand

htonl, ntohl don't handle negative longs  (2006-12-20)
       http://python.org/sf/1619659  closed by  gvanrossum

expanduser tests in test_posixpath fail if $HOME ends in a /  (2007-01-14)
       http://python.org/sf/1635353  closed by  marienz

time.strftime() accepts format which time.strptime doesnt  (2007-01-11)
       http://python.org/sf/1633628  closed by  bcannon

SaveConfigParser.write() doesn't quote %-Sign  (2006-11-27)
       http://python.org/sf/1603688  closed by  gbrandl

logging module / wrong bytecode?  (2007-01-11)
       http://python.org/sf/1633605  closed by  vsajip

description of the beta distribution is incorrect  (2007-01-15)
       http://python.org/sf/1635892  closed by  rhettinger

os.path.expandvars deletes things on w32  (2001-12-18)
       http://python.org/sf/494589  closed by  sjoerd

Python-2.5 segfault with tktreectrl  (2007-01-16)
       http://python.org/sf/1637022  closed by  loewis

Problem packaging wx application with py2exe.  (2007-01-17)
       http://python.org/sf/1637943  closed by  bcannon

typo    http://www.python.org/doc/current/tut/node10.html  (2007-01-17)
       http://python.org/sf/1637952  closed by  gbrandl

langref: missing item in numeric op list  (2007-01-17)
       http://python.org/sf/1637967  closed by  gbrandl

Incorrect type in PyDict_Next() example code  (2007-01-05)
       http://python.org/sf/1629125  closed by  gbrandl

website issue reporter down  (2007-01-03)
       http://python.org/sf/1627036  closed by  gbrandl

Incorrect documentation for random.betavariate()  (2007-01-18)
       http://python.org/sf/1638627  closed by  loewis

Garbage output to file of specific size  (2007-01-08)
       http://python.org/sf/1630894  deleted by  mculbert

Forwarding events and Tk.mainloop problem  (2006-05-05)
       http://python.org/sf/1482402  closed by  loewis

New / Reopened RFE
__________________

csv.DictWriter: Include offending name in error message  (2007-01-13)
       http://python.org/sf/1634717  opened by  Gabriel Genellina

Please provide rsync-method in the urllib[2] module  (2007-01-13)
       http://python.org/sf/1634770  opened by  Matthias Klose

if __name__=='__main__' missing in tutorial  (2007-01-17)
       http://python.org/sf/1637365  opened by  Gabriel Genellina

Empty class 'Object'  (2007-01-17)
       http://python.org/sf/1637926  opened by  kxroberto

add type defintion support  (2007-01-18)
CLOSED http://python.org/sf/1639002  opened by  djnet

RFE Closed
__________

logging.RotatingFileHandler has no "infinite" backupCount  (2006-09-28)
       http://python.org/sf/1567331  closed by  vsajip

Print full exceptions as they occur in logging  (2006-09-06)
       http://python.org/sf/1553380  closed by  vsajip

add type defintion support  (2007-01-18)
       http://python.org/sf/1639002  closed by  gbrandl


From arigo at tunes.org  Sat Jan 20 12:29:34 2007
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 20 Jan 2007 12:29:34 +0100
Subject: [Python-Dev] Floor division
In-Reply-To: <1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
Message-ID: <20070120112933.GA11566@code0.codespeak.net>

Hi Tim,

On Fri, Jan 19, 2007 at 08:33:23PM -0500, Tim Peters wrote:
> >>> decimal.Decimal(-1) % decimal.Decimal("1e100")
> Decimal("-1")

BTW - isn't that case in contradiction with the general Python rule that
if b > 0, then a % b should return a number between 0 included and b
excluded?  We try hard to do that for ints, longs and floats.  The fact
that it works differently with Decimal could be unexpected.


A bientot,

Armin

From mal at egenix.com  Sat Jan 20 12:31:45 2007
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 20 Jan 2007 12:31:45 +0100
Subject: [Python-Dev] Problem between deallocation of modules
	and	func_globals
In-Reply-To: <bbaeab100701191501ueb33b48na4110b8548e237c1@mail.gmail.com>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>	<45B08EA6.3040703@egenix.com>	<bbaeab100701191333y22a56852q18dc62070d5bd982@mail.gmail.com>	<45B1496D.4040403@egenix.com>
	<bbaeab100701191501ueb33b48na4110b8548e237c1@mail.gmail.com>
Message-ID: <45B1FDA1.3000305@egenix.com>

On 2007-01-20 00:01, Brett Cannon wrote:
> On 1/19/07, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 2007-01-19 22:33, Brett Cannon wrote:
>>>> That's a typical error situation you get in __del__ methods at
>>>> the time the interpreter is shut down.
>>>>
>>> Yeah, but in this case this is at the end of Py_Initialize() for the
>>> stuff I am doing to the interpreter.  =)
>> Is that in some error branch of Py_Initialize() ? Otherwise
>> I don't see how the modules could get garbage-collected.
>>
> 
> Nope, it's code I am adding to clean out sys.modules of stuff the user
> didn't import themselves; it's for security reasons.

I'm not sure whether that's really going to increase
security: unloading of modules usually isn't safe and you
cannot be sure that it's possible to reinitialize a C
module once it has been loaded in the process. For Python
modules this is often possible, but there still may be
side-effects of the import that you cannot easily undo.

Perhaps you should just move those modules out to a different
dictionary and keep track of it in the import mechanism, so
that while you can't access the module directly via sys.modules,
the import mechanism still knows that it has been loaded and
reinserts it into sys.modules if it gets imported again.

I think that you get more security by explicitly
limiting which modules and packages you allow to be imported
in the first place and restricting what can be done with
sys.path and sys.modules.

>>>> I'm not exactly sure which global state you are referring to. The
>>>> aliase map, the cache used by the search function ?
>>>>
>>> encodings._cache .
>>>
>>>> Note that the search function registry is a global managed
>>>> in the thread state (it's not stored in any module).
>>>>
>>> Right, but that is not the issue.  If you have deleted the reference
>>> to the encodings module from sys.modules it then sets encodings._cache
>>> to None.  After the deletion, if you try to encode/decode a unicode
>>> string you can an AttributeError about how encodings._cache does not
>>> have a 'get' method since it is now None instead of a dict.  The
>>> function is fine and still runs, it's just that the global state it
>>> depends on is no longer the way it assume it should be.
>> While I could add some tricks to have the cache dictionary stay
>> alive even after the globals were set to None, I doubt that this
>> will really fix the problem.
>>
>> The encoding package relies on the import mechanism, the codecs
>> module and the _codecs builtin module. Any of these could fail
>> to work depending on the order in which the modules get
>> GCed.
>>
>> There's a reason why things in Py_Finalize() are as carefully
>> ordered :-) Perhaps we need to apply some reordering to the
>> steps in Py_Initialize() ?!
>
> 
> Nah, I just  need to not delete the modules.  =)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 20 2007)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From ndbecker2 at gmail.com  Sat Jan 20 16:25:54 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sat, 20 Jan 2007 10:25:54 -0500
Subject: [Python-Dev] syntax misfeature (exception)
Message-ID: <eotca2$obn$1@sea.gmane.org>

I accidentally wrote:

try:
 ...
except a,b:

rather than:

try
 ...
except (a,b):

It appears that the 1st example syntax is silently accepted, but doesn't
seem to work.  Is this true?  If so, I'd say it's a wart.


From g.brandl at gmx.net  Sat Jan 20 16:28:31 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 20 Jan 2007 15:28:31 +0000
Subject: [Python-Dev] syntax misfeature (exception)
In-Reply-To: <eotca2$obn$1@sea.gmane.org>
References: <eotca2$obn$1@sea.gmane.org>
Message-ID: <eotcf0$ot0$1@sea.gmane.org>

Neal Becker schrieb:
> I accidentally wrote:
> 
> try:
>  ...
> except a,b:
> 
> rather than:
> 
> try
>  ...
> except (a,b):
> 
> It appears that the 1st example syntax is silently accepted, but doesn't
> seem to work.  Is this true?  If so, I'd say it's a wart.

Both have a meaning: The first assigns the exception object to b, while the
second catches exception types a and b.

BTW, in Python 3.0, the first will be spelled "except a as b".

Please post questions about using the language to the comp.lang.python group.

cheers,
Georg


From ndbecker2 at gmail.com  Sat Jan 20 16:36:48 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sat, 20 Jan 2007 10:36:48 -0500
Subject: [Python-Dev] syntax misfeature (exception)
References: <eotca2$obn$1@sea.gmane.org> <eotcf0$ot0$1@sea.gmane.org>
Message-ID: <eotcug$qd4$1@sea.gmane.org>

Georg Brandl wrote:

> Neal Becker schrieb:
>> I accidentally wrote:
>> 
>> try:
>>  ...
>> except a,b:
>> 
>> rather than:
>> 
>> try
>>  ...
>> except (a,b):
>> 
>> It appears that the 1st example syntax is silently accepted, but doesn't
>> seem to work.  Is this true?  If so, I'd say it's a wart.
> 
> Both have a meaning: The first assigns the exception object to b, while
> the second catches exception types a and b.
> 
> BTW, in Python 3.0, the first will be spelled "except a as b".
> 
> Please post questions about using the language to the comp.lang.python
> group.
> 
It's not a question, it's a critique.  I believe this is a misfeature since
it's so easy to make this mistake.


From jcarlson at uci.edu  Sat Jan 20 19:17:37 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sat, 20 Jan 2007 10:17:37 -0800
Subject: [Python-Dev] syntax misfeature (exception)
In-Reply-To: <eotcug$qd4$1@sea.gmane.org>
References: <eotcf0$ot0$1@sea.gmane.org> <eotcug$qd4$1@sea.gmane.org>
Message-ID: <20070120101456.59A9.JCARLSON@uci.edu>


Neal Becker <ndbecker2 at gmail.com> wrote:
[snip]
> It's not a question, it's a critique.  I believe this is a misfeature since
> it's so easy to make this mistake.

And it is going away with Py3k.  Making it go away for Python 2.6 would
either allow for two syntaxes to do the same thing, or would require
everyone to change their except clauses.  Neither is very desireable
(especially if writing code for 2.6 makes it not work for 2.5).


 - Josiah


From brett at python.org  Sat Jan 20 19:23:30 2007
From: brett at python.org (Brett Cannon)
Date: Sat, 20 Jan 2007 10:23:30 -0800
Subject: [Python-Dev] Problem between deallocation of modules and
	func_globals
In-Reply-To: <45B1FDA1.3000305@egenix.com>
References: <bbaeab100701181153x405bf17fl3bd9b8f5adae05b0@mail.gmail.com>
	<45B08EA6.3040703@egenix.com>
	<bbaeab100701191333y22a56852q18dc62070d5bd982@mail.gmail.com>
	<45B1496D.4040403@egenix.com>
	<bbaeab100701191501ueb33b48na4110b8548e237c1@mail.gmail.com>
	<45B1FDA1.3000305@egenix.com>
Message-ID: <bbaeab100701201023s3592f821q5652d3d33d52f6ab@mail.gmail.com>

On 1/20/07, M.-A. Lemburg <mal at egenix.com> wrote:
> On 2007-01-20 00:01, Brett Cannon wrote:
> > On 1/19/07, M.-A. Lemburg <mal at egenix.com> wrote:
> >> On 2007-01-19 22:33, Brett Cannon wrote:
> >>>> That's a typical error situation you get in __del__ methods at
> >>>> the time the interpreter is shut down.
> >>>>
> >>> Yeah, but in this case this is at the end of Py_Initialize() for the
> >>> stuff I am doing to the interpreter.  =)
> >> Is that in some error branch of Py_Initialize() ? Otherwise
> >> I don't see how the modules could get garbage-collected.
> >>
> >
> > Nope, it's code I am adding to clean out sys.modules of stuff the user
> > didn't import themselves; it's for security reasons.
>
> I'm not sure whether that's really going to increase
> security: unloading of modules usually isn't safe and you
> cannot be sure that it's possible to reinitialize a C
> module once it has been loaded in the process. For Python
> modules this is often possible, but there still may be
> side-effects of the import that you cannot easily undo.
>
> Perhaps you should just move those modules out to a different
> dictionary and keep track of it in the import mechanism, so
> that while you can't access the module directly via sys.modules,
> the import mechanism still knows that it has been loaded and
> reinserts it into sys.modules if it gets imported again.
>

That's an idea.

> I think that you get more security by explicitly
> limiting which modules and packages you allow to be imported
> in the first place and restricting what can be done with
> sys.path and sys.modules.
>

That's what I am doing.  I just wanted to simplify things by having
import not worry about what is already in sys.modules and just always
assume what is there is safe.

-Brett

From anthony at interlink.com.au  Sun Jan 21 05:51:26 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sun, 21 Jan 2007 15:51:26 +1100
Subject: [Python-Dev] syntax misfeature (exception)
In-Reply-To: <20070120101456.59A9.JCARLSON@uci.edu>
References: <eotcf0$ot0$1@sea.gmane.org> <eotcug$qd4$1@sea.gmane.org>
	<20070120101456.59A9.JCARLSON@uci.edu>
Message-ID: <200701211551.28603.anthony@interlink.com.au>

On Sunday 21 January 2007 05:17, Josiah Carlson wrote:
> Neal Becker <ndbecker2 at gmail.com> wrote:
> [snip]
>
> > It's not a question, it's a critique.  I believe this is a
> > misfeature since it's so easy to make this mistake.
>
> And it is going away with Py3k.  Making it go away for Python 2.6
> would either allow for two syntaxes to do the same thing, or
> would require everyone to change their except clauses.  Neither
> is very desireable (especially if writing code for 2.6 makes it
> not work for 2.5).

Note that we do plan to add "except a as b" to 2.6 - we're just not 
ripping out the old way of doing it.

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From pje at telecommunity.com  Sun Jan 21 15:12:54 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 21 Jan 2007 09:12:54 -0500
Subject: [Python-Dev] syntax misfeature (exception)
In-Reply-To: <20070120101456.59A9.JCARLSON@uci.edu>
References: <eotcf0$ot0$1@sea.gmane.org> <eotcug$qd4$1@sea.gmane.org>
	<20070120101456.59A9.JCARLSON@uci.edu>
Message-ID: <7.0.1.0.0.20070121091208.02108960@telecommunity.com>

At 01:17 PM 1/20/2007, Josiah Carlson wrote:

>Neal Becker <ndbecker2 at gmail.com> wrote:
>[snip]
> > It's not a question, it's a critique.  I believe this is a misfeature since
> > it's so easy to make this mistake.
>
>And it is going away with Py3k.  Making it go away for Python 2.6 would
>either allow for two syntaxes to do the same thing, or would require
>everyone to change their except clauses.  Neither is very desireable
>(especially if writing code for 2.6 makes it not work for 2.5).

With both, you can choose whether you'd rather your code be backward 
compatible with 2.5, or forward-compatible with 3.0.  So 2.6 has to 
have both syntaxes.



>  - Josiah
>
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: 
>http://mail.python.org/mailman/options/python-dev/pje%40telecommunity.com


From jcarlson at uci.edu  Mon Jan 22 00:17:44 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 21 Jan 2007 15:17:44 -0800
Subject: [Python-Dev] syntax misfeature (exception)
In-Reply-To: <7.0.1.0.0.20070121091208.02108960@telecommunity.com>
References: <20070120101456.59A9.JCARLSON@uci.edu>
	<7.0.1.0.0.20070121091208.02108960@telecommunity.com>
Message-ID: <20070121151717.59B3.JCARLSON@uci.edu>


"Phillip J. Eby" <pje at telecommunity.com> wrote:
> 
> At 01:17 PM 1/20/2007, Josiah Carlson wrote:
> 
> >Neal Becker <ndbecker2 at gmail.com> wrote:
> >[snip]
> > > It's not a question, it's a critique.  I believe this is a misfeature since
> > > it's so easy to make this mistake.
> >
> >And it is going away with Py3k.  Making it go away for Python 2.6 would
> >either allow for two syntaxes to do the same thing, or would require
> >everyone to change their except clauses.  Neither is very desireable
> >(especially if writing code for 2.6 makes it not work for 2.5).
> 
> With both, you can choose whether you'd rather your code be backward 
> compatible with 2.5, or forward-compatible with 3.0.  So 2.6 has to 
> have both syntaxes.

If it's already been decided; great!

 - Josiah


From tim.peters at gmail.com  Mon Jan 22 03:08:18 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Sun, 21 Jan 2007 21:08:18 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <20070120112933.GA11566@code0.codespeak.net>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
Message-ID: <1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>

[Tim Peters]
>> ...
>> >>> decimal.Decimal(-1) % decimal.Decimal("1e100")
>> Decimal("-1")

[Armin Rigo]
> BTW - isn't that case in contradiction with the general Python rule that
> if b > 0, then a % b should return a number between 0 included and b
> excluded?

Sure.

> We try hard to do that for ints, longs and floats.

But fail in this case for floats:

>>> -1 % 1e100 < 1e100
False

Can't win.  The infinite-precision result is the mathematical
F(1e100)-1, where F(1e100) is the binary float closest to 10**100, but
F(1e100)-1 isn't representable as a float -- it rounds back up to
F(1e100):

>>> -1 % 1e100 == 1e100
True

There simply is no /representable/ float value in [0, 10**100)
congruent to -1 modulo 10**100 (or modulo F(1e100)), so it's
impossible to return a non-surprising (to everyone) result in that
range.  0 and 1e100 are in some sense "the best" answers in that
range, both off by only 1 part in F(1e100), the smallest possible
error among representable floats in that range.  -1/1e100 certainly
isn't 0, so

    -1 // 1e100 == -1.0

is required.  Picking -1 % 1e100 == 1e100 then follows, to try to preserve that

    a = (a//b)*b + a%b

as closely as is possible.

Ints and longs never have problems here, because the exact % result is
always exactly representable.

That isn't true of floats (whether binary or decimal), but under a
different definition of "mod" the mathematically exact result is
always exactly representable:  a%b takes the sign of `a` rather than
the sign of `b`.  C's fmod (Python's math.fmod), and the proposed
standard for decimal arithmetic implemented by the `decimal` module,
use that meaning for "mod" instead.

>>> math.fmod(-1, 1e100)
-1.0

> The fact that it works differently with Decimal could be unexpected.

Yup.  See "can't win" above :-(

Another good definition of "mod" for floats is to return the
representative of smallest absolute value; i.e., satisfy

    abs(a%b) <= abs(b) / 2

The mathematically exact value for that is also exactly representable
(BTW, the proposed standard for decimal arithmetic calls this
"remainder-near", as opposed to "remainder").

It's just a fact that different definitions of mod are most useful
most often depending on data type.  Python's is good for integers and
often sucks for floats.  The C99 and `decimal` definition(s) is/are
good for floats and often suck(s) for integers.  Trying to pretend
that integers are a subset of floats can't always work ;-)

From guido at python.org  Mon Jan 22 04:40:21 2007
From: guido at python.org (Guido van Rossum)
Date: Sun, 21 Jan 2007 19:40:21 -0800
Subject: [Python-Dev] Floor division
In-Reply-To: <1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
Message-ID: <ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>

On 1/21/07, Tim Peters <tim.peters at gmail.com> wrote:
> It's just a fact that different definitions of mod are most useful
> most often depending on data type.  Python's is good for integers and
> often sucks for floats.  The C99 and `decimal` definition(s) is/are
> good for floats and often suck(s) for integers.  Trying to pretend
> that integers are a subset of floats can't always work ;-)

That really sucks, especially since the whole point of making int
division return a float was to make the integers embedded in the
floats... I think the best solution would be to remove the definition
of % (and then also for divmod()) for floats altogether, deferring to
math.fmod() instead.

The ints aren't really embedded in Decimal, so we don't have to do
that there (but we could).

The floats currently aren't embedded in complex, since f.real and
f.imag don't work for floats (and math.sqrt(-1.0) doesn't return 1.0j,
and many other anomalies). That was a conscious decision because
complex numbers are useless for most folks. But is it still the right
decision, given what we're trying to do with int and float?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.peters at gmail.com  Mon Jan 22 06:21:35 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Mon, 22 Jan 2007 00:21:35 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>
Message-ID: <1f7befae0701212121i6ae713b1v98f527113afc0323@mail.gmail.com>

...

[Tim]
>> It's just a fact that different definitions of mod are most useful
>> most often depending on data type.  Python's is good for integers and
>> often sucks for floats.  The C99 and `decimal` definition(s) is/are
>> good for floats and often suck(s) for integers.  Trying to pretend
>> that integers are a subset of floats can't always work ;-)

[Guido]
> That really sucks, especially since the whole point of making int
> division return a float was to make the integers embedded in the
> floats... I think the best solution would be to remove the definition
> of % (and then also for divmod()) for floats altogether, deferring to
> math.fmod() instead.

In Python 2?  I thought this was already (semi)settled for Py3K --
back last May, on the Py3K list, in a near-repetition of the current
thread:

    [Tim]
    I'd be happiest if P3K floats didn't support __mod__ or __divmod__ at
    all.  Floating mod is so rare it doesn't need syntactic support, and
    the try-to-be-like-integer __mod__ and __divmod__ floats support now
    can deliver surprises to all users incautious enough to use them.

    [Guido]
    OK, this makes sense. I've added this to PEP 3100.

> The ints aren't really embedded in Decimal, so we don't have to do
> that there (but we could).

Floor division is an odd beast for floats, and I don't know of a good
use for it.  As Raymond's original example in this thread showed, it's
not always the case that

    math.floor(x/y) == x // y

The rounding error in computing x/y can cause floor() to see an exact
integer coming in even when the true value of x/y is a little smaller
than that integer (of course it's also possible for x/y to overflow).
This is independent of fp base (it's "just as true" for decimal floats
as for binary floats).

The `decimal` module also has two distinct flavors of "mod", neither
of which match Python's integer-mod definition:

>>> decimal.Decimal(7).__mod__(10)
Decimal("7")
>>> decimal.Decimal(7).remainder_near(10)
Decimal("-3")
>>> decimal.Decimal(-7).__mod__(10)
Decimal("-7")
>>> decimal.Decimal(-7).remainder_near(10)
Decimal("3")

But, again, I think floating mod is so rare it doesn't need syntactic
support, and especially not when there's more than one implementation
-- and `decimal` is a floating type (in Cowlishaw's REXX, this was the
/only/ numeric type, so there was more pressure there to give it a
succinct spelling).

> The floats currently aren't embedded in complex, since f.real and
> f.imag don't work for floats (and math.sqrt(-1.0) doesn't return 1.0j,
> and many other anomalies). That was a conscious decision because
> complex numbers are useless for most folks. But is it still the right
> decision, given what we're trying to do with int and float?

Sounds like a "purity" thing.  The "pure form" of the original
/intent/ was probably just that nobody should get a complex result
from non-complex inputs (they should see an exception instead) unless
they use a `cmath` function to prove they know what they're doing up
front.

I still think that has pragmatic value, since things like sqrt(-1) and
asin(1.05) really are signs of logic errors for most programs.

But under that view, there's nothing surprising about, e.g.,
(math.pi).imag returning 0.0 -- or (3).imag returning 0.0 either.
That sounds fine to me (if someone cares enough to implement it).
Unclear what very_big_int.real should do (it can lose information to
rounding if the int > 2**53, or even overflow -- maybe it should just
return the int!).

From martin at v.loewis.de  Mon Jan 22 08:51:32 2007
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 22 Jan 2007 08:51:32 +0100
Subject: [Python-Dev] Eliminating f_tstate
Message-ID: <45B46D04.5060103@v.loewis.de>

Bug #1579370 reports a crash when accessing the thread state of
a terminated thread, when releasing a generator object.

In analysing the problem, I found that f_tstate doesn't have much
utility: it is used in very few places, and in these places, it
might be as good or better to use the *current* thread state
(rather than the frame's thread state); in some (most?) of these
places, the current thread should be identical to the frame's
thread, anyway.

So I would like to propose that the f_tstate member is removed
from the frame object.

For Python 2.5, for compatibility, it probably has to remain
where it is, and only PyTraceBack_Here should stop using it.
As a consequence, a generator .send() makes exceptions
occur in the current thread, not in the thread where the
generator was created.

What do you think?

Regards,
Martin


From mwh at python.net  Mon Jan 22 09:34:03 2007
From: mwh at python.net (Michael Hudson)
Date: Mon, 22 Jan 2007 09:34:03 +0100
Subject: [Python-Dev] Eliminating f_tstate
In-Reply-To: <45B46D04.5060103@v.loewis.de> (Martin v. =?iso-8859-1?Q?L=F6?=
	=?iso-8859-1?Q?wis's?= message of "Mon, 22 Jan 2007 08:51:32 +0100")
References: <45B46D04.5060103@v.loewis.de>
Message-ID: <87ac0b31fo.fsf@starship.python.net>

"Martin v. L?wis" <martin at v.loewis.de> writes:

> Bug #1579370 reports a crash when accessing the thread state of
> a terminated thread, when releasing a generator object.
>
> In analysing the problem, I found that f_tstate doesn't have much
> utility: it is used in very few places, and in these places, it
> might be as good or better to use the *current* thread state
> (rather than the frame's thread state); in some (most?) of these
> places, the current thread should be identical to the frame's
> thread, anyway.
>
> So I would like to propose that the f_tstate member is removed
> from the frame object.
>
> For Python 2.5, for compatibility, it probably has to remain
> where it is, and only PyTraceBack_Here should stop using it.
> As a consequence, a generator .send() makes exceptions
> occur in the current thread, not in the thread where the
> generator was created.
>
> What do you think?

Without having read the code in detail, I think what you say makes a
great deal of sense.

Cheers,
mwh

-- 
  (FREE|OPEN) BSD: Shire horse. Solid, reliable, only occasionally
  prone to crushing you against a wall and then only because
  you've told it to without knowing.
                           -- Jim's pedigree of operating systems, asr

From guido at python.org  Mon Jan 22 16:57:02 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 Jan 2007 07:57:02 -0800
Subject: [Python-Dev] Floor division
In-Reply-To: <1f7befae0701212121i6ae713b1v98f527113afc0323@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>
	<1f7befae0701212121i6ae713b1v98f527113afc0323@mail.gmail.com>
Message-ID: <ca471dc20701220757gcb24a5dkcf90b39ba22943f6@mail.gmail.com>

On 1/21/07, Tim Peters <tim.peters at gmail.com> wrote:
> [Tim]
> >> It's just a fact that different definitions of mod are most useful
> >> most often depending on data type.  Python's is good for integers and
> >> often sucks for floats.  The C99 and `decimal` definition(s) is/are
> >> good for floats and often suck(s) for integers.  Trying to pretend
> >> that integers are a subset of floats can't always work ;-)
>
> [Guido]
> > That really sucks, especially since the whole point of making int
> > division return a float was to make the integers embedded in the
> > floats... I think the best solution would be to remove the definition
> > of % (and then also for divmod()) for floats altogether, deferring to
> > math.fmod() instead.
>
> In Python 2?

No way!

> I thought this was already (semi)settled for Py3K --
> back last May, on the Py3K list, in a near-repetition of the current
> thread:
>
>     [Tim]
>     I'd be happiest if P3K floats didn't support __mod__ or __divmod__ at
>     all.  Floating mod is so rare it doesn't need syntactic support, and
>     the try-to-be-like-integer __mod__ and __divmod__ floats support now
>     can deliver surprises to all users incautious enough to use them.
>
>     [Guido]
>     OK, this makes sense. I've added this to PEP 3100.

Oops, you're right. I'm suffering from memory loss. :-(

> > The ints aren't really embedded in Decimal, so we don't have to do
> > that there (but we could).
>
> Floor division is an odd beast for floats, and I don't know of a good
> use for it.  As Raymond's original example in this thread showed, it's
> not always the case that
>
>     math.floor(x/y) == x // y
>
> The rounding error in computing x/y can cause floor() to see an exact
> integer coming in even when the true value of x/y is a little smaller
> than that integer (of course it's also possible for x/y to overflow).
> This is independent of fp base (it's "just as true" for decimal floats
> as for binary floats).
>
> The `decimal` module also has two distinct flavors of "mod", neither
> of which match Python's integer-mod definition:
>
> >>> decimal.Decimal(7).__mod__(10)
> Decimal("7")
> >>> decimal.Decimal(7).remainder_near(10)
> Decimal("-3")
> >>> decimal.Decimal(-7).__mod__(10)
> Decimal("-7")
> >>> decimal.Decimal(-7).remainder_near(10)
> Decimal("3")
>
> But, again, I think floating mod is so rare it doesn't need syntactic
> support, and especially not when there's more than one implementation
> -- and `decimal` is a floating type (in Cowlishaw's REXX, this was the
> /only/ numeric type, so there was more pressure there to give it a
> succinct spelling).

So you are proposing that Decimal also rip out the % and // operators
and __divmod__? WFM, but I don't know what Decimal users say (I'm not
one).

> > The floats currently aren't embedded in complex, since f.real and
> > f.imag don't work for floats (and math.sqrt(-1.0) doesn't return 1.0j,
> > and many other anomalies). That was a conscious decision because
> > complex numbers are useless for most folks. But is it still the right
> > decision, given what we're trying to do with int and float?
>
> Sounds like a "purity" thing.  The "pure form" of the original
> /intent/ was probably just that nobody should get a complex result
> from non-complex inputs (they should see an exception instead) unless
> they use a `cmath` function to prove they know what they're doing up
> front.
>
> I still think that has pragmatic value, since things like sqrt(-1) and
> asin(1.05) really are signs of logic errors for most programs.

OK, that's good.

> But under that view, there's nothing surprising about, e.g.,
> (math.pi).imag returning 0.0 -- or (3).imag returning 0.0 either.
> That sounds fine to me (if someone cares enough to implement it).
> Unclear what very_big_int.real should do (it can lose information to
> rounding if the int > 2**53, or even overflow -- maybe it should just
> return the int!).

For ints and floats, real could just return self, and imag could
return a 0 of the same type as self. I guess the conjugate() function
could also just return self (although I see that conjugate() for a
complex with a zero imaginary part returns something whose imaginary
part is -0; is that intentional? I'd rather not have to do that when
the input is an int or float, what do you think?)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Mon Jan 22 19:01:04 2007
From: brett at python.org (Brett Cannon)
Date: Mon, 22 Jan 2007 10:01:04 -0800
Subject: [Python-Dev] Eliminating f_tstate
In-Reply-To: <87ac0b31fo.fsf@starship.python.net>
References: <45B46D04.5060103@v.loewis.de> <87ac0b31fo.fsf@starship.python.net>
Message-ID: <bbaeab100701221001rf84b11bjb8ce5da570133e73@mail.gmail.com>

On 1/22/07, Michael Hudson <mwh at python.net> wrote:
> "Martin v. L?wis" <martin at v.loewis.de> writes:
>
> > Bug #1579370 reports a crash when accessing the thread state of
> > a terminated thread, when releasing a generator object.
> >
> > In analysing the problem, I found that f_tstate doesn't have much
> > utility: it is used in very few places, and in these places, it
> > might be as good or better to use the *current* thread state
> > (rather than the frame's thread state); in some (most?) of these
> > places, the current thread should be identical to the frame's
> > thread, anyway.
> >
> > So I would like to propose that the f_tstate member is removed
> > from the frame object.
> >
> > For Python 2.5, for compatibility, it probably has to remain
> > where it is, and only PyTraceBack_Here should stop using it.
> > As a consequence, a generator .send() makes exceptions
> > occur in the current thread, not in the thread where the
> > generator was created.
> >
> > What do you think?
>
> Without having read the code in detail, I think what you say makes a
> great deal of sense.
>

Ditto from me.

-Brett

From nmm1 at cus.cam.ac.uk  Mon Jan 22 19:14:16 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Mon, 22 Jan 2007 18:14:16 +0000
Subject: [Python-Dev] Floor division
In-Reply-To: Your message of "Sun, 21 Jan 2007 19:40:21 PST."
	<ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com> 
Message-ID: <E1H93gO-0003I8-Qv@draco.cus.cam.ac.uk>

"Guido van Rossum" <guido at python.org> wrote:
>         
> That really sucks, especially since the whole point of making int
> division return a float was to make the integers embedded in the
> floats... I think the best solution would be to remove the definition
> of % (and then also for divmod()) for floats altogether, deferring to
> math.fmod() instead.

Please, NO!!!

At least not without changing the specification of the math module.
The problem with it is that it is specified to be a mapping of the
underlying C library, complete with its <censored> error handling.
fmod isn't bad, as <math.h> goes, BUT:

God alone knows what happens with fmod(x,0.0), let alone fmod(x,-0.0).
C99 says that it is implementation-defined whether a domain error
occurs or the function returns zero, but domain errors are defined
behaviour only in C90 (and not in C99!)  It is properly defined only
if Annex F is in effect (with all the consequences that implies).

Note that I am not saying that syntactic support is needed, because
Fortran gets along perfectly well with this as a function.  All I
am saying is that we want a defined function with decent error
handling!  Returning a NaN is fine on systems with proper NaN support,
which is why C99 Annex F fmod is OK.

> For ints and floats, real could just return self, and imag could
> return a 0 of the same type as self. I guess the conjugate() function
> could also just return self (although I see that conjugate() for a
> complex with a zero imaginary part returns something whose imaginary
> part is -0; is that intentional? I'd rather not have to do that when
> the input is an int or float, what do you think?)

I don't see the problem in doing that - WHEN implicit conversion
to a smaller domain, losing information, raises an exception.  The
errors caused by needing a 'cast' (including Fortran INT, DBLE and
(ugh) COMPLEX, here) causing not just conversion but information
loss have caused major trouble for as long as I have been around.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From martin at v.loewis.de  Mon Jan 22 22:31:21 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 22 Jan 2007 22:31:21 +0100
Subject: [Python-Dev] [Python-checkins] buildbot failure in amd64 gentoo
	2.5
In-Reply-To: <ep33cv$a1o$1@sea.gmane.org>
References: <20070122192355.B20C31E4007@bag.python.org>
	<ep33cv$a1o$1@sea.gmane.org>
Message-ID: <45B52D29.4020001@v.loewis.de>

Georg Brandl schrieb:
> Do I miss something here, or is the buildbot hit by spammers now?

It looks like it is. If that continues, we have to disable the web
triggers.

Regards,
Martin

From facundo at taniquetil.com.ar  Mon Jan 22 22:35:10 2007
From: facundo at taniquetil.com.ar (Facundo Batista)
Date: Mon, 22 Jan 2007 21:35:10 +0000 (UTC)
Subject: [Python-Dev] Floor division
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>
Message-ID: <ep3amd$6t3$1@sea.gmane.org>

Guido van Rossum wrote:

> The ints aren't really embedded in Decimal, so we don't have to do
> that there (but we could).

-0.

If we can't achieve it without disturbing the rest of Python, I'll try
as much as possible to keep what the Spec proposes.

Regards,

-- 
.   Facundo
.
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/



From tdelaney at avaya.com  Mon Jan 22 22:36:03 2007
From: tdelaney at avaya.com (Delaney, Timothy (Tim))
Date: Tue, 23 Jan 2007 08:36:03 +1100
Subject: [Python-Dev] Eliminating f_tstate
Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1EC6A@au3010avexu1.global.avaya.com>

"Martin v. L?wis" wrote:

> For Python 2.5, for compatibility, it probably has to remain
> where it is, and only PyTraceBack_Here should stop using it.
> As a consequence, a generator .send() makes exceptions
> occur in the current thread, not in the thread where the
> generator was created.
> 
> What do you think?

That's the behaviour I would expect. There is nothing I can find in either the Python docs nor PEP 342 to suggest that interactions with generators happen on anything except the current thread. Anything else IMO is a bug, and should be a candidate for 2.5.1.

Tim Delaney

From tim.peters at gmail.com  Tue Jan 23 08:01:35 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 23 Jan 2007 02:01:35 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <ca471dc20701220757gcb24a5dkcf90b39ba22943f6@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>
	<1f7befae0701212121i6ae713b1v98f527113afc0323@mail.gmail.com>
	<ca471dc20701220757gcb24a5dkcf90b39ba22943f6@mail.gmail.com>
Message-ID: <1f7befae0701222301g2d9e0a47y9692b072ddf7823@mail.gmail.com>

[Guido]
> ...
> So you are proposing that Decimal also rip out the % and // operators
> and __divmod__? WFM, but I don't know what Decimal users say (I'm not
> one).

Yes:  it's just as much a floating type as HW binary floats, and all
the same issues come up.  For example, decimal floats are just as
prone to the floor division surprise Raymond started this thread with;
e.g.,

>>> a
Decimal("2.172839486617283948661728393E+29")
>>> b
Decimal("1234567890123456789012345678")
>>> a / b
Decimal("176.0000000000000000000000000")
>>> a/b == 176
True
>>> a // b
Decimal("175")

That is, floor division of `a` by `b` isn't necessarily the same as
the floor of `a` divided by `b` for decimal floats either, and for
exactly the same reason as when using binary floats:  a/b can suffer a
rounding error due to finite precision, but floor division computes
the floor of the quotient "as if" infinite precision were available.
At least using `decimal` it's easy to /explain/ just by boosting the
precision:

>>> decimal.getcontext().prec *= 2
>>> a / b
Decimal("175.99999999999999999999999997731999979587999814250798308")

This shows quite clearly why a/b rounded up to exactly 176 when
working with half this precision.

There's also that the decimal __mod__ implementation is like math.fmod
for binary floats, not like Python's int/long __mod__.  Having just
one builtin meaning for numeric `%` as an infix operator is a good
thing, and the int/long meaning is both by far the most common use but
only "works" for types with exactly representable results (ints and
longs built in; rationals if someone adds them; ditto constructive
reals; ... -- but not floats).

> ...
> For ints and floats, real could just return self, and imag could
> return a 0 of the same type as self.

Cool!  Works for me.

> I guess the conjugate() function could also just return self (although I see
> that conjugate() for a complex with a zero imaginary part returns
> something whose imaginary part is -0; is that intentional?

That's wrong, if true:  it should return something with the opposite
sign on the imaginary part, whether or not that equals 0 (+0. and -0.
both "equal 0").

This is harder to check than it should be because it appears there's a
bug in the complex constructor (at least in Python 2.5):  complex(1.,
0.0) and complex(1., -0.0) both appear to create a complex with a +0
imaginary part:

>>> def is_minus_0(x):
...     import math
...     return x == 0.0 and math.atan2(x, x) != 0
>>> is_minus_0(+0.0)  # just showing that "it works"
False
>>> is_minus_0(-0.0)   # ditto
True
>>> is_minus_0(complex(1, 0.0).imag)
False
>>> is_minus_0(complex(1, -0.0).imag)  # should be True
False

OTOH, the complex constructor does respect the sign of the real part:

>>> is_minus_0(complex(0.0, 0.0).real)
False
>>> is_minus_0(complex(-0.0, 0.0).real)
True

complex_new() ends with:

	cr.real -= ci.imag;
	cr.imag += ci.real;

and I have no idea what that thinks it's doing.  Surely this isn't intended?!:

>>> complex(complex(1.0, 2.0), complex(10.0, 20.0))
(-19+12j)

WTF?  In any case, that's also what's destroying the sign of the
imaginary part in complex(1.0, -0.0).

Knowing that a -0 imaginary part can't be constructed in the obvious way:

>>> is_minus_0(complex(0, 0).conjugate().imag)
True

So conjugate() does flip the sign on a +0 imaginary part, and:

>>> is_minus_0(complex(0, 0).conjugate().conjugate().imag)
False

so it also flips the sign on a -0 imaginary part.  That's all as it should be.

Hmm.  You meant to ask something different, but I actually answered that too ;-)

> I'd rather not have to do that when the input is an int or float, what do you
> think?)

Returning `self` is fine with me for those, although, e.g., it does mean that

    (3).conjugate().imag
and
    (complex(3)).conjugate().imag

are distinguishable with enough pain.  I don't care.  I think of
integers and floats as "not having" an imaginary part more than as
having a 0 imaginary part (but happy to invent a 0 imaginary part if
someone asks for one).

From tim.peters at gmail.com  Tue Jan 23 08:28:20 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 23 Jan 2007 02:28:20 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <E1H93gO-0003I8-Qv@draco.cus.cam.ac.uk>
References: <ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>
	<E1H93gO-0003I8-Qv@draco.cus.cam.ac.uk>
Message-ID: <1f7befae0701222328v6d8b9ff6hf534bff242951084@mail.gmail.com>

[Guido]
>> That really sucks, especially since the whole point of making int
>> division return a float was to make the integers embedded in the
>> floats... I think the best solution would be to remove the definition
>> of % (and then also for divmod()) for floats altogether, deferring to
>> math.fmod() instead.

[Nick Maclaren]
> Please, NO!!!
>
> At least not without changing the specification of the math module.
> The problem with it is that it is specified to be a mapping of the
> underlying C library, complete with its <censored> error handling.
> fmod isn't bad, as <math.h> goes, BUT:
>
> God alone knows what happens with fmod(x,0.0), let alone fmod(x,-0.0).
> C99 says that it is implementation-defined whether a domain error
> occurs or the function returns zero, but domain errors are defined
> behaviour only in C90 (and not in C99!)  It is properly defined only
> if Annex F is in effect (with all the consequences that implies).
>
> Note that I am not saying that syntactic support is needed, because
> Fortran gets along perfectly well with this as a function.  All I
> am saying is that we want a defined function with decent error
> handling!  Returning a NaN is fine on systems with proper NaN support,
> which is why C99 Annex F fmod is OK.

math.fmod is 15 years old -- whether or not someone likes it has
nothing to do with whether Python should stop trying to use the
current integer-derived meaning of % for floats.

On occasion we've added additional error checking around functions
inherited from C.  But adding code to return a NaN has never been
done.  If you want special error checking added to the math.fmod
wrapper, it would be easiest to "sell" by far to request that it raise
ZeroDivisionError (as integer mod does) for a modulus of 0, or
ValueError (Python's historic mapping of libm EDOM, and what Python's
fmod(1, 0) already does on some platforms).  The `decimal` module
raises InvalidOperation in this case, but that exception is specific
to the `decimal` module for now.

>> For ints and floats, real could just return self, and imag could
>> return a 0 of the same type as self. I guess the conjugate() function
>> could also just return self (although I see that conjugate() for a
>> complex with a zero imaginary part returns something whose imaginary
>> part is -0; is that intentional? I'd rather not have to do that when
>> the input is an int or float, what do you think?)

> I don't see the problem in doing that - WHEN implicit conversion
> to a smaller domain, losing information, raises an exception.

Take it as a pragmatic fact that it wouldn't.  Besides, e.g., the
conjugate of 10**50000 is exactly 10**50000 mathematically.  Why raise
an exception just because it can't be represented as a float?  The
exact result is easily supplied with a few lines of "obviously
correct" implementation code (incref `self` and return it).

From tim.peters at gmail.com  Tue Jan 23 08:43:04 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 23 Jan 2007 02:43:04 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <ep3amd$6t3$1@sea.gmane.org>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>
	<ep3amd$6t3$1@sea.gmane.org>
Message-ID: <1f7befae0701222343o36aff902w140220bbc3eb9bec@mail.gmail.com>

[Guido]
>> The ints aren't really embedded in Decimal, so we don't have to do
>> that there (but we could).

[Facundo Batista]
> -0.
>
> If we can't achieve it without disturbing the rest of Python, I'll try
> as much as possible to keep what the Spec proposes.

Which "Spec"?  For example, floor division isn't mentioned at all in
IBM's proposed decimal standard, or in PEP 327, or in the Python
Library Reference section on `decimal`.  It's an artifact of trying to
extend Python's integer mod definition to floats, and for reasons
explained in this thread (for the 27th time ;-)), that definition
doesn't make good sense for floats.  The IBM spec defines `remainder`
and `remainder-near` for floats, and those do make good sense for
floats.  But they're /different/ definitions than Python uses for
integer mod.

Do note that this discussion is only about Python 3.  Under the view
that it was a (minor, but real) design error to /try/ extending
Python's integer mod definition to floats, if __mod__, and __divmod__
and __floordiv__ go away for binary floats in P3K they should
certainly go away for decimal floats in P3K too.  And that's about
syntax, not functionality:  the IBM spec's "remainder" and
"remainder-near" still need to be there, it's "just" that a user would
have to get at "remainder" in a more long-winded way (analogous to
that a P3K user would have to spell out "math.fmod" to get at a mod
function for binary floats).

From warner at lothar.com  Tue Jan 23 10:20:18 2007
From: warner at lothar.com (Brian Warner)
Date: Tue, 23 Jan 2007 01:20:18 -0800
Subject: [Python-Dev] [Python-checkins] buildbot failure in amd64 gentoo
	2.5
In-Reply-To: <45B52D29.4020001@v.loewis.de> (Martin v. =?iso-8859-1?Q?L=F6?=
	=?iso-8859-1?Q?wis's?= message of
	"Mon, 22 Jan 2007 22:31:21 +0100")
References: <20070122192355.B20C31E4007@bag.python.org>
	<ep33cv$a1o$1@sea.gmane.org> <45B52D29.4020001@v.loewis.de>
Message-ID: <874pqip0a5.fsf@lothar.com>

>> Do I miss something here, or is the buildbot hit by spammers now?
>
> It looks like it is. If that continues, we have to disable the web
> triggers.

Good grief. If anyone has any bright ideas about simple ways to change that
form to make it less vulnerable to the spambots, I'd be happy to incorporate
them into Buildbot.

sigh,
 -Brian

From ajm at flonidan.dk  Tue Jan 23 10:19:25 2007
From: ajm at flonidan.dk (Anders J. Munch)
Date: Tue, 23 Jan 2007 10:19:25 +0100
Subject: [Python-Dev] Floor division
Message-ID: <9B1795C95533CA46A83BA1EAD4B01030031F75@flonidanmail.flonidan.net>

Tim Peters wrote:
> Do note that this discussion is only about Python 3.  Under the view
> that it was a (minor, but real) design error to /try/ extending
> Python's integer mod definition to floats, if __mod__, and __divmod__
> and __floordiv__ go away for binary floats in P3K they should
> certainly go away for decimal floats in P3K too.  

What design error?  float.__mod__ works perfectly.

>>> -1 % 50
49
>>> -1.0 % 50.0
49.0
>>> 

It's only Decimal.__mod__ that's inconsistent.  float.__mod__ has the
usual floating-point inaccuracies, but then with float that goes with
the territory.

I've had occasion to use it, and it was a pleasant surprise that it
"just worked", so I didn't have to go to the math module and ponder
over the difference between modulo or remainder.  float modulo is
useful, consistent and intuitive; -1 on removing that.

- Anders

From rasky at develer.com  Tue Jan 23 10:44:57 2007
From: rasky at develer.com (Giovanni Bajo)
Date: Tue, 23 Jan 2007 10:44:57 +0100
Subject: [Python-Dev] [Python-checkins] buildbot failure in amd64 gentoo
	2.5
In-Reply-To: <874pqip0a5.fsf@lothar.com>
References: <20070122192355.B20C31E4007@bag.python.org>	<ep33cv$a1o$1@sea.gmane.org>
	<45B52D29.4020001@v.loewis.de> <874pqip0a5.fsf@lothar.com>
Message-ID: <ep4leo$1k9$1@sea.gmane.org>

On 23/01/2007 10.20, Brian Warner wrote:

>>> Do I miss something here, or is the buildbot hit by spammers now?
>> It looks like it is. If that continues, we have to disable the web
>> triggers.
> 
> Good grief. If anyone has any bright ideas about simple ways to change that
> form to make it less vulnerable to the spambots, I'd be happy to incorporate
> them into Buildbot.

I'd throw a CAPTCHA in. There are even some written in Python.
-- 
Giovanni Bajo


From nmm1 at cus.cam.ac.uk  Tue Jan 23 10:58:06 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Tue, 23 Jan 2007 09:58:06 +0000
Subject: [Python-Dev] Floor division
Message-ID: <E1H9IPn-0003Uj-0y@libra.cus.cam.ac.uk>

"Tim Peters" <tim.peters at gmail.com> wrote:
>
> > I guess the conjugate() function could also just return self (although I see
> > that conjugate() for a complex with a zero imaginary part returns
> > something whose imaginary part is -0; is that intentional?
> 
> That's wrong, if true:  it should return something with the opposite
> sign on the imaginary part, whether or not that equals 0 (+0. and -0.
> both "equal 0").

Grrk.  Why?  Seriously.  IEEE 754 signed zeroes are deceptive enough
for float, but are a gibbering nightmare for complex; Kahan may be
able to handle them, but mere mortals can't.  Inter alia, the only
sane forms of infinity for complex numbers are a SINGLE one (the
compactified model) and to may infinity into NaN (which I prefer,
as it leads to less nonsense).

And, returning to 'floor' - if one is truncating towards -infinity,
should floor(-0.0) deliver -1.0, 0.0 or -0.0?

> math.fmod is 15 years old -- whether or not someone likes it has
> nothing to do with whether Python should stop trying to use the
> current integer-derived meaning of % for floats.

Eh?  No, it isn't.  Because of the indirection to the C library, it
is changing specification as we speak!  THAT is all I am getting at;
not that the answer might not be A math.fmod with defined behaviour.

> On occasion we've added additional error checking around functions
> inherited from C.  But adding code to return a NaN has never been
> done.  If you want special error checking added to the math.fmod
> wrapper, it would be easiest to "sell" by far to request that it raise
> ZeroDivisionError (as integer mod does) for a modulus of 0, or
> ValueError (Python's historic mapping of libm EDOM, and what Python's
> fmod(1, 0) already does on some platforms).  The `decimal` module
> raises InvalidOperation in this case, but that exception is specific
> to the `decimal` module for now.

I never said that it should; I said that it is reasonable behaviour
on systems that support them.  I personally much prefer an exception
in this case.  What I was trying to point out is that the current
behaviour is UNDEFINED (and may give total nonsense).  That is not
good.

> >> For ints and floats, real could just return self, and imag could
> >> return a 0 of the same type as self. I guess the conjugate() function
> >> could also just return self (although I see that conjugate() for a
> >> complex with a zero imaginary part returns something whose imaginary
> >> part is -0; is that intentional? I'd rather not have to do that when
> >> the input is an int or float, what do you think?)
> 
> > I don't see the problem in doing that - WHEN implicit conversion
> > to a smaller domain, losing information, raises an exception.
> 
> Take it as a pragmatic fact that it wouldn't.  Besides, e.g., the
> conjugate of 10**50000 is exactly 10**50000 mathematically.  Why raise
> an exception just because it can't be represented as a float?  The
> exact result is easily supplied with a few lines of "obviously
> correct" implementation code (incref `self` and return it).

Eh?  I don't understand.  Are you referring to float("1.0e50000"),
pow(10,50000), pow(10.0,50000), or a conjugate (and, if so, of what?)

float(conjg(1.23)) obviously need not raise an exception, except
possibly "Sanity failure" :-)  


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From tim.peters at gmail.com  Tue Jan 23 10:52:51 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 23 Jan 2007 04:52:51 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <9B1795C95533CA46A83BA1EAD4B01030031F75@flonidanmail.flonidan.net>
References: <9B1795C95533CA46A83BA1EAD4B01030031F75@flonidanmail.flonidan.net>
Message-ID: <1f7befae0701230152xe7661d5u78201bc5ccf157fb@mail.gmail.com>

[Anders J. Munch]
> What design error?  float.__mod__ works perfectly.
>
> >>> -1 % 50
> 49
> >>> -1.0 % 50.0
> 49.0
> >>>

Please read the whole thread.  Maybe you did, but you said nothing
here that indicated you had.  The issues aren't about tiny integers
that happen to be in float format, where the result is exactly
representable as a float too.  Those don't create problems for any
definition of mod.  But even non-tiny exact integers can.

> It's only Decimal.__mod__ that's inconsistent.  float.__mod__ has the
> usual floating-point inaccuracies, but then with float that goes with
> the territory.

No.  Decimal.__mod_  always returns the mathematically exact result.
It has this in common with math.fmod() (in fact, math.fmod() and
Decimal.__mod__() have the same definition, ignoring IEEE endcases).
It's impossible to do so under Python's integer-derived mod
definition.  Read the whole thread for why -- we can't even guarantee
that abs(a%b) < abs(b) for non-zero finite floats under the current
mod, and that has in fact been a source of complaints over the years.
math.fmod and Decimal.__mod__ do guarantee abs(a%b) < abs(b) for all
non-zero finite floats, and moreover guarantee that a%b is an exact
integer multiple of `b` away from `a` (although that integer may not
be representable as a float) -- again it's impossible for the a -
floor(a/b)*b definition to do so for floats.

> I've had occasion to use it, and it was a pleasant surprise that it
> "just worked", so I didn't have to go to the math module and ponder
> over the difference between modulo or remainder.

There's no such distinction in Python's math module -- fmod is the
only modular reduction function in the math module.  The `decimal`
module has two such functions (see earlier messages in this thread for
examples -- neither matches Python's mod function).

From mwh at python.net  Tue Jan 23 11:06:25 2007
From: mwh at python.net (Michael Hudson)
Date: Tue, 23 Jan 2007 11:06:25 +0100
Subject: [Python-Dev] [Python-checkins] buildbot failure in amd64 gentoo
	2.5
In-Reply-To: <ep4leo$1k9$1@sea.gmane.org> (Giovanni Bajo's message of "Tue,
	23 Jan 2007 10:44:57 +0100")
References: <20070122192355.B20C31E4007@bag.python.org>
	<ep33cv$a1o$1@sea.gmane.org> <45B52D29.4020001@v.loewis.de>
	<874pqip0a5.fsf@lothar.com> <ep4leo$1k9$1@sea.gmane.org>
Message-ID: <87lkju12hq.fsf@starship.python.net>

Giovanni Bajo <rasky at develer.com> writes:

> On 23/01/2007 10.20, Brian Warner wrote:
>
>>>> Do I miss something here, or is the buildbot hit by spammers now?
>>> It looks like it is. If that continues, we have to disable the web
>>> triggers.
>> 
>> Good grief. If anyone has any bright ideas about simple ways to change that
>> form to make it less vulnerable to the spambots, I'd be happy to incorporate
>> them into Buildbot.
>
> I'd throw a CAPTCHA in. There are even some written in Python.

I'd guess even the simplest thing would work:

"Type "Python" into this box: ______"

Cheers,
mwh

-- 
  ZAPHOD:  Who are you?
  ROOSTA:  A friend.
  ZAPHOD:  Oh yeah? Anyone's friend in particular, or just generally 
           well-disposed to people?               -- HHGttG, Episode 7

From skip at pobox.com  Tue Jan 23 11:55:12 2007
From: skip at pobox.com (skip at pobox.com)
Date: Tue, 23 Jan 2007 04:55:12 -0600
Subject: [Python-Dev] Floor division
In-Reply-To: <1f7befae0701222343o36aff902w140220bbc3eb9bec@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>
	<ep3amd$6t3$1@sea.gmane.org>
	<1f7befae0701222343o36aff902w140220bbc3eb9bec@mail.gmail.com>
Message-ID: <17845.59792.579425.543071@montanaro.dyndns.org>


    Tim> For example, floor division isn't mentioned at all in IBM's
    Tim> proposed decimal standard, or in PEP 327, or in the Python Library
    Tim> Reference section on `decimal`.  It's an artifact of trying to
    Tim> extend Python's integer mod definition to floats, and for reasons
    Tim> explained in this thread (for the 27th time ;-)), that definition
    Tim> doesn't make good sense for floats....

    Tim> Do note that this discussion is only about Python 3....

Since Python 3 is about cleaning things up (at least in part) and Tim is
telling us this is going to clean things up in this area, I'm +1 on this.
Besides, I've found it's generally best to agree with Tim. ;-)

Skip

From skip at pobox.com  Tue Jan 23 11:57:12 2007
From: skip at pobox.com (skip at pobox.com)
Date: Tue, 23 Jan 2007 04:57:12 -0600
Subject: [Python-Dev] [Python-checkins] buildbot failure in amd64 gentoo
 2.5
In-Reply-To: <874pqip0a5.fsf@lothar.com>
References: <20070122192355.B20C31E4007@bag.python.org>
	<ep33cv$a1o$1@sea.gmane.org> <45B52D29.4020001@v.loewis.de>
	<874pqip0a5.fsf@lothar.com>
Message-ID: <17845.59912.10367.788801@montanaro.dyndns.org>


    Brian> Good grief. If anyone has any bright ideas about simple ways to
    Brian> change that form to make it less vulnerable to the spambots, I'd
    Brian> be happy to incorporate them into Buildbot.

Require a password?  It can be widely known throughout each buildbot
community with little danger of being guessed by a spambot.

Skip

From tim.peters at gmail.com  Tue Jan 23 11:58:52 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 23 Jan 2007 05:58:52 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <E1H9IPn-0003Uj-0y@libra.cus.cam.ac.uk>
References: <E1H9IPn-0003Uj-0y@libra.cus.cam.ac.uk>
Message-ID: <1f7befae0701230258g19bbeac6if0b3096e16628051@mail.gmail.com>

[Guido]
>>> I guess the conjugate() function could also just return self (although I see
>>> that conjugate() for a complex with a zero imaginary part returns
>>> something whose imaginary part is -0; is that intentional?

[TIm Peters]
>> That's wrong, if true:  it should return something with the opposite
>> sign on the imaginary part, whether or not that equals 0 (+0. and -0.
>> both "equal 0").

|[Nick Maclaren]
> Grrk.  Why?  Seriously.

Seriously:  because there's some reason to do so and no good reason
not to.  This is the current complex conjugate implementation:

static PyObject *
complex_conjugate(PyObject *self)
{
	Py_complex c;
	c = ((PyComplexObject *)self)->cval;
	c.imag = -c.imag;
	return PyComplex_FromCComplex(c);
}

Complicating that to make a special case of c.imag == 0 is simply
goofy without a /compelling/ reason to do so.  As I read the C99
section on "conj", it's also their intent that the sign on the
imaginary part be flipped regardless of value.

> IEEE 754 signed zeroes are deceptive enough for float, but are
> a gibbering nightmare for complex; Kahan may be
> able to handle them, but mere mortals can't.  Inter alia, the only
> sane forms of infinity for complex numbers are a SINGLE one (the
> compactified model) and to may infinity into NaN (which I prefer,
> as it leads to less nonsense).
>
> And, returning to 'floor' - if one is truncating towards -infinity,
> should floor(-0.0) deliver -1.0, 0.0 or -0.0?

I'd leave a zero argument alone (for ceiling too), and am quite sure
that's "the right" 754-ish behavior.

>> math.fmod is 15 years old -- whether or not someone likes it has
>> nothing to do with whether Python should stop trying to use the
>> current integer-derived meaning of % for floats.

> Eh?  No, it isn't.  Because of the indirection to the C library, it
> is changing specification as we speak!  THAT is all I am getting at;
> not that the answer might not be A math.fmod with defined behaviour.

Couldn't quite parse that, but nearly all of Python's math-module
functions inherit most behavior from the platform libm.  This is often
considered to be a feature:  the functions called from Python
generally act much like they do when called from C or Fortran on the
same platform, easing cross-language development on a single platform.

It's never been Python's intent to define all behavior here, and
largely because Python isn't a math-library development project.  To
the extent that enough people care enough to standardize C's libm
endcase behavior across platforms, Python inherits that too.  Not much
of an inheritance so far ;-)

Do note the flip side:  to the extent that different platform
religions refuse to standardize libm endcase behavior, Python plays
along with whatever libm gods the platform it's running on worships.
That's of value to some too.

>> On occasion we've added additional error checking around functions
>> inherited from C.  But adding code to return a NaN has never been
>> done.  If you want special error checking added to the math.fmod
>> wrapper, it would be easiest to "sell" by far to request that it raise
>> ZeroDivisionError (as integer mod does) for a modulus of 0, or
>> ValueError (Python's historic mapping of libm EDOM, and what Python's
>> fmod(1, 0) already does on some platforms).  The `decimal` module
>> raises InvalidOperation in this case, but that exception is specific
>> to the `decimal` module for now.

> I never said that it should; I said that it is reasonable behaviour on systems
> that support them.  I personally much prefer an exception in this case.

So which one would you prefer?  As explained, there are 3 plausible candidates.

You seem to be having some trouble taking "yes" for an answer here ;-)

> What I was trying to point out is that the current behaviour is
> UNDEFINED (and may give total nonsense).  That is not
> good.

Eh -- I can't get excited about it.  AFAIK, in 15 years nobody has
complained about passing a 0 modulus to math.fmod (possibly because
most newbies use the Windows distro, and it does raise ValueError
there).

>>>> For ints and floats, real could just return self, and imag could
>>>> return a 0 of the same type as self. I guess the conjugate() function
>>>> could also just return self (although I see that conjugate() for a
>>>> complex with a zero imaginary part returns something whose imaginary
>>>> part is -0; is that intentional? I'd rather not have to do that when
>>>> the input is an int or float, what do you think?)

>>> I don't see the problem in doing that - WHEN implicit conversion
>>> to a smaller domain, losing information, raises an exception.

>> Take it as a pragmatic fact that it wouldn't.  Besides, e.g., the
>> conjugate of 10**50000 is exactly 10**50000 mathematically.  Why raise
>> an exception just because it can't be represented as a float?  The
>> exact result is easily supplied with a few lines of "obviously
>> correct" implementation code (incref `self` and return it).

> Eh?  I don't understand.  Are you referring to float("1.0e50000"),
> pow(10,50000), pow(10.0,50000), or a conjugate (and, if so, of what?)

"The conjugate of 10**50000", which is an example of something Guido
was asking about ("I'd rather not have to do that when the input is an
int ...").  In a Python extended in the way he's imagining, one
concrete way to spell it would be:

    (10**50000).conjugate()

Complex numbers already have a conjugate() method:

>>> 1j.conjugate()
-1j

Guido is asking about adding that method to floats/ints/longs too.  If
he did, what should it do?  The only way I could read your original
response is that you wanted, e.g., (1.23).conjugate() to return a
complex number with a -0.0 imaginary part, likewise for
(123).conjugate(), but that you wanted some_large_int.conjugate() to
raise an exception since the implicit coercion to float would lose
information.

What Guido would rather do, which I agreed with, was to have
x.conjugate() simply return x when x is float/int/long.  No change in
value, no change in type, and the obvious implementation would even
make

    x.conjugate() is x

true for x an int/long/float.

> float(conjg(1.23)) obviously need not raise an exception, except
> possibly "Sanity failure" :-)

Well, Python /does/ raise an exception (TypeError) if you try to apply
float() to a complex argument.  That's solely type-based -- the value
of the imaginary part is irrelevant.

    >>> float(complex(0))
    Traceback (most recent call last):
       ...
    TypeError: can't convert complex to float; use abs(z)

So if (1.23).conjugate() returned a complex number, applying float()
to that would raise an exception.  If it returned 1.23, then of course
applying float() to that would not be exceptional.

From ajm at flonidan.dk  Tue Jan 23 12:00:59 2007
From: ajm at flonidan.dk (Anders J. Munch)
Date: Tue, 23 Jan 2007 12:00:59 +0100
Subject: [Python-Dev] Floor division
Message-ID: <9B1795C95533CA46A83BA1EAD4B01030031F76@flonidanmail.flonidan.net>

Tim Peters wrote:
> Please read the whole thread.  Maybe you did, but you said nothing
> here that indicated you had.  The issues aren't about tiny integers
> that happen to be in float format, where the result is exactly
> representable as a float too.  Those don't create problems for any
> definition of mod.  But even non-tiny exact integers can.

I did read the whole thread, and I saw your -1%1e100 example.  Mixing
floating-point numbers of very different magnitude can get you in
trouble - e.g. -1+1e100==1e100.  I don't think -1%1e100 is all that
worse.

> > It's only Decimal.__mod__ that's inconsistent.  float.__mod__ has the
> > usual floating-point inaccuracies, but then with float that goes with
> > the territory.
> 
> No.  Decimal.__mod_  always returns the mathematically exact result.

I meant inconsistent with integers.  People are familiar with the
semantics of % on integers, because they use it all the time.  % on
float is a natural extension of that and hence unsurprising.  % on
Decimal is exact and correct, but surprising all the same.

regards, Anders

From tim.peters at gmail.com  Tue Jan 23 12:27:29 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 23 Jan 2007 06:27:29 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <9B1795C95533CA46A83BA1EAD4B01030031F76@flonidanmail.flonidan.net>
References: <9B1795C95533CA46A83BA1EAD4B01030031F76@flonidanmail.flonidan.net>
Message-ID: <1f7befae0701230327x7cabd6fcwcebf2daa8a5fd733@mail.gmail.com>

[Tim Peters]
>> Please read the whole thread.  Maybe you did, but you said nothing
>> here that indicated you had.  The issues aren't about tiny integers
>> that happen to be in float format, where the result is exactly
>> representable as a float too.  Those don't create problems for any
>> definition of mod.  But even non-tiny exact integers can.

[Anders J. Munch]
> I did read the whole thread, and I saw your -1%1e100 example.  Mixing
> floating-point numbers of very different magnitude can get you in
> trouble - e.g. -1+1e100==1e100.  I don't think -1%1e100 is all that
> worse.

Except that it's very easy to return an exactly correct result in that
case:  -1.  This isn't like floating addition, where rounding errors
/must/ occur at times.  It's never necessary to suffer rounding errors
for a mod function defined with floats in mind.  Indeed, that's why
the C standards define fmod the way they do, and why IBM's proposed
standard for decimal floating arithmetic defines it the same way.

Python's definition of mod makes great sense for integers, but doesn't
generalize nicely.

>>> It's only Decimal.__mod__ that's inconsistent.  float.__mod__ has the
>>> usual floating-point inaccuracies, but then with float that goes with
>>> the territory.

>> No.  Decimal.__mod_  always returns the mathematically exact result.

> I meant inconsistent with integers.

While I was responding to your second sentence there, not your first.
You can tell because I didn't say anything after your first sentence
;-)

No, it's not always true that "with float [inaccuracies] goes with the
territory".  mod "should be" like absolute value and unary minus this
way:  always exact.

> People are familiar with the semantics of % on integers, because they use
> it all the time.

I'm not sure how many people are actually familiar with the semantics
of integer % when mixing signs, in part because there's no consistency
across languages in that area so people with a lot of experience tend
to avoid it.  I agree integer % is heavily used regardless.

> % on float is a natural extension of that and hence unsurprising.

It was natural to /want/ to extend it to floats.  That didn't work
well, and to the contrary it's surprising precisely /because/ the
behavior people enjoy with integers can fail when it's applied to
floats.  Having cases where abs(a%b) >= abs(b) is a
crash-the-space-shuttle level of surprise, especially since I know of
no other language in which that's possible.  It's not possible with
ints or longs in Python either -- or with math.fmod applied to floats.

> % on Decimal is exact and correct, but surprising all the same.

Which is why I don't want binary or decimal floats to support infix
"%" as a spelling in P3K.  I don't believe floating mod is heavily
used, and if so there's scant need for a one-character spelling -- and
if there's a method or function name to look up in the docs, a user
can read about what they're getting.

In fact, I expect the decimal module's "remainder-near" is what most
people using mod on floats actually want most of the time:  they
couldn't care less about the sign of the result, but they do want its
absolute value to as small as possible.  This is because floating
mod's one "natural" mixed-sign use is for argument reduction before
feeding the remainder into a series expansion.

From ncoghlan at gmail.com  Tue Jan 23 13:26:30 2007
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Jan 2007 22:26:30 +1000
Subject: [Python-Dev] [Python-checkins] buildbot failure in amd64 gentoo
 2.5
In-Reply-To: <87lkju12hq.fsf@starship.python.net>
References: <20070122192355.B20C31E4007@bag.python.org>	<ep33cv$a1o$1@sea.gmane.org>
	<45B52D29.4020001@v.loewis.de>	<874pqip0a5.fsf@lothar.com>
	<ep4leo$1k9$1@sea.gmane.org> <87lkju12hq.fsf@starship.python.net>
Message-ID: <45B5FEF6.1030503@gmail.com>

Michael Hudson wrote:
> Giovanni Bajo <rasky at develer.com> writes:
> 
>> On 23/01/2007 10.20, Brian Warner wrote:
>>
>>>>> Do I miss something here, or is the buildbot hit by spammers now?
>>>> It looks like it is. If that continues, we have to disable the web
>>>> triggers.
>>> Good grief. If anyone has any bright ideas about simple ways to change that
>>> form to make it less vulnerable to the spambots, I'd be happy to incorporate
>>> them into Buildbot.
>> I'd throw a CAPTCHA in. There are even some written in Python.
> 
> I'd guess even the simplest thing would work:
> 
> "Type "Python" into this box: ______"

Allowing a project to define a question & answer should do the trick - 
essentially a password like Skip suggested, but with a reminder right 
there on the page:

"Type this project's name in lowercase in this box: _____"
"Type the BDFL's first name in lowercase in this box: ____"
"Type the branch name in this box: ____"

etc.

It's easier than a normal CAPTCHA because its OK to assume a lot more 
knowledge on the part of the intended audience.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From nmm1 at cus.cam.ac.uk  Tue Jan 23 13:26:57 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Tue, 23 Jan 2007 12:26:57 +0000
Subject: [Python-Dev] Floor division
Message-ID: <E1H9Kjp-0003Ln-SK@draco.cus.cam.ac.uk>

A generic comment.  Many of my postings seem to be being misunderstood. 
I hold no brief for ANY particular floating-point religion, sect or
heresy, except insofar as it affects robustness and portability (i.e.
"software engineering").  I can work with and teach almost any model,
and have done so with some pretty weird ones.

My objections to some proposals is that they are sacrificing those
in favour of some ill-defined objectives.


"Tim Peters" <tim.peters at gmail.com> wrote:
> [TIm Peters]
> >> That's wrong, if true:  it should return something with the opposite
> >> sign on the imaginary part, whether or not that equals 0 (+0. and -0.
> >> both "equal 0").
> 
> |[Nick Maclaren]
> > Grrk.  Why?  Seriously.
> 
> Seriously:  because there's some reason to do so and no good reason
> not to.

Hmm.  That doesn't fully support the practice, except for IEEE 754(R)
numbers.  To require a floating-point format to have signed zeroes
is a religious matter.  But I agree that specifying something different
if the numbers are an IEEE 754(R) format makes no sense.
 
> > And, returning to 'floor' - if one is truncating towards -infinity,
> > should floor(-0.0) deliver -1.0, 0.0 or -0.0?
> 
> I'd leave a zero argument alone (for ceiling too), and am quite sure
> that's "the right" 754-ish behavior.

It's not clear, and there was a debate about it!  But it is what
IEEE 754R ended up specifying.

> Couldn't quite parse that, but nearly all of Python's math-module
> functions inherit most behavior from the platform libm.  This is often
> considered to be a feature:  the functions called from Python
> generally act much like they do when called from C or Fortran on the
> same platform, easing cross-language development on a single platform.

And making it impossible to write robust, portable code :-(  Note that
most platforms have several libms and the behaviour even for a single
libm can be wildly variable.  It can also ENHANCE cross-language
problems, where a user needs to use a library that expects a different
libm or libm option.

> Do note the flip side:  to the extent that different platform
> religions refuse to standardize libm endcase behavior, Python plays
> along with whatever libm gods the platform it's running on worships.
> That's of value to some too.

Actually, no, it doesn't.  Because Python doesn't support any libm
behaviour other than the one that it was compiled with, and that is
often NOT what is wanted.

> So which one would you prefer?  As explained, there are 3 plausible
> candidates.
> 
> You seem to be having some trouble taking "yes" for an answer here ;-)

Actually, there are a lot more candidates, but let that pass.  All
I am saying is that there should be SOME defined AND SANE behaviour.
While I would prefer an exception, I am not dogmatic about it.  What
I can't stand is completely undefined behaviour, as was introduced
into Python by C99.

> > What I was trying to point out is that the current behaviour is
> > UNDEFINED (and may give total nonsense).  That is not
> > good.
> 
> Eh -- I can't get excited about it.  AFAIK, in 15 years nobody has
> complained about passing a 0 modulus to math.fmod (possibly because
> most newbies use the Windows distro, and it does raise ValueError
> there).      

Some people write Python that is intended to be robust and portable;
it is those people who suffer.

> What Guido would rather do, which I agreed with, was to have
> x.conjugate() simply return x when x is float/int/long.  No change in
> value, no change in type, and the obvious implementation would even
> make ...

Fine.  I am happy with that.  What I was pointing out that forcible
changes of type aren't harmful if you remove the "gotchas" of loss
of information with coercions that aren't intended to do so.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From anthony at interlink.com.au  Tue Jan 23 14:20:35 2007
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed, 24 Jan 2007 00:20:35 +1100
Subject: [Python-Dev] Floor division
In-Reply-To: <1f7befae0701230327x7cabd6fcwcebf2daa8a5fd733@mail.gmail.com>
References: <9B1795C95533CA46A83BA1EAD4B01030031F76@flonidanmail.flonidan.net>
	<1f7befae0701230327x7cabd6fcwcebf2daa8a5fd733@mail.gmail.com>
Message-ID: <200701240020.35812.anthony@interlink.com.au>

On Tuesday 23 January 2007 22:27, Tim Peters wrote:
> Which is why I don't want binary or decimal floats to support
> infix "%" as a spelling in P3K.  I don't believe floating mod is
> heavily used, and if so there's scant need for a one-character
> spelling -- and if there's a method or function name to look up
> in the docs, a user can read about what they're getting.

While I agree with this, my only slight concern is that under 2.x
(int/int)%(int) will work, while it will fail under 3.x, because 
int/int will return a float. Eh - we can always make 2.6 warn about 
the floatobject's __mod__ function being called if the -W py3k 
option is on, that gets us part of the way there. And if we have 
a "-3" option or the like that also turns on maximum 3.x compat, 
that will enable true division, producing the warning.

Like I said, it's only a slight concern...


-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From facundo at taniquetil.com.ar  Tue Jan 23 14:39:02 2007
From: facundo at taniquetil.com.ar (Facundo Batista)
Date: Tue, 23 Jan 2007 13:39:02 +0000 (UTC)
Subject: [Python-Dev] Floor division
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<ca471dc20701211940p7d2718e5i53cc4aee11e8f6a1@mail.gmail.com>
	<ep3amd$6t3$1@sea.gmane.org>
	<1f7befae0701222343o36aff902w140220bbc3eb9bec@mail.gmail.com>
Message-ID: <ep535l$nuo$1@sea.gmane.org>

Tim Peters wrote:

> Which "Spec"?  For example, floor division isn't mentioned at all in
> IBM's proposed decimal standard, or in PEP 327, or in the Python

Oops, you're right. My fault, sorry.


> Library Reference section on `decimal`.  It's an artifact of trying to
> extend Python's integer mod definition to floats, and for reasons
> explained in this thread (for the 27th time ;-)), that definition
> doesn't make good sense for floats.  The IBM spec defines `remainder`
> and `remainder-near` for floats, and those do make good sense for
> floats.  But they're /different/ definitions than Python uses for
> integer mod.

I knew the definition of IBM, but assumed (inmersed in my ignorance)
that "remainder" was the way that standard float worked for __mod__. Now
I know that it's not like that, :)


> Do note that this discussion is only about Python 3.  Under the view
> that it was a (minor, but real) design error to /try/ extending
> Python's integer mod definition to floats, if __mod__, and __divmod__
> and __floordiv__ go away for binary floats in P3K they should
> certainly go away for decimal floats in P3K too.  And that's about
> syntax, not functionality:  the IBM spec's "remainder" and
> "remainder-near" still need to be there, it's "just" that a user would
> have to get at "remainder" in a more long-winded way (analogous to

We'll have to deprecate that functionality, with proper warnings (take
not I'm not following the thread that discuss the migration path to 3k).

And we'll have to add the method "remainder" to decimal objects (right
now we have only "remainder_near" in decimal objects, and both
"remainder" and "remainder_near" in context).

Regards,

-- 
.   Facundo
.
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/



From python at rcn.com  Tue Jan 23 14:59:57 2007
From: python at rcn.com (python at rcn.com)
Date: Tue, 23 Jan 2007 08:59:57 -0500 (EST)
Subject: [Python-Dev] Fwd: Re:  Floor division
Message-ID: <20070123085957.AVJ56885@ms09.lnh.mail.rcn.net>



>> Do note that this discussion is only about Python 3.  Under the view
>> that it was a (minor, but real) design error to /try/ extending
>> Python's integer mod definition to floats, if __mod__, and __divmod__
>> and __floordiv__ go away for binary floats in P3K they should
>> certainly go away for decimal floats in P3K too.  And that's about
>> syntax, not functionality:  the IBM spec's "remainder" and
>> "remainder-near" still need to be there, it's "just" that a user would
>> have to get at "remainder" in a more long-winded way (analogous to

[Facundo Batista]
>We'll have to deprecate that functionality, with proper warnings (take
>not I'm not following the thread that discuss the migration path to 3k).
>
>And we'll have to add the method "remainder" to decimal objects (right
>now we have only "remainder_near" in decimal objects, and both
>"remainder" and "remainder_near" in context).

Whoa. This needs more discussion with respect to the decimal module.  I'm not ready to throw-out operator access to the methods provided by the spec.  Also, I do not want the Py2.x decimal module releases to be complexified and slowed-down by Py3.0 deprecation warnings.  The module is already slow and has ease-of-use issues.


Raymond

From tim.peters at gmail.com  Tue Jan 23 15:53:05 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 23 Jan 2007 09:53:05 -0500
Subject: [Python-Dev] Fwd: Re: Floor division
In-Reply-To: <20070123085957.AVJ56885@ms09.lnh.mail.rcn.net>
References: <20070123085957.AVJ56885@ms09.lnh.mail.rcn.net>
Message-ID: <1f7befae0701230653q67007ed9x1e773312575d71d2@mail.gmail.com>

...

[Facundo]
>> We'll have to deprecate that functionality, with proper warnings (take
>> not I'm not following the thread that discuss the migration path to 3k).
>>
>> And we'll have to add the method "remainder" to decimal objects (right
>> now we have only "remainder_near" in decimal objects, and both
>> "remainder" and "remainder_near" in context).

[Raymond]
> Whoa. This needs more discussion with respect to the decimal module.  I'm
> not ready to throw-out operator access to the methods provided by the spec.

decimal's __floordiv__ and __divmod__ have nothing to do with the
spec.  They were Python-specific extensions apparently added to try to
fit in better with Python's entirely different (than either of
decimal's) definition of modular reduction.  In that respect they
succeeded by being jarringly perverse:  decimal's __floordiv__(a, b)
does not return the floor of a/b, it returns a/b truncated toward 0:

>>> -1 // 10
-1
>>> Decimal(-1) // Decimal(10)
Decimal("-0")

Even you might agree that's an odd thing for a function named "floor
div" to do ;-)

I say that's not useful.  The only method here that does have
something to do with the spec is decimal's __mod__, an implementation
of the spec's "remainder" operation.  Guido agreed last May to take
away the "%" infix spelling of mod for binary floats, and I can't
think of a reason for why decimal floats should be immune.

>  Also, I do not want the Py2.x decimal module releases to be complexified
> and slowed-down by Py3.0 deprecation warnings.  The module is already slow
> and has ease-of-use issues.

We're not talking about the usual arithmetic operations here, just
"%".  The functionality wouldn't go away, just the infix 1-character
spelling.  Note that the implementation of "remainder" is
extraordinarily expensive, just as the implementation of fmod for
binary floats is extraordinarily expensive:  it's not just the
precision at work, but also the difference in exponents.

+ - * / don't have the latter factor to contend with (hence
"extraordinarily").  Couple this with that floating mod of any kind is
almost surely rarely used, and worry about the overhead of a method or
function spelling seems a waste of tears.

Irony:  I've used decimal's remainder_near in real life, which has no
operator spelling.  I haven't used decimal's `remainder` except to
give examples in this thread :-)

From kristjan at ccpgames.com  Tue Jan 23 15:21:56 2007
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_V._J=F3nsson?=)
Date: Tue, 23 Jan 2007 14:21:56 +0000
Subject: [Python-Dev] Object creation hook
Message-ID: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>

Hello there.
I am trying to insert a hook into python enabling a callback for all just-created objects.  The intention is to debug and find memory leaks, e.g. by having the hook function insert the object into a WeakKeyDictionary.
I have already added a method to "object" to set such a hook, and "object_new" now calls it upon completion, but this is far from covering all places.  Initially, I thought object_init were the place, but almost no classes call object.__init__ from their __init__ method.  Then there is the separate case of old-style classes.

Any suggestions on how to do a global object creation hook in python?

Kristj?n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070123/66ca027d/attachment.htm 

From arigo at tunes.org  Tue Jan 23 17:03:03 2007
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 23 Jan 2007 17:03:03 +0100
Subject: [Python-Dev] Floor division
In-Reply-To: <1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
Message-ID: <20070123160303.GA12004@code0.codespeak.net>

Hi Tim,

On Sun, Jan 21, 2007 at 09:08:18PM -0500, Tim Peters wrote:
> >BTW - isn't that case in contradiction with the general Python rule that
> >if b > 0, then a % b should return a number between 0 included and b
> >excluded?
> 
> Sure.

You're not addressing my point, though, so I was probably not clear
enough.  My surprize was not that a % b was equal to b in this case, and
so not strictly smaller than b (because I expect such things when
working with floats).  My surprize was that Decimals don't even try to
be between 0 and b, and sometimes give a result that is far outside the
range.  This surprized me because I expected floats and Decimals to try
to give "approximately equal" results...


A bientot,

Armin

From ajm at flonidan.dk  Tue Jan 23 17:07:49 2007
From: ajm at flonidan.dk (Anders J. Munch)
Date: Tue, 23 Jan 2007 17:07:49 +0100
Subject: [Python-Dev] Floor division
Message-ID: <9B1795C95533CA46A83BA1EAD4B01030031F77@flonidanmail.flonidan.net>

[Tim Peters]
> [Anders J. Munch]
> > I did read the whole thread, and I saw your -1%1e100 example.  Mixing
> > floating-point numbers of very different magnitude can get you in
> > trouble - e.g. -1+1e100==1e100.  I don't think -1%1e100 is all that
> > worse.
> 
> Except that it's very easy to return an exactly correct result in that
> case:  -1.  

Except that the function that computes -1 is a different function.

Either it makes a difference which function is computed or it
doesn't:

If it makes a difference, because the first operand may be negative,
then the programmer absolutely has to choose the correct function, or
the program will have a bug.  If the correct function is one whose
result cannot be exactly represented, then there's nothing that can be
done about that.  Whether the function is called using a % infix
operator or using math.untrustworthy_intlike_fmod doesn't change the
fact that the result is going to be inexact.

If it doesn't make a difference, because the first operand is always
positive, then both functions provide exactly representable results.
So regardless, the current float.__mod__ doesn't create inexact
results where exact results are possible.

Suppose for example I'm normalising radians to the [0; 2*math.pi]
range using

   radians %= 2*math.pi

Changing that to

   radians = math.fmod(radians, 2*math.pi)

would simply be incorrect.  I could of course rewrite that to give the
correct result with a conditional and a bit more calculation, still
using math.fmod, but then the inexactness would reappear.

> It was natural to /want/ to extend it to floats.  That didn't work
> well, and to the contrary it's surprising precisely /because/ the
> behavior people enjoy with integers can fail when it's applied to
> floats.  

What makes you say it didn't work well?  What user experiences do you
base that on?

- Anders

From jimjjewett at gmail.com  Tue Jan 23 18:05:05 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Tue, 23 Jan 2007 12:05:05 -0500
Subject: [Python-Dev] Floor division
Message-ID: <fb6fbf560701230905m4398f727g8a7c8b81330099b3@mail.gmail.com>

Nick Maclaren wrote:
>... I can work with and teach almost any model,
> and have done so with some pretty weird ones.

I think python's model is "Whatever your other tools use.  Ask them."
And I think that is a reasonable choice.

For sensible input, the various models all work the same.

For dubious input, you get the same answer as you got from some other
tool, so you can automate some of your regression testing.

>... Python doesn't support any libm
>behaviour other than the one that it was compiled with, and that is
>often NOT what is wanted.

If you care (and know) enough to want a specific behavior on dubious
output, then you know more about your specfic problem then python ever
could.  If you can go on to say "and this platform libm is wrong, but
the other one over there is right", then you can probably recompile.

> Some people write Python that is intended to be robust and portable;
> it is those people who suffer.

If your users stick to sensible inputs, then it doesn't matter which
model you used.

If not, there is no way to get robust and portable; it is just a
matter of which users you annoy.

I could understand "throw an error, and *make* them choose a model if
they want to process bad data", but that isn't really any easier to
for a user to fix than the current non-model.

-jJ

From nmm1 at cus.cam.ac.uk  Tue Jan 23 18:35:50 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Tue, 23 Jan 2007 17:35:50 +0000
Subject: [Python-Dev]  Floor division
Message-ID: <E1H9PYk-0006f0-Bt@draco.cus.cam.ac.uk>

"Jim Jewett" <jimjjewett at gmail.com> wrote:
>
> >... I can work with and teach almost any model,
> > and have done so with some pretty weird ones.
>
> I think python's model is "Whatever your other tools use.  Ask them."
> And I think that is a reasonable choice.

Answer:  It's undefined.  Just because you have tested your code
today doesn't mean it will work tomorrow, or on a different set of
values (however similar), or that it will give the same answer
every time you do the same operation on the same input, or that the
effects will be limited to wrong answers and stray exceptions.

Still think that it is reasonable?

> > Some people write Python that is intended to be robust and portable;
> > it is those people who suffer.
> 
> If your users stick to sensible inputs, then it doesn't matter which
> model you used.

Sigh.  Let's step back a step.  Who decides when inputs are sensible?
And where is it documented?  Answers:  God alone knows, and nowhere.

One of Python's general principles is that its operations should
either do roughly what a reasonable user would expect, or it will
raise an exception.  It doesn't always get there, but it isn't bad.
What you are saying is that is undesirable.

The old Fortran and C model of saying that any user error can cause
any effect (including nasal demons) is tolerable only if there is
agreement on what IS an error, and there is some way for a user to
find that out.  In the case of C, neither is true.

> If not, there is no way to get robust and portable; it is just a
> matter of which users you annoy.

Well, actually, there is.  Though I agree that the techniques have
rather been forgotten in the past 30 years.  Python implements more
of them than most languages.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From ulissesf at gmail.com  Tue Jan 23 20:37:09 2007
From: ulissesf at gmail.com (Ulisses Furquim)
Date: Tue, 23 Jan 2007 16:37:09 -0300
Subject: [Python-Dev] Problem with signals in a single threaded application
Message-ID: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>

Hi,

I'm aware of the problems with signals in a multithreaded application,
but I was using signals in a single-threaded application and noticed
something that seemed wrong. Some signals were apparently being lost,
but when another signal came in the python handler for that "lost"
signal was being called.

The problem seems to be inside the signal module. The global variable
is_tripped is incremented every time a signal arrives. Then, inside
PyErr_CheckSignals() (the pending call that calls all python handlers
for signals that arrived) we can return immediately if is_tripped is
zero. If is_tripped is different than zero, we loop through all
signals calling the registered python handlers and after that we zero
is_tripped. This seems to be ok, but what happens if a signal arrives
after we've returned from its handler (or even after we've checked if
that signal arrived) and before we zero is_tripped? I guess we can
have a situation where is_tripped is zero but some Handlers[i].tripped
are not. In fact, I've inserted some debugging output and could see
that this actually happens and then I've written the following test
program to reproduce the problem.

#!/usr/bin/env python2.5

import sys
import os
import time
import signal

def alarm_handler(*args):
        sys.stderr.write('alarmmmmmmmmmmm!\n')

def sigio_handler(*args):
        sys.stderr.write('Entering SIGIO handler\n')
        os.kill(os.getpid(), signal.SIGALRM)
        sys.stderr.write('Leaving SIGIO handler\n')

signal.signal(signal.SIGIO, sigio_handler)
signal.signal(signal.SIGALRM, alarm_handler)

os.kill(os.getpid(), signal.SIGIO)

ini = time.time()
while True :
        if time.time() - ini > 3.0:
                sys.stderr.write('Loop!\n')
                ini = time.time()

When we run this program, the handler for the SIGALRM isn't called
after we return from the  SIGIO handler. We return to our main loop
and print 'Loop!' every 3 seconds aprox. and the SIGALRM handler is
called only when another signal arrives (like when we hit Ctrl-C).

I've read some threads about signals in the archives and I was under
the impression signals should work reliably on single-threaded
applications. Am I right?  I've thought about a way to fix this, but I
don't know what is the current plan for signals support in python, so
what can be done?

Best regards,

-- Ulisses

From jimjjewett at gmail.com  Tue Jan 23 20:47:26 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Tue, 23 Jan 2007 14:47:26 -0500
Subject: [Python-Dev] compex numbers (was Floor Division)
Message-ID: <fb6fbf560701231147k3e7e4fc0u16cc16a60cf6b1d8@mail.gmail.com>

Tim Peters wrote:

>  complex_new() ends with:

>  	cr.real -= ci.imag;
>  	cr.imag += ci.real;

> and I have no idea what that thinks it's doing.  Surely this isn't intended?!:

I think it is.  python.org/sf/1642844 adds comments to make it less unclear.

>  >>> complex(complex(1.0, 2.0), complex(10.0, 20.0))
>  (-19+12j)

>  WTF?

    >>> help(complex)
    Help on class complex in module __builtin__:

    class complex(object)
     |  complex(real[, imag]) -> complex number
     |
     |  Create a complex number from a real part and an optional imaginary part.
     |  This is equivalent to (real + imag*1j) where imag defaults to 0.

If "real" and "imag" are themselves complex numbers, then normalizing
the result will move the imaginary portion of the "real" vector into
the imaginary part and vice versa.

Note that changing this (to discard the imaginary parts) would break
passing complex numbers to their own constructor.

    >>> a=complex(1,2)
    >>> b=complex(a)
    >>> a==b
    True

-jJ

From nmm1 at cus.cam.ac.uk  Tue Jan 23 21:39:53 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Tue, 23 Jan 2007 20:39:53 +0000
Subject: [Python-Dev]  compex numbers (was Floor Division)
Message-ID: <E1H9SQr-0001zI-HK@libra.cus.cam.ac.uk>

"Jim Jewett" <jimjjewett at gmail.com> wrote:
> Tim Peters wrote:
> 
> >  complex_new() ends with:
> 
> >       cr.real -= ci.imag;
> >       cr.imag += ci.real;
> 
> > and I have no idea what that thinks it's doing.  Surely this isn't intended?!
> :
> 
> I think it is.  python.org/sf/1642844 adds comments to make it less unclear.

Agreed.

> 
> If "real" and "imag" are themselves complex numbers, then normalizing
> the result will move the imaginary portion of the "real" vector into
> the imaginary part and vice versa.

Not really.  What it does is to make complex(a,b) exactly equivalent
to a+1j*b.  For example:

>>> a = 1+2j
>>> b = 3+4j
>>> complex(a)
(1+2j)
>>> b*1j
(-4+3j)
>>> complex(a,b)
(-3+5j)

> Note that changing this (to discard the imaginary parts) would break
> passing complex numbers to their own constructor.

Eh?  Now, I am baffled.  There are several ways of changing it, all
of which would turn one bizarre behaviour into another - or would
raise an exception.  Personally, I would do the following:

complex(a) would permit a to be complex.

complex(a,b) would raise an exception if either a or b were complex.

But chacun a son gout (accents omitted).


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From mike.klaas at gmail.com  Tue Jan 23 21:51:38 2007
From: mike.klaas at gmail.com (Mike Klaas)
Date: Tue, 23 Jan 2007 12:51:38 -0800
Subject: [Python-Dev] Object creation hook
In-Reply-To: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>
Message-ID: <3d2ce8cb0701231251o2b2ea814ve29c3c42d19c3816@mail.gmail.com>

On 1/23/07, Kristj?n V. J?nsson <kristjan at ccpgames.com> wrote:

> Hello there.
>
> I am trying to insert a hook into python enabling a callback for all
> just-created objects.  The intention is to debug and find memory leaks, e.g.
> by having the hook function insert the object into a WeakKeyDictionary.
>
> I have already added a method to "object" to set such a hook, and
> "object_new" now calls it upon completion, but this is far from covering all
> places.  Initially, I thought object_init were the place, but almost no
> classes call object.__init__ from their __init__ method.  Then there is the
> separate case of old-style classes.
>
>
>
> Any suggestions on how to do a global object creation hook in python?

When I've used such things in the past, I usually had some idea which
classes I was interested in targeting.  I used a metaclass for doing
the tracking, and either invoked it on individual classes, or used
__metaclass__ = X to apply it (something like "class object(object):
__metaclass__ = X" would do the try for new-style class that inherit
from object directly).

-Mike

From tim.peters at gmail.com  Tue Jan 23 23:14:29 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 23 Jan 2007 17:14:29 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <20070123160303.GA12004@code0.codespeak.net>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<20070123160303.GA12004@code0.codespeak.net>
Message-ID: <1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>

[Armin]
>>> BTW - isn't that case in contradiction with the general Python rule that
>>> if b > 0, then a % b should return a number between 0 included and b
>>> excluded?

[Tim]
>> Sure.

[Armin]
> You're not addressing my point, though, so I was probably not clear
> enough.

"Sure" is the answer to all possible points :-)

> My surprize was not that a % b was equal to b in this case, and
> so not strictly smaller than b (because I expect such things when
> working with floats).

That's unnecessarily pessimistic:  there's no problem defining modular
reduction for floats that's exact.

>  My surprize was that Decimals don't even try to
> be between 0 and b, and sometimes give a result that is far outside the
> range.  This surprized me because I expected floats and Decimals to try
> to give "approximately equal" results...

Sure, you do expect that, and sure, they don't.  Now what?

The `decimal` module is an implementation of IBM's proposed standard
for decimal arithmetic:

    http://www2.hursley.ibm.com/decimal/

It requires two mod-like operations, neither of which behave like
Python's "number theoretic" mod.  Nobody involved cared to extend the
proposed standard by adding a third mod-like operation.

For some reason `decimal` implemented __mod__ as the proposed
standard's "remainder" operation.  That's the immediate source of your
surprise.  IMO `decimal` should not have implemented __mod__ at all,
as Python's number-theoretic mod is not part of the proposed standard,
is a poor basis for a floating-point mod regardess, and it was a
mistake to implement decimal % decimal in a way so visibly different
from float % float and integer % integer:  it confuses the meaning of
"%".  That's your complaint, right?

My preferred "solution" is to remove __mod__, __divmod__, and
__floordiv__ from all flavors of floats (both binary and decimal) in
P3K.

That addresses your concern in that "decimal % decimal" would raise an
exception in P3K (it wouldn't be implemented).  A user who wanted some
form of decimal float modular reduction would need to read the docs,
decide which of the two supported such functions they want, and
explicitly ask for it.  Similarly, a user who wanted some from of
binary float modular reduction would need to invoke math.fmod()
explicitly -- or, possibly, if someone contributes the code needed to
implement C99's additional "remainder" function cross-platform, that
too:

IBM's spec          same-as C
----------          ---------
remainder           fmod (C89 & C99)
remainder-near      remainder (C99 & IEEE-754)

It's not a coincidence that all standards addressing modular reduction
for floats converge on essentially those two definitions.

I can't be made to feel guilty about this :-)

For P2 I don't propose any changes -- and in which case my only
response is just "sure - yes - right - decimal's __mod__ in fact does
not act like Python's integer __mod__ in mixed-sign cases -- and
neither does decimal's __floordiv__ or decimal's __divmod__ act like
their integer counterparts in mixed-sign cases".

From steve at holdenweb.com  Tue Jan 23 23:18:02 2007
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 23 Jan 2007 22:18:02 +0000
Subject: [Python-Dev] Object creation hook
In-Reply-To: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>
Message-ID: <45B6899A.8090106@holdenweb.com>

Kristj?n V. J?nsson wrote:
> Hello there.
> 
> I am trying to insert a hook into python enabling a callback for all 
> just-created objects.  The intention is to debug and find memory leaks, 
> e.g. by having the hook function insert the object into a WeakKeyDictionary.
> 
> I have already added a method to ?object? to set such a hook, and 
> ?object_new? now calls it upon completion, but this is far from covering 
> all places.  Initially, I thought object_init were the place, but almost 
> no classes call object.__init__ from their __init__ method.  Then there 
> is the separate case of old-style classes.
> 
>  
I suppose there's no requirement on C-defined types to actually call 
object_new, so you will need to audit them and make sure they do.

PyClass_New in classobject.c is another obvious place - the creation of 
old-style instances doesn't appear to involve object_new.

> 
> Any suggestions on how to do a global object creation hook in python?
> 
Nothing other than the above.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com


From gareth.mccaughan at pobox.com  Tue Jan 23 23:21:42 2007
From: gareth.mccaughan at pobox.com (Gareth McCaughan)
Date: Tue, 23 Jan 2007 22:21:42 +0000
Subject: [Python-Dev] Floor division
In-Reply-To: <1f7befae0701222301g2d9e0a47y9692b072ddf7823@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<ca471dc20701220757gcb24a5dkcf90b39ba22943f6@mail.gmail.com>
	<1f7befae0701222301g2d9e0a47y9692b072ddf7823@mail.gmail.com>
Message-ID: <200701232221.43590.gareth.mccaughan@pobox.com>

On Tuesday 23 January 2007 07:01, Tim Peters wrote:

> complex_new() ends with:
>
> 	cr.real -= ci.imag;
> 	cr.imag += ci.real;
>
> and I have no idea what that thinks it's doing.  Surely this isn't 
intended?!:
> >>> complex(complex(1.0, 2.0), complex(10.0, 20.0))
>
> (-19+12j)
>
> WTF?  In any case, that's also what's destroying the sign of the
> imaginary part in complex(1.0, -0.0).

It seems pretty clear what it thinks it's doing -- namely,
defining complex(a,b) = a + ib even when a,b are complex.
And half of why it does that is clear: you want complex(a)=a
when a is complex. Why b should be allowed to be complex too,
though, it's hard to imagine.

-- 
g

From martin at v.loewis.de  Wed Jan 24 00:32:09 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 24 Jan 2007 00:32:09 +0100
Subject: [Python-Dev] Object creation hook
In-Reply-To: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>
Message-ID: <45B69AF9.3030607@v.loewis.de>

Kristj?n V. J?nsson schrieb:
> I am trying to insert a hook into python enabling a callback for all
> just-created objects.  The intention is to debug and find memory leaks,
> e.g. by having the hook function insert the object into a WeakKeyDictionary.

I'd like to point out that this isn't a python-dev question, but more
appropriate for comp.lang.python (as it is of the "how do I x with
Python?" kind).

I would use a debug build, and use sys.getobjects to determine all
objects and find leaks.

Regards,
Martin

From aahz at pythoncraft.com  Wed Jan 24 00:44:30 2007
From: aahz at pythoncraft.com (Aahz)
Date: Tue, 23 Jan 2007 15:44:30 -0800
Subject: [Python-Dev] Problem with signals in a single threaded
	application
In-Reply-To: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
Message-ID: <20070123234430.GA3636@panix.com>

On Tue, Jan 23, 2007, Ulisses Furquim wrote:
>
> I've read some threads about signals in the archives and I was under
> the impression signals should work reliably on single-threaded
> applications. Am I right?  I've thought about a way to fix this, but I
> don't know what is the current plan for signals support in python, so
> what can be done?

For starters, please post a bug report to SF so that this issue doesn't
get lost.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

Help a hearing-impaired person: http://rule6.info/hearing.html

From ark at acm.org  Wed Jan 24 05:11:18 2007
From: ark at acm.org (Andrew Koenig)
Date: Tue, 23 Jan 2007 23:11:18 -0500
Subject: [Python-Dev] compex numbers (was Floor Division)
In-Reply-To: <fb6fbf560701231147k3e7e4fc0u16cc16a60cf6b1d8@mail.gmail.com>
Message-ID: <002e01c73f6d$b2e389b0$6402a8c0@arkdesktop>

> If "real" and "imag" are themselves complex numbers, then normalizing
> the result will move the imaginary portion of the "real" vector into
> the imaginary part and vice versa.

Not quite.

>>> complex(1,1j)
0j
>>> complex(0,1j)
(-1+0j)

So it moves the imaginary portion of the "imag" argument into the real part
of the result with change of sign, on the basis that 1j*1j == -1.




From martin at v.loewis.de  Wed Jan 24 08:46:41 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 24 Jan 2007 08:46:41 +0100
Subject: [Python-Dev] Problem with signals in a single threaded
	application
In-Reply-To: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
Message-ID: <45B70EE1.7080809@v.loewis.de>

Ulisses Furquim schrieb:
> I've read some threads about signals in the archives and I was under
> the impression signals should work reliably on single-threaded
> applications. Am I right?  I've thought about a way to fix this, but I
> don't know what is the current plan for signals support in python, so
> what can be done?

I agree it's a bug, and I agree with your proposed analysis. Please
try to come up with a patch (e.g. by putting a while(is_tripped) loop
around the for loop). Also make sure you include test case.

Thanks for contributing,

Martin

From snaury at gmail.com  Wed Jan 24 09:39:59 2007
From: snaury at gmail.com (Alexey Borzenkov)
Date: Wed, 24 Jan 2007 11:39:59 +0300
Subject: [Python-Dev] Floor division
In-Reply-To: <200701232221.43590.gareth.mccaughan@pobox.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<ca471dc20701220757gcb24a5dkcf90b39ba22943f6@mail.gmail.com>
	<1f7befae0701222301g2d9e0a47y9692b072ddf7823@mail.gmail.com>
	<200701232221.43590.gareth.mccaughan@pobox.com>
Message-ID: <e2480c70701240039g48816902keb55244f521ecf3@mail.gmail.com>

On 1/24/07, Gareth McCaughan <gareth.mccaughan at pobox.com> wrote:
>
> > >>> complex(complex(1.0, 2.0), complex(10.0, 20.0))
> >
> > (-19+12j)
> >
> > WTF?  In any case, that's also what's destroying the sign of the
> > imaginary part in complex(1.0, -0.0).
>
> It seems pretty clear what it thinks it's doing -- namely,
> defining complex(a,b) = a + ib even when a,b are complex.
> And half of why it does that is clear: you want complex(a)=a
> when a is complex. Why b should be allowed to be complex too,
> though, it's hard to imagine.


I think that's the right thing to do, because that is mathematically
correct. j is just an imaginary number with a property that j*j = -1. So
(a+bj) + (c+dj)j = (a-d) + (b+c)j. Complex numbers are not just magic
pairs with two numbers and have actual mathematical rules.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070124/3e327604/attachment.htm 

From gareth.mccaughan at pobox.com  Wed Jan 24 10:00:04 2007
From: gareth.mccaughan at pobox.com (Gareth McCaughan)
Date: Wed, 24 Jan 2007 09:00:04 +0000
Subject: [Python-Dev] Floor division
In-Reply-To: <e2480c70701240039g48816902keb55244f521ecf3@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<200701232221.43590.gareth.mccaughan@pobox.com>
	<e2480c70701240039g48816902keb55244f521ecf3@mail.gmail.com>
Message-ID: <200701240900.04245.gareth.mccaughan@pobox.com>

On Wednesday 24 January 2007 08:39, Alexey Borzenkov wrote:

[me, about complex():]
> > It seems pretty clear what it thinks it's doing -- namely,
> > defining complex(a,b) = a + ib even when a,b are complex.
> > And half of why it does that is clear: you want complex(a)=a
> > when a is complex. Why b should be allowed to be complex too,
> > though, it's hard to imagine.

[Alexey:]
> I think that's the right thing to do, because that is mathematically
> correct. j is just an imaginary number with a property that j*j = -1. So
> (a+bj) + (c+dj)j = (a-d) + (b+c)j.

Yes, thanks, I know what j is, and I know how to multiply
complex numbers. (All of which you could have deduced from
reading what I wrote, as it happens.) The question is whether
it makes sense to define complex(a,b) = a+ib for all a,b
or whether the two-argument form is always in practice going
to be used with real numbers[1]. If it is, which seems pretty
plausible to me, then changing complex() to complain when
passed two complex numbers would (1) notify users sooner
when they have errors in their programs, (2) simplify the
code, and (3) avoid the arguably broken behaviour Tim was
remarking on, where complex(-0.0).real is +0 instead of -0.

    [1] For the avoidance of ambiguity: "real" is not
        synonymous with "double-precision floating-point".

>                                    Complex numbers are not just magic 
> pairs with two numbers and have actual mathematical rules.

Gosh, and there I was thinking that complex numbers were magic
pairs where you just make stuff up at random. Silly me.

-- 
Gareth McCaughan
PhD in pure mathematics, University of Cambridge

From snaury at gmail.com  Wed Jan 24 11:20:42 2007
From: snaury at gmail.com (Alexey Borzenkov)
Date: Wed, 24 Jan 2007 13:20:42 +0300
Subject: [Python-Dev] Floor division
In-Reply-To: <200701240900.04245.gareth.mccaughan@pobox.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<200701232221.43590.gareth.mccaughan@pobox.com>
	<e2480c70701240039g48816902keb55244f521ecf3@mail.gmail.com>
	<200701240900.04245.gareth.mccaughan@pobox.com>
Message-ID: <e2480c70701240220q10701759ra7647fde807fc73@mail.gmail.com>

On 1/24/07, Gareth McCaughan <gareth.mccaughan at pobox.com> wrote:

> [Alexey:]
> > I think that's the right thing to do, because that is mathematically
> > correct. j is just an imaginary number with a property that j*j = -1. So
>
> > (a+bj) + (c+dj)j = (a-d) + (b+c)j.
>
> Yes, thanks, I know what j is, and I know how to multiply
> complex numbers. (All of which you could have deduced from
> reading what I wrote, as it happens.) The question is whether
> it makes sense to define complex(a,b) = a+ib for all a,b
> or whether the two-argument form is always in practice going
> to be used with real numbers[1]. If it is, which seems pretty
> plausible to me, then changing complex() to complain when
> passed two complex numbers would (1) notify users sooner
> when they have errors in their programs, (2) simplify the
> code, and (3) avoid the arguably broken behaviour Tim was
> remarking on, where complex(-0.0).real is +0 instead of -0.


Haven't looked in source code for complex constructor yet, but I think that
if it changes sign of -0.0 then it just does something wrong and needs
fixing without change in behaviour. Maybe it could check if numbers it got
on input are real or complex and proceed accordingly so that it never gets
to computing -0.0-(+0.0), i.e. when second argument is not a complex number
it could just add it to imaginary part of first argument, but skip
substracting inexistant 0.0 from first argument's real part. Change of
behaviour like ignoring imaginary part of second argument seems bad to me,
and that's my only point. Besides, documentation (at least for Python 2.4)
clearly states that second argument can be a complex number:

  *complex*( [real[, imag]]) Create a complex number with the value *real +
imag*j* or convert a string or number to a complex number. If the first
parameter is a string, it will be interpreted as a complex number and the
function must be called without a second parameter. The second parameter can
never be a string. Each argument may be *any numeric type* (*including
complex*). If imag is omitted, it defaults to zero and the function serves
as a numeric conversion function like int(), long() and float(). If both
arguments are omitted, returns 0j.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070124/4e5e6fd2/attachment.htm 

From gareth.mccaughan at pobox.com  Wed Jan 24 12:22:09 2007
From: gareth.mccaughan at pobox.com (Gareth McCaughan)
Date: Wed, 24 Jan 2007 11:22:09 +0000
Subject: [Python-Dev] Floor division
In-Reply-To: <e2480c70701240220q10701759ra7647fde807fc73@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<200701240900.04245.gareth.mccaughan@pobox.com>
	<e2480c70701240220q10701759ra7647fde807fc73@mail.gmail.com>
Message-ID: <200701241122.09903.gareth.mccaughan@pobox.com>

On Wednesday 24 January 2007 10:20, Alexey Borzenkov wrote:

> > > I think that's the right thing to do, because that is mathematically
> > > correct. j is just an imaginary number with a property that j*j = -1.
> > > So
> > >
> > > (a+bj) + (c+dj)j = (a-d) + (b+c)j.
> >
> > Yes, thanks, I know what j is, and I know how to multiply
> > complex numbers. (All of which you could have deduced from
> > reading what I wrote, as it happens.) The question is whether
> > it makes sense to define complex(a,b) = a+ib for all a,b
> > or whether the two-argument form is always in practice going
> > to be used with real numbers[1]. If it is, which seems pretty
> > plausible to me, then changing complex() to complain when
> > passed two complex numbers would (1) notify users sooner
> > when they have errors in their programs, (2) simplify the
> > code, and (3) avoid the arguably broken behaviour Tim was
> > remarking on, where complex(-0.0).real is +0 instead of -0.
>
> Haven't looked in source code for complex constructor yet, but I think that
> if it changes sign of -0.0 then it just does something wrong and needs
> fixing without change in behaviour. Maybe it could check if numbers it got
> on input are real or complex and proceed accordingly so that it never gets
> to computing -0.0-(+0.0), i.e. when second argument is not a complex number
> it could just add it to imaginary part of first argument, but skip
> substracting inexistant 0.0 from first argument's real part. Change of
> behaviour like ignoring imaginary part of second argument seems bad to me,
> and that's my only point. Besides, documentation (at least for Python 2.4)
> clearly states that second argument can be a complex number:

I'm not suggesting that it should ignore the imaginary part of
the second argument, and I don't think anyone else is either.
What might make sense is for passing a complex second argument
to raise an exception.

(I don't particularly object to the present behaviour either.)

The fact that the documentation states explicitly that the
second argument can be a complex number is probably sufficient
reason for not changing the behaviour, at least before 3.0.

-- 
g

From nmm1 at cus.cam.ac.uk  Wed Jan 24 12:22:09 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Wed, 24 Jan 2007 11:22:09 +0000
Subject: [Python-Dev] Problem with signals in a single threaded
	application
Message-ID: <E1H9gCf-00072A-1B@libra.cus.cam.ac.uk>

On Tue, Jan 23, 2007, Ulisses Furquim wrote:
>
> I've read some threads about signals in the archives and I was under
> the impression signals should work reliably on single-threaded
> applications. Am I right?  I've thought about a way to fix this, but I
> don't know what is the current plan for signals support in python, so
> what can be done?

This one looks like an oversight in Python code, and so is a bug,
but it is important to note that signals do NOT work reliably under
any Unix or Microsoft system.  Inter alia, all of the following are
likely to lead to lost signals:

Two related signals received between two 'checkpoints' (i.e. when
the signal is tested and cleared).  You may only get one of them,
and 'related' does not mean 'the same'.

A second signal received while the first is being 'handled' by the
operating system or language run-time system.

A signal sent while the operating system is doing certain things to
the application (including, sometimes, when it is swapped out or
deep in I/O.)

And there is more, some of which can cause program misbehaviour or
crashes.  You are also right that threading makes the situation a
lot worse.

Obviously, Unix and Microsoft systems depend on signals, so you
can't simply regard them as hopelessly broken, but you can't assume
that they are RELIABLE.  All code should be designed to cope with
the case of signals getting lost, if at all possible.  Defending
yourself against the other failures is an almost hopeless task,
but luckily they are extremely rare except on specialist systems.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Wed Jan 24 12:28:53 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Wed, 24 Jan 2007 11:28:53 +0000
Subject: [Python-Dev] Complex constructors [was Re:  Floor division]
Message-ID: <E1H9gJB-00075U-73@libra.cus.cam.ac.uk>

Gareth McCaughan <gareth.mccaughan at pobox.com> wrote:
> 
> ...  The question is whether
> it makes sense to define complex(a,b) = a+ib for all a,b
> or whether the two-argument form is always in practice going
> to be used with real numbers[1]. If it is, which seems pretty
> plausible to me, then changing complex() to complain when
> passed two complex numbers would (1) notify users sooner
> when they have errors in their programs, (2) simplify the
> code, and (3) avoid the arguably broken behaviour Tim was
> remarking on, where complex(-0.0).real is +0 instead of -0.
> 
>     [1] For the avoidance of ambiguity: "real" is not
>         synonymous with "double-precision floating-point".

Precisely.  On this matter, does anyone know of an application
where making that change would harm anything?  I cannot think of
a circumstance under which the current behaviour adds any useful
function over the one that raises an exception if there are two
arguments and either is complex.

Yes, of course, SOME people will find it cool to write complex(a,b)
when they really mean a+1j*b, but ....


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From kristjan at ccpgames.com  Wed Jan 24 12:27:49 2007
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_V._J=F3nsson?=)
Date: Wed, 24 Jan 2007 11:27:49 +0000
Subject: [Python-Dev] Object creation hook
In-Reply-To: <45B69AF9.3030607@v.loewis.de>
References: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>
	<45B69AF9.3030607@v.loewis.de>
Message-ID: <4E9372E6B2234D4F859320D896059A9504F58E83@exchis.ccp.ad.local>

Thanks, but the question is really, "how do I build a better debug hook than sys.getobjects?" so I argue this is a valid python-dev question.

We have been using gc.get_objects() but it has several problems:
1) It returns all objects in the system.  This results in a list so long that it often kills the system.  Our system is of a scale that makes this very risky.
2) There is no way to frame certain operations and get just those objects that were created during their execution.  In our case, we would like to get the server cluster running, then frame a certain operation to get a callback for all created objects, so that we could track that they were later destroyed correctly.  I have done this previously by storing the id()s of all objects returned from gc.get_objects() and comparing them "before" and "after" but this suffers from 1) above, and the ambiguity of id() in the face of objects being created and destroyed.


Working with the metaclasses sounds reasonable if one has the luxury of changing the entire codebase to use a different metaclass.  It also doesn't work with old style classes (plenty of those), c extensions, and builtins.

(I was a bit dismayed that I couldn't assign to object.__init__ post-hoc from a python script, I'm fairly sure that is possible in Ruby :)  but I digress...)

My latest attempt was to modify _PyObject_GC_TRACK(o) in objimpl.h, adding this final line:
if (PyCCP_CreationHook) PyCCP_CreationHookFunc(o);\

The function then looks like this:

void PyCCP_CreationHookFunc(PyObject * obj)
{
        if (PyCCP_CreationHook) {
                PyObject *result, *tmp = PyCCP_CreationHook;
                PyCCP_CreationHook = 0; //guard against recursion
                result = PyObject_CallFunctionObjArgs(PyCCP_CreationHook, obj, 0);
                Py_XDECREF(result);
                if (!result)
                        PyErr_Clear();
                PyCCP_CreationHook = tmp;
        }
}

Doing this, and setting a no-op function as as the PyCCP_CreationHook, does seem to work for a while in the interactive python shell, but it then crashes and I haven't been able to work out the crash.  In any case, doing stuff at the point of GC list insertion is very hairy, especially in the face of __del__ methods and such (of which we don't have many)

I am waiting to get Rational Purify set up on my machine and then I'll maybe be able to understand the crash case better.

Cheers,
Kristj?n


-----Original Message-----
From: "Martin v. L?wis" [mailto:martin at v.loewis.de]
Sent: 23. jan?ar 2007 23:32
To: Kristj?n V. J?nsson
Cc: 'python-dev at python.org'
Subject: Re: [Python-Dev] Object creation hook

Kristj?n V. J?nsson schrieb:
> I am trying to insert a hook into python enabling a callback for all
> just-created objects.  The intention is to debug and find memory leaks,
> e.g. by having the hook function insert the object into a WeakKeyDictionary.

I'd like to point out that this isn't a python-dev question, but more
appropriate for comp.lang.python (as it is of the "how do I x with
Python?" kind).

I would use a debug build, and use sys.getobjects to determine all
objects and find leaks.

Regards,
Martin

From aahz at pythoncraft.com  Wed Jan 24 15:58:05 2007
From: aahz at pythoncraft.com (Aahz)
Date: Wed, 24 Jan 2007 06:58:05 -0800
Subject: [Python-Dev] [Python-checkins] buildbot failure in amd64 gentoo
	2.5
In-Reply-To: <45B5FEF6.1030503@gmail.com>
References: <20070122192355.B20C31E4007@bag.python.org>
	<ep33cv$a1o$1@sea.gmane.org> <45B52D29.4020001@v.loewis.de>
	<874pqip0a5.fsf@lothar.com> <ep4leo$1k9$1@sea.gmane.org>
	<87lkju12hq.fsf@starship.python.net> <45B5FEF6.1030503@gmail.com>
Message-ID: <20070124145805.GA13284@panix.com>

On Tue, Jan 23, 2007, Nick Coghlan wrote:
>
> Allowing a project to define a question & answer should do the trick - 
> essentially a password like Skip suggested, but with a reminder right 
> there on the page:
> 
> "Type this project's name in lowercase in this box: _____"
> "Type the BDFL's first name in lowercase in this box: ____"
> "Type the branch name in this box: ____"
> 
> etc.
> 
> It's easier than a normal CAPTCHA because its OK to assume a lot more 
> knowledge on the part of the intended audience.

...and it works with text-based browsing.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

Help a hearing-impaired person: http://rule6.info/hearing.html

From benji at benjiyork.com  Wed Jan 24 17:09:19 2007
From: benji at benjiyork.com (Benji York)
Date: Wed, 24 Jan 2007 11:09:19 -0500
Subject: [Python-Dev] [Python-checkins] buildbot failure in amd64 gentoo
 2.5
In-Reply-To: <87lkju12hq.fsf@starship.python.net>
References: <20070122192355.B20C31E4007@bag.python.org>	<ep33cv$a1o$1@sea.gmane.org>
	<45B52D29.4020001@v.loewis.de>	<874pqip0a5.fsf@lothar.com>
	<ep4leo$1k9$1@sea.gmane.org> <87lkju12hq.fsf@starship.python.net>
Message-ID: <45B784AF.5040005@benjiyork.com>

Michael Hudson wrote:
> Giovanni Bajo <rasky at develer.com> writes:
> 
>> On 23/01/2007 10.20, Brian Warner wrote:
>>
>>>>> Do I miss something here, or is the buildbot hit by spammers now?
>>>> It looks like it is. If that continues, we have to disable the web
>>>> triggers.
>>> Good grief. If anyone has any bright ideas about simple ways to change that
>>> form to make it less vulnerable to the spambots, I'd be happy to incorporate
>>> them into Buildbot.
>> I'd throw a CAPTCHA in. There are even some written in Python.
> 
> I'd guess even the simplest thing would work:
> 
> "Type "Python" into this box: ______"

The approach (outlined here 
http://damienkatz.net/2007/01/negative_captch.html) of having a hidden 
field that is "attractive" to spam bots, but hidden from humans would 
seem to be as effective without the (slight) annoyances of filling in an 
extra field every time.
--
Benji York

From ulissesf at gmail.com  Wed Jan 24 18:32:06 2007
From: ulissesf at gmail.com (Ulisses Furquim)
Date: Wed, 24 Jan 2007 14:32:06 -0300
Subject: [Python-Dev] Problem with signals in a single threaded
	application
In-Reply-To: <45B70EE1.7080809@v.loewis.de>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
	<45B70EE1.7080809@v.loewis.de>
Message-ID: <d23110170701240932g46252d20nfa2e4e664f7587bc@mail.gmail.com>

On 1/24/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> I agree it's a bug, and I agree with your proposed analysis. Please
> try to come up with a patch (e.g. by putting a while(is_tripped) loop
> around the for loop). Also make sure you include test case.

Ok. I was discussing this problem with a colleague of mine and he had
a nice idea on how to fix this. We couldn't think of anything wrong
with the solution, so I'm testing it right now and gonna write a test
case afterwards.

> Thanks for contributing,

No problem. :-)

Regards,

-- Ulisses

From ulissesf at gmail.com  Wed Jan 24 18:51:49 2007
From: ulissesf at gmail.com (Ulisses Furquim)
Date: Wed, 24 Jan 2007 14:51:49 -0300
Subject: [Python-Dev] Problem with signals in a single threaded
	application
In-Reply-To: <E1H9gCf-00072A-1B@libra.cus.cam.ac.uk>
References: <E1H9gCf-00072A-1B@libra.cus.cam.ac.uk>
Message-ID: <d23110170701240951v573ba55cy32c7941afc96cd39@mail.gmail.com>

On 1/24/07, Nick Maclaren <nmm1 at cus.cam.ac.uk> wrote:
> Obviously, Unix and Microsoft systems depend on signals, so you
> can't simply regard them as hopelessly broken, but you can't assume
> that they are RELIABLE.  All code should be designed to cope with
> the case of signals getting lost, if at all possible.  Defending
> yourself against the other failures is an almost hopeless task,
> but luckily they are extremely rare except on specialist systems.

I couldn't agree more. I might have misused the word "reliably". At
first glance I thought the signal was being lost but then I've
discovered what was really happening.

Regards,

-- Ulisses

From gjcarneiro at gmail.com  Wed Jan 24 19:54:52 2007
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Wed, 24 Jan 2007 18:54:52 +0000
Subject: [Python-Dev] Problem with signals in a single threaded
	application
In-Reply-To: <45B70EE1.7080809@v.loewis.de>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
	<45B70EE1.7080809@v.loewis.de>
Message-ID: <a467ca4f0701241054q6191afa1ge8f2d57ffafbe02a@mail.gmail.com>

On 1/24/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>
> Ulisses Furquim schrieb:
> > I've read some threads about signals in the archives and I was under
> > the impression signals should work reliably on single-threaded
> > applications. Am I right?  I've thought about a way to fix this, but I
> > don't know what is the current plan for signals support in python, so
> > what can be done?
>
> I agree it's a bug, and I agree with your proposed analysis. Please
> try to come up with a patch (e.g. by putting a while(is_tripped) loop
> around the for loop). Also make sure you include test case.


   What about http://www.python.org/sf/1564547 ?  It tries to solve a
different problem, but I think it also fixes this one; at least as much as
possible with the braindead unix signal programming interface...

-- 
Gustavo J. A. M. Carneiro
"The universe is always one step beyond logic."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070124/bb26a3da/attachment-0001.html 

From martin at v.loewis.de  Wed Jan 24 20:12:01 2007
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 24 Jan 2007 20:12:01 +0100
Subject: [Python-Dev] Problem with signals in a single threaded
	application
In-Reply-To: <a467ca4f0701241054q6191afa1ge8f2d57ffafbe02a@mail.gmail.com>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>	
	<45B70EE1.7080809@v.loewis.de>
	<a467ca4f0701241054q6191afa1ge8f2d57ffafbe02a@mail.gmail.com>
Message-ID: <45B7AF81.9010504@v.loewis.de>

Gustavo Carneiro schrieb:
>    What about http://www.python.org/sf/1564547 ?  It tries to solve a
> different problem, but I think it also fixes this one; at least as much
> as possible with the braindead unix signal programming interface...

I'm sceptical. It is way too much code for me to review, so I'm unable
to comment whether it fixes the problem under discussion. I feel that
this problem should find a much simpler solution.

Regards,
Martin

From ulissesf at gmail.com  Wed Jan 24 20:52:10 2007
From: ulissesf at gmail.com (Ulisses Furquim)
Date: Wed, 24 Jan 2007 16:52:10 -0300
Subject: [Python-Dev] Problem with signals in a single threaded
	application
In-Reply-To: <d23110170701240932g46252d20nfa2e4e664f7587bc@mail.gmail.com>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
	<45B70EE1.7080809@v.loewis.de>
	<d23110170701240932g46252d20nfa2e4e664f7587bc@mail.gmail.com>
Message-ID: <d23110170701241152j7a016705lae8c072991e5c909@mail.gmail.com>

On 1/24/07, Ulisses Furquim <ulissesf at gmail.com> wrote:
> On 1/24/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > I agree it's a bug, and I agree with your proposed analysis. Please
> > try to come up with a patch (e.g. by putting a while(is_tripped) loop
> > around the for loop). Also make sure you include test case.
>
> Ok. I was discussing this problem with a colleague of mine and he had
> a nice idea on how to fix this. We couldn't think of anything wrong
> with the solution, so I'm testing it right now and gonna write a test
> case afterwards.

Here is the link for the bug with both the test program and first
attempt to solve it.

http://www.python.org/sf/1643738

Regards,

-- Ulisses

From arigo at tunes.org  Thu Jan 25 10:56:51 2007
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 25 Jan 2007 10:56:51 +0100
Subject: [Python-Dev] Floor division
In-Reply-To: <1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<20070123160303.GA12004@code0.codespeak.net>
	<1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>
Message-ID: <20070125095651.GA14446@code0.codespeak.net>

Hi Tim,

On Tue, Jan 23, 2007 at 05:14:29PM -0500, Tim Peters wrote:
> For some reason `decimal` implemented __mod__ as the proposed
> standard's "remainder" operation.  That's the immediate source of your
> surprise.  IMO `decimal` should not have implemented __mod__ at all,
> as Python's number-theoretic mod is not part of the proposed standard,
> is a poor basis for a floating-point mod regardess, and it was a
> mistake to implement decimal % decimal in a way so visibly different
> from float % float and integer % integer:  it confuses the meaning of
> "%".  That's your complaint, right?

Thanks for the clarification.  Yes, it makes sense that __mod__,
__divmod__ and __floordiv__ on float and decimal would eventually follow
the same path as for complex (where they make even less sense and
already raise a DeprecationWarning).


A bientot,

Armin.

From nmm1 at cus.cam.ac.uk  Thu Jan 25 11:24:38 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Thu, 25 Jan 2007 10:24:38 +0000
Subject: [Python-Dev] Floor division
Message-ID: <E1HA1mY-00059L-93@virgo.cus.cam.ac.uk>

Armin Rigo <arigo at tunes.org> wrote:
>
> Thanks for the clarification.  Yes, it makes sense that __mod__,
> __divmod__ and __floordiv__ on float and decimal would eventually follow
> the same path as for complex (where they make even less sense and
> already raise a DeprecationWarning).

Yes.  Though them not doing so would also make sense.  The difference
is that they make no mathematical sense for complex, but the problems
with float are caused by floating-point (and do not occur for the
mathematical reals).

There is an argument for saying that divmod should return a long
quotient and a float remainder, which is what C99 has specified for
remquo (except that it requires only the last 3 bits of the quotient
for reasons that completely baffle me).  Linux misimplemented that
the last time I looked.

Personally, I think that it is bonkers, as it is fiendishly expensive
compared to its usefulness - especially with Decimal!  But it isn't
obviously WRONG.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From gjcarneiro at gmail.com  Thu Jan 25 19:05:31 2007
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Thu, 25 Jan 2007 18:05:31 +0000
Subject: [Python-Dev] Problem with signals in a single threaded
	application
In-Reply-To: <45B7AF81.9010504@v.loewis.de>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
	<45B70EE1.7080809@v.loewis.de>
	<a467ca4f0701241054q6191afa1ge8f2d57ffafbe02a@mail.gmail.com>
	<45B7AF81.9010504@v.loewis.de>
Message-ID: <a467ca4f0701251005q32c21e1fpfce204b3f1bc523e@mail.gmail.com>

On 1/24/07, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>
> Gustavo Carneiro schrieb:
> >    What about http://www.python.org/sf/1564547 ?  It tries to solve a
> > different problem, but I think it also fixes this one; at least as much
> > as possible with the braindead unix signal programming interface...
>
> I'm sceptical. It is way too much code for me to review, so I'm unable
> to comment whether it fixes the problem under discussion.


  Due to a bug, my patch  didn't fix this bug.  But I fixed the patch and it
now fixes both my problem and Ulisses'.

I feel that
> this problem should find a much simpler solution.


   The problem is that if you apply Ulisses' patch then my patch, Ulisses'
changes will simply disappear because my patch handles signals and a much
safer way, completely bypassing the 'add/make pending calls' system, since
this system is patently *not* async safe, no matter how much you tweak it.

  Yes, I know my patch is not very small (though not that big either), but
the signal module was in dire need of refactoring.  And you can observe how
much simpler the signal module becomes after that patch.

  Regards.

-- 
Gustavo J. A. M. Carneiro
"The universe is always one step beyond logic."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070125/b221ccdd/attachment.html 

From guido at python.org  Thu Jan 25 22:22:54 2007
From: guido at python.org (Guido van Rossum)
Date: Thu, 25 Jan 2007 13:22:54 -0800
Subject: [Python-Dev] Floor division
In-Reply-To: <20070125095651.GA14446@code0.codespeak.net>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<20070123160303.GA12004@code0.codespeak.net>
	<1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>
	<20070125095651.GA14446@code0.codespeak.net>
Message-ID: <ca471dc20701251322l1475908ao1ceb68e2b8d92f3d@mail.gmail.com>

The only thing I would miss about this is that I am used to write
certain timing loops that like to sync on whole seconds, by taking
time.time() % 1.0 which nicely gives me the milliseconds in the
current second. E.g.

while True:
  do_something_expensive_once_a_second_on_the_second()
  now = time.time()
  time.sleep(1.0 - (now % 1.0))

I guess I could use (now - int(now)) in a pinch, assuming int()
continues to truncate.

--Guido

On 1/25/07, Armin Rigo <arigo at tunes.org> wrote:
> Hi Tim,
>
> On Tue, Jan 23, 2007 at 05:14:29PM -0500, Tim Peters wrote:
> > For some reason `decimal` implemented __mod__ as the proposed
> > standard's "remainder" operation.  That's the immediate source of your
> > surprise.  IMO `decimal` should not have implemented __mod__ at all,
> > as Python's number-theoretic mod is not part of the proposed standard,
> > is a poor basis for a floating-point mod regardess, and it was a
> > mistake to implement decimal % decimal in a way so visibly different
> > from float % float and integer % integer:  it confuses the meaning of
> > "%".  That's your complaint, right?
>
> Thanks for the clarification.  Yes, it makes sense that __mod__,
> __divmod__ and __floordiv__ on float and decimal would eventually follow
> the same path as for complex (where they make even less sense and
> already raise a DeprecationWarning).
>
>
> A bientot,
>
> Armin.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Thu Jan 25 22:27:45 2007
From: brett at python.org (Brett Cannon)
Date: Thu, 25 Jan 2007 13:27:45 -0800
Subject: [Python-Dev] Having trouble committing
Message-ID: <bbaeab100701251327ra2dd960jd4f68f99e246a03d@mail.gmail.com>

I am trying to commit to the 2.5 branch and I am getting an error:

svn: Commit failed (details follow):
svn: Can't create directory
'/data/repos/projects/db/transactions/53566-1.txn': Permission denied

Anyone know what is going on?

-Brett

From tim.peters at gmail.com  Thu Jan 25 23:45:33 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 25 Jan 2007 17:45:33 -0500
Subject: [Python-Dev] Having trouble committing
In-Reply-To: <bbaeab100701251327ra2dd960jd4f68f99e246a03d@mail.gmail.com>
References: <bbaeab100701251327ra2dd960jd4f68f99e246a03d@mail.gmail.com>
Message-ID: <1f7befae0701251445i8cc83bar4c0e7e83878b3a8d@mail.gmail.com>

[Brett Cannon]
> I am trying to commit to the 2.5 branch and I am getting an error:
>
> svn: Commit failed (details follow):
> svn: Can't create directory
> '/data/repos/projects/db/transactions/53566-1.txn': Permission denied
>
> Anyone know what is going on?

Did you do `svn info` in that directory to make sure you have a
writable ("svn+ssh") checkout of that part?

From martin at v.loewis.de  Thu Jan 25 23:45:50 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 25 Jan 2007 23:45:50 +0100
Subject: [Python-Dev] Having trouble committing
In-Reply-To: <bbaeab100701251327ra2dd960jd4f68f99e246a03d@mail.gmail.com>
References: <bbaeab100701251327ra2dd960jd4f68f99e246a03d@mail.gmail.com>
Message-ID: <45B9331E.8030109@v.loewis.de>

Brett Cannon schrieb:
> I am trying to commit to the 2.5 branch and I am getting an error:
> 
> svn: Commit failed (details follow):
> svn: Can't create directory
> '/data/repos/projects/db/transactions/53566-1.txn': Permission denied
> 
> Anyone know what is going on?

It's not clear to me. The permissions on the transactions folder look
good (pythondev has write permissions), so I can't see anything wrong.
Also, assuming that the error happened just before you sent the email,
I can't find anything suspicious in the log files around this time
(22:27 +0100)
Can you try again?

Regards,
Martin

From tim.peters at gmail.com  Fri Jan 26 00:11:23 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 25 Jan 2007 18:11:23 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <ca471dc20701251322l1475908ao1ceb68e2b8d92f3d@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<20070123160303.GA12004@code0.codespeak.net>
	<1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>
	<20070125095651.GA14446@code0.codespeak.net>
	<ca471dc20701251322l1475908ao1ceb68e2b8d92f3d@mail.gmail.com>
Message-ID: <1f7befae0701251511w4896c106xa585ab7b2c48cec@mail.gmail.com>

[Guido]
> The only thing I would miss about this is that I am used to write
> certain timing loops that like to sync on whole seconds, by taking
> time.time() % 1.0 which nicely gives me the milliseconds in the
> current second. E.g.
>
> while True:
>   do_something_expensive_once_a_second_on_the_second()
>   now = time.time()
>   time.sleep(1.0 - (now % 1.0))

Indeed, the only use of floating % in the standard library I recall is
the ancient

    return (x/30269.0 + y/30307.0 + z/30323.0) % 1.0

from the original Wichman-Hill random() generator.  Maybe we could
introduce "%" as a unary prefix operator, where %x means "the
fractional part of x" ;-)

Are you opposed to importing `math`?  The infix spelling had important
speed benefits in random.py (random() was the time-critical function
in that module), but the use above couldn't care less.

     time.sleep(1.0 - math.fmod(now, 1.0))

would do the same, except would be easier to reason about because it's
trivially guaranteed that 0.0 <= math.fmod(x, 1.0) < 1.0 for any
finite float x >= 0.0.  The same may or may not be true of % (I would
have to think about that, and craft a proof one way or the other -- if
it is true, it would have to invoke something special about the
modulus 1.0, as the inequality doesn't hold for % for some other
modulus values).

Better, you could use the more obvious:

    time.sleep(math.ceil(now) - now)

That "says" as directly as possible that you want the number of
seconds needed to reach the next integer value (or 0, if `now` is
already an integer).

> I guess I could use (now - int(now)) in a pinch,

That would need to be

    time.sleep(1.0 - (now - int(now)))

I'd use the `ceil` spelling myself, even today -- but I don't suffer
"it's better if it's syntax or __builtin__" disease ;-)

> assuming int() continues to truncate.

Has anyone suggested to change that?  I'm not aware of any complaints
or problems due to int() truncating.  There have been requests to add
new kinds of round-to-integer functions, but in addition to int().

From brett at python.org  Fri Jan 26 00:14:27 2007
From: brett at python.org (Brett Cannon)
Date: Thu, 25 Jan 2007 15:14:27 -0800
Subject: [Python-Dev] Having trouble committing
In-Reply-To: <1f7befae0701251445i8cc83bar4c0e7e83878b3a8d@mail.gmail.com>
References: <bbaeab100701251327ra2dd960jd4f68f99e246a03d@mail.gmail.com>
	<1f7befae0701251445i8cc83bar4c0e7e83878b3a8d@mail.gmail.com>
Message-ID: <bbaeab100701251514q1a483436w9541eae7de57bac@mail.gmail.com>

On 1/25/07, Tim Peters <tim.peters at gmail.com> wrote:
> [Brett Cannon]
> > I am trying to commit to the 2.5 branch and I am getting an error:
> >
> > svn: Commit failed (details follow):
> > svn: Can't create directory
> > '/data/repos/projects/db/transactions/53566-1.txn': Permission denied
> >
> > Anyone know what is going on?
>
> Did you do `svn info` in that directory to make sure you have a
> writable ("svn+ssh") checkout of that part?
>

Damn it, I think that was it.  I did a framework build in there and
``make distclean`` didn't clean up all the Mac-related files (which
should get fixed at some point), so I nuked the directory (which
itself had permission stuff that required sudo) and did a fresh
checkout.  Must have accidentally done a read-only checkout.

Sorry for the false alarm.

-Brett

From ndbecker2 at gmail.com  Fri Jan 26 01:01:51 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 25 Jan 2007 19:01:51 -0500
Subject: [Python-Dev] Having trouble committing
References: <bbaeab100701251327ra2dd960jd4f68f99e246a03d@mail.gmail.com>
Message-ID: <epbgdg$ae6$1@sea.gmane.org>

I've heard it claimed that men often have this problem.


From guido at python.org  Fri Jan 26 01:39:08 2007
From: guido at python.org (Guido van Rossum)
Date: Thu, 25 Jan 2007 16:39:08 -0800
Subject: [Python-Dev] Floor division
In-Reply-To: <1f7befae0701251511w4896c106xa585ab7b2c48cec@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<20070123160303.GA12004@code0.codespeak.net>
	<1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>
	<20070125095651.GA14446@code0.codespeak.net>
	<ca471dc20701251322l1475908ao1ceb68e2b8d92f3d@mail.gmail.com>
	<1f7befae0701251511w4896c106xa585ab7b2c48cec@mail.gmail.com>
Message-ID: <ca471dc20701251639h2ef2f93cxd22eed6c27b5da4e@mail.gmail.com>

On 1/25/07, Tim Peters <tim.peters at gmail.com> wrote:
> [Guido]
> > The only thing I would miss about this is that I am used to write
> > certain timing loops that like to sync on whole seconds, by taking
> > time.time() % 1.0 which nicely gives me the milliseconds in the
> > current second. E.g.
> >
> > while True:
> >   do_something_expensive_once_a_second_on_the_second()
> >   now = time.time()
> >   time.sleep(1.0 - (now % 1.0))
>
> Indeed, the only use of floating % in the standard library I recall is
> the ancient
>
>     return (x/30269.0 + y/30307.0 + z/30323.0) % 1.0
>
> from the original Wichman-Hill random() generator.  Maybe we could
> introduce "%" as a unary prefix operator, where %x means "the
> fractional part of x" ;-)
>
> Are you opposed to importing `math`?  The infix spelling had important
> speed benefits in random.py (random() was the time-critical function
> in that module), but the use above couldn't care less.
>
>      time.sleep(1.0 - math.fmod(now, 1.0))
>
> would do the same, except would be easier to reason about because it's
> trivially guaranteed that 0.0 <= math.fmod(x, 1.0) < 1.0 for any
> finite float x >= 0.0.  The same may or may not be true of % (I would
> have to think about that, and craft a proof one way or the other -- if
> it is true, it would have to invoke something special about the
> modulus 1.0, as the inequality doesn't hold for % for some other
> modulus values).

I don't care about the speed, but having to import math (which I
otherwise almost never need) is a distraction, and (perhaps more so) I
can never remember whether it's modf() or fmod() that I want.

> Better, you could use the more obvious:
>
>     time.sleep(math.ceil(now) - now)
>
> That "says" as directly as possible that you want the number of
> seconds needed to reach the next integer value (or 0, if `now` is
> already an integer).

Yeah, once math is imported the possibilities are endless. :-)

> > I guess I could use (now - int(now)) in a pinch,
>
> That would need to be
>
>     time.sleep(1.0 - (now - int(now)))
>
> I'd use the `ceil` spelling myself, even today -- but I don't suffer
> "it's better if it's syntax or __builtin__" disease ;-)
>
> > assuming int() continues to truncate.
>
> Has anyone suggested to change that?  I'm not aware of any complaints
> or problems due to int() truncating.  There have been requests to add
> new kinds of round-to-integer functions, but in addition to int().

I thought those semantics were kind of poorly specified. But maybe
that was long ago (when int() just did whatever (int) did in C) and
it's part of the language now.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.peters at gmail.com  Fri Jan 26 02:15:13 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 25 Jan 2007 20:15:13 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <ca471dc20701251639h2ef2f93cxd22eed6c27b5da4e@mail.gmail.com>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<20070123160303.GA12004@code0.codespeak.net>
	<1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>
	<20070125095651.GA14446@code0.codespeak.net>
	<ca471dc20701251322l1475908ao1ceb68e2b8d92f3d@mail.gmail.com>
	<1f7befae0701251511w4896c106xa585ab7b2c48cec@mail.gmail.com>
	<ca471dc20701251639h2ef2f93cxd22eed6c27b5da4e@mail.gmail.com>
Message-ID: <1f7befae0701251715r6e1617bm9a9bac607c22679e@mail.gmail.com>

[Guido]
> ...
> I don't care about the speed, but having to import math (which I
> otherwise almost never need) is a distraction, and (perhaps more so) I
> can never remember whether it's modf() or fmod() that I want.

fractional part of x == fmod(x, 1.0) == modf(x)[0], so you could use
either.  Since modf returns a tuple and fmod returns a float, you'll
get an exception quickly if you pick the wrong one :-)  The name
"modf" certainly sucks.

...

>>> assuming int() continues to truncate.

>> Has anyone suggested to change that?  I'm not aware of any complaints
>> or problems due to int() truncating.  There have been requests to add
>> new kinds of round-to-integer functions, but in addition to int().

> I thought those semantics were kind of poorly specified. But maybe
> that was long ago (when int() just did whatever (int) did in C) and
> it's part of the language now.

"(int)float_or_double" truncates in C (even in K&R C) /provided that/
the true result is representable as an int.  Else behavior is
undefined (may return -1, may cause a HW fault, ...).

So Python uses C's modf() for float->int now, which is always defined
for finite floats, and also truncates.

From tim.peters at gmail.com  Fri Jan 26 03:05:47 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 25 Jan 2007 21:05:47 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <20070125095651.GA14446@code0.codespeak.net>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<20070123160303.GA12004@code0.codespeak.net>
	<1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>
	<20070125095651.GA14446@code0.codespeak.net>
Message-ID: <1f7befae0701251805q36200a95s8a15da5a1d060e99@mail.gmail.com>

[Armin]
> Thanks for the clarification.  Yes, it makes sense that __mod__,
> __divmod__ and __floordiv__ on float and decimal would eventually follow
> the same path as for complex (where they make even less sense and
> already raise a DeprecationWarning).

This truly has nothing to do with complex.  All meanings for "mod"
(whether in Python, IEEE-754, C89, C99, or IBM's proposed decimal
standard) are instances of this mathematical schema:

[1]    x%y = x - R(x/y)*y

by definition, where R(z) is a specific way of rounding z to an exact
mathematical ("infinite precision") integer.

For int, long, and float, Python uses R=floor (and again it's
important to note that these are mathematical statements, not
statements about computer arithmetic).

For ints, longs, and (mathematical) reals, that's the usual
"number-theoretic" definition of mod, as, e.g., given by Knuth:

    mod(x, y) = x - floor(x/y)*y

It's the only definition of all those mentioned here that guarantees
the result is non-negative when the modulus (y) is positive, and
that's a very nice property for integers.  It's /also/ the only
definition off all those mentioned here where the exact mathematical
result may /not/ be exactly representable as a computer float when x
and y are computer floats.

It's that last point that makes it a poor definition for working with
computer floats:  for any other plausible way of defining "mod", the
exact result /is/ exactly representable as a computer float.  That
makes reasoning much easier, just as you don't have to think twice
about seeing abs() or unary minus applied to a float.  No information
is lost, and you can rely on expected invariants like

[2]    0 <= abs(x%y) < abs(y)        if y != 0 and finite

provided one of the non- R=floor definitions of mod is used for computer floats.

For complex, Python uses R(z) = floor(real_part_of(z)).  AFAICT,
Python just made that up out of thin air.  There are no known use
cases, and it's bizarre.  For example, [2] isn't even approximately
reliable:

    >>> x = 5000 + 100j
    >>> y = 1j
    >>> x % y
    (5000+0j)
    >>> print abs(x%y), abs(y)
    5000.0 1.0

In short, while Python "does something" for complex % complex, what it
does seems more-than-less arbitrary.  That's why it was deprecated
years ago, and nobody complained.

But for computer floats, there are ways to instantiate R in [1] that
work fine, returning a result that is truly (exactly) congruent to x
modulo y, even though x, y and the result are all computer floats.
Two of those ways:

The C89 fmod = C99 fmod = C99 integral "%" = IBM spec "remainder"
picks R(z) = round z to the closest integer in the direction of 0.

The C99 "remainder" = IBM spec "remainder-near" = IEEE-754 REM picks
R(z) = round z to the nearest integer, or if z is exactly halfway
between integers to the nearest even integer.  This one has the nice
property (for floats!) that [2] can be strengthened to:

    0 <= abs(x%y) <= abs(y)/2

That's often useful for argument reduction of periodic functions,
which is an important use case for a floating-point mod.  You
typically don't care about the sign of the result in that case, but do
want the absolute value as small as possible.  That's probably why
this was the only definition standardized by 754.

In contrast, I don't know of any use for complex %.

From tim.peters at gmail.com  Fri Jan 26 03:38:58 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 25 Jan 2007 21:38:58 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <45B95F38.7000201@jmunch.dk>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<20070123160303.GA12004@code0.codespeak.net>
	<1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>
	<20070125095651.GA14446@code0.codespeak.net>
	<ca471dc20701251322l1475908ao1ceb68e2b8d92f3d@mail.gmail.com>
	<1f7befae0701251511w4896c106xa585ab7b2c48cec@mail.gmail.com>
	<ca471dc20701251639h2ef2f93cxd22eed6c27b5da4e@mail.gmail.com>
	<1f7befae0701251715r6e1617bm9a9bac607c22679e@mail.gmail.com>
	<45B95F38.7000201@jmunch.dk>
Message-ID: <1f7befae0701251838p39b14bdayca167267210f3ad1@mail.gmail.com>

...

[Tim]
>> fractional part of x == fmod(x, 1.0) == modf(x)[0], so you could use
>> either.

[Anders J. Munch]
> Actually, on the off chance that time.time() is negative, he can use
> neither.  It has to be math.ceil, float.__mod__ or divmod.

If time.time() is negative, I expect this would be the least of his worries :-)

Even on most Unixish boxes, Python's time.time() is immune to "the
year 2038 problem" anyway, since it uses POSIX's gettimeofday()
instead of C's time() when it can.

From tim.peters at gmail.com  Fri Jan 26 04:28:03 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 25 Jan 2007 22:28:03 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <E1HA1mY-00059L-93@virgo.cus.cam.ac.uk>
References: <E1HA1mY-00059L-93@virgo.cus.cam.ac.uk>
Message-ID: <1f7befae0701251928j75097945jb89cc81bb8a4f03f@mail.gmail.com>

[Armin Rigo]
>> Thanks for the clarification.  Yes, it makes sense that __mod__,
>> __divmod__ and __floordiv__ on float and decimal would eventually follow
>> the same path as for complex (where they make even less sense and
>> already raise a DeprecationWarning).

[Nick Maclaren]
> Yes.  Though them not doing so would also make sense.  The difference
> is that they make no mathematical sense for complex, but the problems
> with float are caused by floating-point (and do not occur for the
> mathematical reals).
>
> There is an argument for saying that divmod should return a long
> quotient and a float remainder,

It could, but who would have a (sane) use for a possibly 2000-bit quotient?

> which is what C99 has specified for remquo (except that it requires
> only the last 3 bits of the quotient for reasons that completely baffle
> me).

C99 has no integral type capable of representing the full quotient,
but the last few bits may suffice for performing argument reduction in
the implementation of a periodic function.  Surely you did that for
trig functions in the bad old days?  For example, express input x as
N*(pi/2) + r, where |r| <= pi/4.  Then N can be expressed as 4*n1 +
n2, with n2 in [0, 1, 2, 3], and:

    cos(x) =
    cos((4*n1+n2)*(pi/2) + r) =
    cos(n1*(2*pi) + n2*pi/2 + r) =  [can ignore integral multiples of 2*pi]
    cos(n2*pi/2 + r)

Then the combination of n2 and the sign of r tell you which quadrant
you're in, and various cos-specific rules can deliver the result from
that and cos(r) or sin(r).

The point is that only the last 2 bits of the quotient are needed to
determine n2, and all other bits in N are irrelevant to the result.

> Linux misimplemented that the last time I looked.
>
> Personally, I think that it is bonkers, as it is fiendishly expensive
> compared to its usefulness - especially with Decimal!

Ah, but the IBM spec weasels out:  it raises an exception if you try
to compute "remainder" or "remainder-near" of inputs when the
exponents differ by "too much".

This is a bit peculiar to me, because there are ways to compute
"remainder" using a number of operations proportional to the log of
the exponent difference.  It could be that people who spend their life
doing floating point forget how to work with integers ;-)

For example, what about 9e99999 % 3.14?

    9e99999 = q*3.14 + r

if and only if (multiplying both sides by 100)

    9e100001 = 314*q + 100*r

So what's mod(9 * 10**100001, 314)?  Binary-method modular
exponentiation goes fast:

>>> pow(10, 100001, 314)
148
>>> _ * 9 % 314
76

So

9e100001 = 314*q + 76

exactly for some integer q, so (dividing both sides by 100)

9e99999 = 3.14*q + 0.76

exactly for the same integer q.  Done.  It doesn't require 100000
long-division steps, it only requires about two dozen modular
multiplications wrt the relatively tiny modulus 314.

OTOH, I don't know how to get the last bit of `q` with comparable
efficiency, and that's required to implement the related
"remainder-near" in "halfway cases".

> But it isn't obviously WRONG.

For floats, fmod(x, y) is exactly congruent to x modulo y -- I don't
think it's possible to get more right than exactly right ;-)

From tim.peters at gmail.com  Fri Jan 26 07:16:22 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 26 Jan 2007 01:16:22 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <45B958CF.20208@jmunch.dk>
References: <00a901c73c29$e23183d0$ea146b0a@RaymondLaptop1>
	<1f7befae0701191733l4626352dhc228dc73a4ee6b23@mail.gmail.com>
	<20070120112933.GA11566@code0.codespeak.net>
	<1f7befae0701211808p1e4648c0xd99d7c3a9583ff85@mail.gmail.com>
	<20070123160303.GA12004@code0.codespeak.net>
	<1f7befae0701231414p1fede3a3k327b976c3e5c3e3d@mail.gmail.com>
	<20070125095651.GA14446@code0.codespeak.net>
	<ca471dc20701251322l1475908ao1ceb68e2b8d92f3d@mail.gmail.com>
	<1f7befae0701251511w4896c106xa585ab7b2c48cec@mail.gmail.com>
	<45B958CF.20208@jmunch.dk>
Message-ID: <1f7befae0701252216s3c035f1av1479eff174aa867e@mail.gmail.com>

[Tim Peters]
>> ...
>> Maybe we could introduce "%" as a unary prefix operator, where
>> %x means "the fractional part of x" ;-)

[Anders J. Munch]
> What'ya talking about?  Obviously it should be a suffix operator ;-)

Na -- that would be confusing ;-)

...

>>      time.sleep(1.0 - math.fmod(now, 1.0))
>>
>> would do the same, except would be easier to reason about because it's
>> trivially guaranteed that 0.0 <= math.fmod(x, 1.0) < 1.0 for any
>> finite float x >= 0.0.  The same may or may not be true of % (I would
>> have to think about that, and craft a proof one way or the other -- if
>> it is true, it would have to invoke something special about the
>> modulus 1.0, as the inequality doesn't hold for % for some other
>> modulus values).

And as you note later, x%y == fmod(x, y) whenever x and y have the
same sign (well, given the way CPython implements float.__mod__
today), so there's actually an easy proof.

> Other modulus values are important:

On an importance scale of 1 to 10, 9 or 10 ;-) ?

> The attraction of Guido's formula is that he could just as easily have
> used 60.0 or 0.001 if minute or millisecond intervals were desired, or
> even som user-specified arbitrary dt.  Then we're comparing dt-now%dt
> to (1.0-int(now/dt))*dt or (math.ceil(now/dt)-now/dt)*dt.

time.time() is never negative in Python (see other reply), so the
trivial respelling dt-fmod(now, dt) does the same.

> Fortunately, for all a,b>0, mathematically math.fmod(a,b) is equal to
> a%b, so if the former is exactly representable, so is the latter.

Yup.  Also when `a` and `b` both less than 0.  This /follows/ from
that when `a` and `b` have the same sign, the mathematical a/b is >=
0, so truncation is the same as the floor.  Therefore the mathematical

    a - floor(a/b)*b   # Python __mod__
and
    a - truncate(a/b)*b  # C fmod

are exactly the same whenever a and b have the same sign.

> Which is borne out in floatobject.c: float_rem and float_divmod just
> pass on the C fmod result if (a < 0) == (b < 0).

Yes.  In fact, I wrote all that code :-)

From nmm1 at cus.cam.ac.uk  Fri Jan 26 10:19:23 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 09:19:23 +0000
Subject: [Python-Dev] Floor division
Message-ID: <E1HANEx-0007Bo-3f@draco.cus.cam.ac.uk>

"Guido van Rossum" <guido at python.org> wrote:
>
> "(int)float_or_double" truncates in C (even in K&R C) /provided that/
> the true result is representable as an int.  Else behavior is
> undefined (may return -1, may cause a HW fault, ...).

Actually, I have used Cs that didn't, but haven't seen any in over
10 years.  C90 is unclear about its intent, but C99 is specific that
truncation is towards zero.  This is safe, at least for now.

> So Python uses C's modf() for float->int now, which is always defined
> for finite floats, and also truncates.

Yes.  And that is clearly documented and not currently likely to
change, as far as I know.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Fri Jan 26 10:58:20 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 09:58:20 +0000
Subject: [Python-Dev] Floor division
Message-ID: <E1HANqe-0007c1-NL@draco.cus.cam.ac.uk>

"Tim Peters" <tim.peters at gmail.com> wrote:
>
> It could, but who would have a (sane) use for a possibly 2000-bit quotient?

Well, the 'exact rounding' camp in IEEE 754 seem to think that there
is one :-)

As you can gather, I can't think of one.  Floating-point is an inherently
inaccurate representation for anything other than small integers.

> This is a bit peculiar to me, because there are ways to compute
> "remainder" using a number of operations proportional to the log of
> the exponent difference.  It could be that people who spend their life
> doing floating point forget how to work with integers ;-)

Aargh!  That is indeed the key!  Given that I claim to know something
about integer arithmetic, too, how can I have been so STUPID?  Yes,
you are right, and that is the only plausible way to calculate the
remainder precisely.  You don't get the quotient precisely, which is
what my (insane) specification would have provided.

I would nitpick with your example, because you don't want to reduce
modulo 3.14 but modulo pi and therefore the modular arithmetic is
rather more expensive (given Decimal).  However, it STILL doesn't
help to make remquo useful!

The reason is that pi is input only to the floating-point precision,
and so the result of remquo for very large arguments will depend
more on the inaccuracy of pi as input than on the mathematical
result.  That makes remquo totally useless for the example you quote.

Yes, I have implemented 'precise' range reduction, and there is no
substitute for using an arbitrary precision pi value :-(

> > But it isn't obviously WRONG.
>
> For floats, fmod(x, y) is exactly congruent to x modulo y -- I don't
> think it's possible to get more right than exactly right ;-)

But, as a previous example of yours pointed out, it's NOT exactly
right.  It is also supposed to be in the range [0,y) and it isn't.
-1%1e100 is mathematically wrong on two counts.
1a
Cc: "Tim Peters" <tim.peters at gmail.com>


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From tim.peters at gmail.com  Fri Jan 26 11:03:44 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 26 Jan 2007 05:03:44 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <E1HANEx-0007Bo-3f@draco.cus.cam.ac.uk>
References: <E1HANEx-0007Bo-3f@draco.cus.cam.ac.uk>
Message-ID: <1f7befae0701260203n5c1640b0re993e6431730b294@mail.gmail.com>

[Tim (misattributed to Guido)]
>> "(int)float_or_double" truncates in C (even in K&R C) /provided that/
>> the true result is representable as an int.  Else behavior is
>> undefined (may return -1, may cause a HW fault, ...).

[Nick Maclaren]
> Actually, I have used Cs that didn't, but haven't seen any in over
> 10 years.

I believe those.

> C90 is unclear about its intent,

But am skeptical of that.  I don't have a copy of C90 here, but before
I wrote that I checked Kernighan & Ritchie's seminal C book, Harbison
& Steele's generally excellent "C: A Reference Manual" (2nd ed), and a
web version of Plauger & Brodie's "Standard C":

     http://www-ccs.ucsd.edu/c/

They all agree that the Cs they describe (all of which predate C99)
convert floating to integral types via truncation, when possible.

> but C99 is specific that truncation is towards zero.

As opposed to what?  Truncation away from zero?  I read "truncation"
as implying toward 0, although the Plauger & Brodie source is explicit
about "the integer part of X, truncated toward zero" for the sake of
logic choppers ;-)

> This is safe, at least for now.

>> So Python uses C's modf() for float->int now, which is always defined
>> for finite floats, and also truncates.

> Yes.  And that is clearly documented and not currently likely to
> change, as far as I know.

I don't expect to see another C standard in my lifetime, given that
some major C compiler vendors still ignore C99 (and given that my
expected remaining lifetime is much less than that of most people
reading this ;-)).

From nmm1 at cus.cam.ac.uk  Fri Jan 26 11:26:18 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 10:26:18 +0000
Subject: [Python-Dev] Floor division
Message-ID: <E1HAOHi-0000Ew-CR@draco.cus.cam.ac.uk>

"Tim Peters" <tim.peters at gmail.com> wrote:
>
> [Tim (misattributed to Guido)]

Apologies to both!

> > C90 is unclear about its intent,
>
> But am skeptical of that.  I don't have a copy of C90 here, but before
> I wrote that I checked Kernighan & Ritchie's seminal C book, Harbison
> & Steele's generally excellent "C: A Reference Manual" (2nd ed), and a
> web version of Plauger & Brodie's "Standard C":
>
>      http://www-ccs.ucsd.edu/c/
> 
> They all agree that the Cs they describe (all of which predate C99)
> convert floating to integral types via truncation, when possible.

I do.  Kernighan & Ritchie's seminal C book describes the Unix style
of "K&R" C - one of the reasons that ANSI/ISO had to make incompatible
changes was that many important PC and embedded Cs differed.  Harbison
and Steele is generally reliable, but not always; I haven't looked at
the last, but I would regard it suspiciously.

What C90 says is:

    When a value of floating type is converted to integer type, the
    fractional part is discarded.

There is other wording, but none relevant to this issue.  Now, given
the history of floating-point remainder, that is seriously ambiguous.

> > but C99 is specific that truncation is towards zero.
>
> As opposed to what?  Truncation away from zero?  I read "truncation"
> as implying toward 0, although the Plauger & Brodie source is explicit
> about "the integer part of X, truncated toward zero" for the sake of
> logic choppers ;-)

Towards -infinity, of course.  That was as common as truncation towards
zero up until the 1980s.  It was near-universal on twos complement
floating-point systems, and not rare on signed magnitude ones.  During
the standardisation of C90, the BSI tried to explain to ANSI that this
needed spelling out, but were ignored.  C99 did not add the normative
text "(i.e., the value is truncated toward zero)" because there was
no ambiguity, after all!


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From tim.peters at gmail.com  Fri Jan 26 12:04:02 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 26 Jan 2007 06:04:02 -0500
Subject: [Python-Dev] Floor division
In-Reply-To: <E1HANqe-0007c1-NL@draco.cus.cam.ac.uk>
References: <E1HANqe-0007c1-NL@draco.cus.cam.ac.uk>
Message-ID: <1f7befae0701260304r3573dcc9wfe2d5fae01660443@mail.gmail.com>

[Tim Peters]
>> ...
>> [it would be possible for float.__divmod__ to return the exact
>> quotient], but who would have a (sane) use for a possibly 2000-bit
>> quotient?

[Nick Maclaren]
> Well, the 'exact rounding' camp in IEEE 754 seem to think that there
> is one :-)
>
> As you can gather, I can't think of one.  Floating-point is an inherently
> inaccurate representation for anything other than small integers.

Well, bounded.  Python's decimal fp can handle millions of digits, if
you want that and are very patient ;-)

OTOH, I am a fan of analyzing FP operations as if the inputs were in
fact exactly what they claim to be, which 754 went a long way toward
popularizing.  That largely replaced mountains of idiosyncratic
"probabilistic arguments" (and where it seemed no two debaters ever
agreed on the "proper" approach)  with a common approach that
sometimes allows surprisingly sharp analysis.  Since I spent a good
part of my early career as a professional apologist for Seymour Cray's
"creative" floating point, I'm probably much more grateful to leave
sloppy arithmetic behind than most.

...

>> This [that IBM's spec calls for an exception if remainder's inputs'
>> exponents differ by "too much"] is a bit peculiar to me, because
>> there are ways to compute "remainder" using a number of operations
>> proportional to the log of the exponent difference.  It could be that
>> people who spend their life doing floating point forget how to work
>> with integers ;-)

> Aargh!  That is indeed the key!  Given that I claim to know something
> about integer arithmetic, too, how can I have been so STUPID?

You're not alone.  I thought of this a decade ago, but have never seen
it implemented, or even mentioned.  All implementations of fmod I've
seen emulate long binary division one bit at a time; the IBM spec
seems to assume that's how it would be done for decimal too (why else
mandate an exception instead if the exponent delta "is large"?); and
so on.

> Yes, you are right, and that is the only plausible way to calculate the
> remainder precisely.  You don't get the quotient precisely, which is
> what my (insane) specification would have provided.

And, alas, without the last bit of the quotient IBM's "remainder-near"
(== C99's "remainder" == 754's REM) can't resolve halfway cases in the
mandated way.

> I would nitpick with your example, because you don't want to reduce
> modulo 3.14 but modulo pi

I didn't have pi in mind at all.  I just picked 3.14 as an arbitrary
decimal modulus with few digits, to make the example easy to follow.
Could just as well have been 2.72 (which has nothing to do with e
either ;-)).

> and therefore the modular arithmetic is rather more expensive (given
> Decimal).  However, it STILL doesn't help to make remquo useful!
>
> The reason is that pi is input only to the floating-point precision,
> and so the result of remquo for very large arguments will depend
> more on the inaccuracy of pi as input than on the mathematical
> result.  That makes remquo totally useless for the example you quote.

That's not our choice to make.  Many math libraries still use a
"small" approximation to pi for trig argument reduction, and that's
their choice.  Heck, a 66-bit value for pi is built in to the
Pentium's FSIN and FCOS instructions.

>>> math.sin(math.pi)
1.2246063538223773e-016

That was on Windows with Python 2.5, using the MSVC compiler.  The
result indicates it uses the FSIN instruction.  Same thing but under
the Cygwin Python, whose libm uses "infinite precision pi" trig
reduction:

>>> math.sin(math.pi)
1.2246467991473532e-16

That one is "correct" (close to the best double approximation to the
sine of the best double approximation to pi).  The FSIN result is off
by about 164 billion(!) ULP, but few people care.

Anyway, I simply gave cosine arg reduction as /an example/ of the kind
of reduction strategy for which remquo can be useful.  You said you
were baffled by why C99 only required the last 3 bits.  The example
was a hint about why some functions can indeed find that to be useful,
not an exhaustive rationale.  It's really off-topic for Python-Dev, so
I didn't/don't want to belabor it.

> Yes, I have implemented 'precise' range reduction, and there is no
> substitute for using an arbitrary precision pi value :-(

I have too (for 754 doubles), coincidentally at the same time KC Ng
was implementing it for fdlibm.  I found some bugs in fdlibm's trig
functions at the time by generating billions of random inputs and
comparing his library-in-progress's results to mine.  That was fun.
My users (this was done for Kendall Square Research's libm) only
noticed that "the new" trig functions were much faster than the old
ones for small arguments -- nobody seemed to notice that the results
were much better than the old ones, or arrived at much more slowly,
for very large arguments.  How satisfying ;-)

>>> But it isn't obviously WRONG.

>> For floats, fmod(x, y) is exactly congruent to x modulo y -- I don't
>> think it's possible to get more right than exactly right ;-)

> But, as a previous example of yours pointed out, it's NOT exactly
> right.  It is also supposed to be in the range [0,y) and it isn't.
> -1%1e100 is mathematically wrong on two counts.
> 1a

No, /Python's/ definition of mod is inexact for that example.  fmod
(which is not Python's definition) is always exact:  fmod(-1, 1e100) =
-1, and -1 is trivially exactly congruent to -1 modulo anything
(including modulo 1e100).  The result of fmod(x, y) has the same sign
as x; Python's x.__mod__(y) has the same sign as y; and that makes all
the difference in the world as to whether the exact result is always
exactly representable as a float.

From nmm1 at cus.cam.ac.uk  Fri Jan 26 12:40:59 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 11:40:59 +0000
Subject: [Python-Dev] Floor division
Message-ID: <E1HAPRz-00014r-EM@draco.cus.cam.ac.uk>

"Tim Peters" <tim.peters at gmail.com> wrote:
>
> OTOH, I am a fan of analyzing FP operations as if the inputs were in
> fact exactly what they claim to be, which 754 went a long way toward
> popularizing.  That largely replaced mountains of idiosyncratic
> "probabilistic arguments" (and where it seemed no two debaters ever
> agreed on the "proper" approach)  with a common approach that
> sometimes allows surprisingly sharp analysis.  Since I spent a good
> part of my early career as a professional apologist for Seymour Cray's
> "creative" floating point, I'm probably much more grateful to leave
> sloppy arithmetic behind than most.

Well, I spent some of it working with code (and writing code) that
was expected to work, unchanged, on an ICL 1900, CDC 6600/7600,
IBM 370 and others.  I have seen the harm caused by the 'exact
arithmetic' mindset and so don't like it, but I agree about your
objections to the "probabilistic arguments" (which were and are
mostly twaddle).  But that is seriously off-topic.

> [remquo]  It's really off-topic for Python-Dev, so
> I didn't/don't want to belabor it.

Agreed, except in one respect.  I stand by my opinion that the C99
specification has no known PRACTICAL use (your example is correct,
but I know of no such use in a real application), and so PLEASE
don't copy it as a model for Python divmod/remainder.

> No, /Python's/ definition of mod is inexact for that example.  fmod
> (which is not Python's definition) is always exact:  fmod(-1, 1e100) =
> -1, and -1 is trivially exactly congruent to -1 modulo anything
> (including modulo 1e100).  The result of fmod(x, y) has the same sign
> as x; Python's x.__mod__(y) has the same sign as y; and that makes all
> the difference in the world as to whether the exact result is always
> exactly representable as a float.

Oops.  You're right, of course.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Fri Jan 26 14:51:04 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 13:51:04 +0000
Subject: [Python-Dev] Python's C interface for types
Message-ID: <E1HARTs-0002Ly-1H@draco.cus.cam.ac.uk>

I have a fair amount of my binary floating-point model written,
though even of what I have done only some is debugged (and none
has been rigorously tested).  But I have hit some things that I
can't work out, and one query reduced comp.lang.python to a
stunned silence :-)

Note that I am not intending to do all the following, at least for
now, but I have had to restructure half a dozen times to match my
implementation requirements to the C interface (as I have learnt
more about Python!) and designing to avoid that is always good.

Any pointers appreciated.

I can't find any detailed description of the methods that I need
to provide.  Specifically:

Does Python use classic division (nb_divide) and inversion (nb_invert)
or are they entirely historical?  Note that I can very easily provide
the latter.

Is there any documentation on the coercion function (nb_coerce)?  It
seems to have unusual properties.

How critical is the 'numeric' property of the nb_hash function?  I
can certainly honour it, but is it worth it?

I assume that Python will call nb_richcompare if defined and 
nb_compare if not.  Is that right?

Are the inplace methods used and, if so, what is their specification?

I assume that I can ignore all of the allocation, deallocation and
attribute handling functions, as the default for a VAR object is
fine.  That seems to work.

Except for one thing!  My base type is static, but I create some
space for every derivation (and it can ONLY be used in derived form).
The space creation is donein C but the derivation in Python.  I
assume that I need a class (not instance) destructor, but what
should it do to free the space?  Call C to Py_DECREF it?

I assume that a class structure will never go away until after all
instances have gone away (unless I use Py_DECREF), so a C pointer
from an instance to something owned by the class is OK.

Is there any documentation on how to support marshalling/pickling
and the converse from C types?

I would quite like to provide some attributes.  They are 'simple'
but need code executing to return them.  I assume that means that
they aren't simple enough, and have to be provided as methods
(like conjugate).  That's what I have done, anyway.

Is there any obvious place for a reduction method to be hooked in?
That is a method that takes a sequence, all members of which must
be convertible to a single class, and returns a member of that
class.  Note that it specifically does NOT make sense on a single
value of that class.

Sorry about the length of this!


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From guido at python.org  Fri Jan 26 16:01:44 2007
From: guido at python.org (Guido van Rossum)
Date: Fri, 26 Jan 2007 07:01:44 -0800
Subject: [Python-Dev] Floor division
In-Reply-To: <E1HANEx-0007Bo-3f@draco.cus.cam.ac.uk>
References: <E1HANEx-0007Bo-3f@draco.cus.cam.ac.uk>
Message-ID: <ca471dc20701260701x34902488g695b422c33f4d941@mail.gmail.com>

Nick, I didn't write that; Tim did. If you're going to enter into a
pedantic discussion, at least get your attributions right,

On 1/26/07, Nick Maclaren <nmm1 at cus.cam.ac.uk> wrote:
> "Guido van Rossum" <guido at python.org> wrote:
> >
> > "(int)float_or_double" truncates in C (even in K&R C) /provided that/
> > the true result is representable as an int.  Else behavior is
> > undefined (may return -1, may cause a HW fault, ...).
>
> Actually, I have used Cs that didn't, but haven't seen any in over
> 10 years.  C90 is unclear about its intent, but C99 is specific that
> truncation is towards zero.  This is safe, at least for now.
>
> > So Python uses C's modf() for float->int now, which is always defined
> > for finite floats, and also truncates.
>
> Yes.  And that is clearly documented and not currently likely to
> change, as far as I know.
>
>
> Regards,
> Nick Maclaren,
> University of Cambridge Computing Service,
> New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
> Email:  nmm1 at cam.ac.uk
> Tel.:  +44 1223 334761    Fax:  +44 1223 334679
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From thomas at python.org  Fri Jan 26 17:03:57 2007
From: thomas at python.org (Thomas Wouters)
Date: Fri, 26 Jan 2007 17:03:57 +0100
Subject: [Python-Dev] Python's C interface for types
In-Reply-To: <E1HARTs-0002Ly-1H@draco.cus.cam.ac.uk>
References: <E1HARTs-0002Ly-1H@draco.cus.cam.ac.uk>
Message-ID: <9e804ac0701260803n4d5b2455t2b366d97e7df824@mail.gmail.com>

On 1/26/07, Nick Maclaren <nmm1 at cus.cam.ac.uk> wrote:

> Does Python use classic division (nb_divide) and inversion (nb_invert)
> or are they entirely historical?  Note that I can very easily provide
> the latter.


nb_divide is used for the division operation (/) when not using 'from
__future__ import division', and PyNumber_Divide(). It doesn't fall back to
foor_divide or true_divide.
nb_invert is used for bitwise inversion (~) and PyNumber_Invert(). It's not
historical, it's actual.

Is there any documentation on the coercion function (nb_coerce)?  It
> seems to have unusual properties.


I don't recall ever seeing useful documentation on coerce() and nb_coerce. I
suggest not to use it; it's gone in Python 3.0 anyway.

How critical is the 'numeric' property of the nb_hash function?  I
> can certainly honour it, but is it worth it?


Which numeric property? the fact that it returns a C long? Or that, for
natural numbers, it *seems* to return self? The former is quite important,
the latter not very. The important bit is that an object's hash() be equal
to the hash() of objects that it is considered equal to. That is to say, if
you have 'd = {5: "five"}' and you want 'f = yourtype(5); d[f]' to result in
"five", hash(f) must be equal to hash(5). The builtin floats do that.
There's no strict requirement that equal objects must have equal hashes, but
people using sets and dicts really appreciate it. ('f == 5 and f not in
set([5])' to be True would be confusing.)

I assume that Python will call nb_richcompare if defined and
> nb_compare if not.  Is that right?


Yes.

Are the inplace methods used and, if so, what is their specification?


(Hah, my specialty.) Inplace methods are used for the augmented-assignment
statements: '+=' and the like. They are free to modify 'self' or return a
new object. Python falls back to the normal, non-inplace operations if the
inplace methods are not defined. I assume your floating-point type is
immutable, so you won't have to implement them. (If the type is to be
mutable, don't forget to make them unhashable, by not defining nb_hash.)

I assume that I can ignore all of the allocation, deallocation and
> attribute handling functions, as the default for a VAR object is
> fine.  That seems to work.
>
> Except for one thing!  My base type is static, but I create some
> space for every derivation (and it can ONLY be used in derived form).
> The space creation is donein C but the derivation in Python.  I
> assume that I need a class (not instance) destructor, but what
> should it do to free the space?  Call C to Py_DECREF it?


Where do you allocate this space, and how do you allocate it? If it's space
you malloc() and store somewhere in the type struct, yecchh. You should not
just allocate stuff at the end of the type struct, as the type struct's
layout is not under your control (we actually extend the type struct as
needed, which is why newer features end up in less logical places at the end
of the struct ;) I would suggest using attributes of the type instead, with
the normal Python refcounting. That means the 'extra space' has to be an
actual Python object, though.


I assume that a class structure will never go away until after all
> instances have gone away (unless I use Py_DECREF), so a C pointer
> from an instance to something owned by the class is OK.


Correct.

Is there any documentation on how to support marshalling/pickling
> and the converse from C types?


I don't you can make your own type marshallable. For pickle it's more or
less the same as for Python types. The pickle docs (and maybe
http://www.python.org/dev/peps/pep-0307/) probably cover what you want to
know. You can also look at one of the complexer builtin types that support
pickling, like the datetime types.

I would quite like to provide some attributes.  They are 'simple'
> but need code executing to return them.  I assume that means that
> they aren't simple enough, and have to be provided as methods
> (like conjugate).  That's what I have done, anyway.


You can use PyGetSetDef to get 'easy' attributes with getters and setters.
http://docs.python.org/api/type-structs.html#l2h-1020

Is there any obvious place for a reduction method to be hooked in?
> That is a method that takes a sequence, all members of which must
> be convertible to a single class, and returns a member of that
> class.  Note that it specifically does NOT make sense on a single
> value of that class.


There's nothing I can think of that is a natural match for that in standard
Python methods. I would suggest just making it a classmethod.
(dict.fromkeysis a good example of a classmethod in C.)

As a final note: Python's source itself is a good source of answers and
examples. Almost all of the features of the Python API are used in a builtin
type or stdlib module, and for C source, Python's source is remarkably
readable ;-)

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070126/fe246156/attachment-0001.htm 

From nmm1 at cus.cam.ac.uk  Fri Jan 26 17:37:08 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 16:37:08 +0000
Subject: [Python-Dev] Python's C interface for types
In-Reply-To: Your message of "Fri, 26 Jan 2007 17:03:57 +0100."
	<9e804ac0701260803n4d5b2455t2b366d97e7df824@mail.gmail.com> 
Message-ID: <E1HAU4a-0004K2-Jv@draco.cus.cam.ac.uk>

Thanks very much!  That answers most things.  Yes, I had got many
of my answers from searching the source, but there is clearly some
history there, and it isn't always clear what is current.  Here are
a few responses to the areas of confusion:

> nb_invert is used for bitwise inversion (~) and PyNumber_Invert(). It's not
> historical, it's actual.

Ah!  So it's NOT 1/x!  No relevant to floating-point, then.

> I don't recall ever seeing useful documentation on coerce() and nb_coerce.
> I suggest not to use it; it's gone in Python 3.0 anyway.

Excellent!  Task completed :-)

> Which numeric property? the fact that it returns a C long? Or that, for
> natural numbers, it *seems* to return self? 

The latter.  hash(123) == hash(123.0) for example.  It is a real
pain for advanced formats.  Making it the same for things that compare
equal isn't a problem.

> [inplace ] I assume your floating-point type is
> immutable, so you won't have to implement them.

I haven't done anything special to flag it as such, but it is.

> Where do you allocate this space, and how do you allocate it? If it's space
> you malloc() and store somewhere in the type struct, yecchh. You should not
> just allocate stuff at the end of the type struct, as the type struct's
> layout is not under your control (we actually extend the type struct as
> needed, which is why newer features end up in less logical places at the end
> of the struct ;) I would suggest using attributes of the type instead, with
> the normal Python refcounting. That means the 'extra space' has to be an
> actual Python object, though.

PyMem_Malloc.  I can certainly make it an attribute, as the overhead
isn't large for a per-class object.  It is just a block of mutable
memory, opaque to the Python layer, and NOT containing any pointers!

> I don't you can make your own type marshallable. For pickle it's more or
> less the same as for Python types. The pickle docs (and maybe
> http://www.python.org/dev/peps/pep-0307/) probably cover what you want to
> know. You can also look at one of the complexer builtin types that support
> pickling, like the datetime types.

The only documentation I have found is how to do it in Python.  Is
that what you mean?  I will look at the datetime types.

> You can use PyGetSetDef to get 'easy' attributes with getters and setters.
> http://docs.python.org/api/type-structs.html#l2h-1020

I was put off by some of the warnings.  I will revisit it.

> There's nothing I can think of that is a natural match for that in standard
> Python methods. I would suggest just making it a classmethod.
> (dict.fromkeysis a good example of a classmethod in C.)

Thanks.  That is a useful reference.  Reductions are a problem in
many languages.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Fri Jan 26 19:55:58 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 18:55:58 +0000
Subject: [Python-Dev] Python's C interface for types
Message-ID: <E1HAWEw-0005q0-Mj@draco.cus.cam.ac.uk>

Oops.  Something else fairly major I forgot to ask.  Python long.
I can't find any clean way of converting to or from this, and
would much rather not build a knowledge of long's internals into
my code.  Going via text is, of course, possible - but is not very
efficient, even using hex/octal.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From rasky at develer.com  Fri Jan 26 20:27:14 2007
From: rasky at develer.com (Giovanni Bajo)
Date: Fri, 26 Jan 2007 20:27:14 +0100
Subject: [Python-Dev] Python's C interface for types
In-Reply-To: <9e804ac0701260803n4d5b2455t2b366d97e7df824@mail.gmail.com>
References: <E1HARTs-0002Ly-1H@draco.cus.cam.ac.uk>
	<9e804ac0701260803n4d5b2455t2b366d97e7df824@mail.gmail.com>
Message-ID: <epdkmj$k0g$1@sea.gmane.org>

On 26/01/2007 17.03, Thomas Wouters wrote:

>     How critical is the 'numeric' property of the nb_hash function?  I
>     can certainly honour it, but is it worth it?
> 
> [...]
> There's no strict requirement that 
> equal objects must have equal hashes, 

Uh? I thought that was the *only* strict requirement of hash. In fact the docs 
agree:

====================================================
__hash__( self)

Called for the key object for dictionary operations, and by the built-in 
function hash(). Should return a 32-bit integer usable as a hash value for 
dictionary operations. The only required property is that objects which 
compare equal have the same hash value; [...]
====================================================

I personally consider *very* important that hash(5.0) == hash(5) (and that 5.0 
== 5, of course).
-- 
Giovanni Bajo


From jcarlson at uci.edu  Fri Jan 26 20:49:27 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 26 Jan 2007 11:49:27 -0800
Subject: [Python-Dev] Python's C interface for types
In-Reply-To: <E1HAWEw-0005q0-Mj@draco.cus.cam.ac.uk>
References: <E1HAWEw-0005q0-Mj@draco.cus.cam.ac.uk>
Message-ID: <20070126114830.5A00.JCARLSON@uci.edu>


Nick Maclaren <nmm1 at cus.cam.ac.uk> wrote:
> 
> Oops.  Something else fairly major I forgot to ask.  Python long.
> I can't find any clean way of converting to or from this, and
> would much rather not build a knowledge of long's internals into
> my code.  Going via text is, of course, possible - but is not very
> efficient, even using hex/octal.

See _PyLong_FromByteArray and _PyLong_AsByteArray .

 - Josiah


From nmm1 at cus.cam.ac.uk  Fri Jan 26 22:25:04 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 21:25:04 +0000
Subject: [Python-Dev] Python's C interface for types
Message-ID: <E1HAYZE-0006mG-Im@draco.cus.cam.ac.uk>

Giovanni Bajo <rasky at develer.com> wrote:
>
> I personally consider *very* important that hash(5.0) == hash(5) (and
> that 5.0 == 5, of course).

It gets a bit problematic with floating-point, when you can have
different values "exactly 5.0" and "approximately 5.0".  IEEE 754
has signed zeroes.  And so it goes.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Fri Jan 26 22:26:01 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 21:26:01 +0000
Subject: [Python-Dev] Python's C interface for types
Message-ID: <E1HAYa9-0006mr-HR@draco.cus.cam.ac.uk>

Josiah Carlson <jcarlson at uci.edu> wrote:
>
> See _PyLong_FromByteArray and _PyLong_AsByteArray .

Oops!  Thanks very much.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From aahz at pythoncraft.com  Fri Jan 26 22:38:35 2007
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 26 Jan 2007 13:38:35 -0800
Subject: [Python-Dev] Python's C interface for types
In-Reply-To: <epdkmj$k0g$1@sea.gmane.org>
References: <E1HARTs-0002Ly-1H@draco.cus.cam.ac.uk>
	<9e804ac0701260803n4d5b2455t2b366d97e7df824@mail.gmail.com>
	<epdkmj$k0g$1@sea.gmane.org>
Message-ID: <20070126213835.GA25873@panix.com>

On Fri, Jan 26, 2007, Giovanni Bajo wrote:
> On 26/01/2007 17.03, Thomas Wouters wrote:
>> 
>> There's no strict requirement that equal objects must have equal     
>> hashes,                                                              
>
> Uh? I thought that was the *only* strict requirement of hash. In fact
> the docs agree:
>
> ====================================================
> __hash__( self)
> 
> Called for the key object for dictionary operations, and by the built-in 
> function hash(). Should return a 32-bit integer usable as a hash value for 
> dictionary operations. The only required property is that objects which 
> compare equal have the same hash value; [...]
> ====================================================

Possibly the docs should be updated, but this is only intended to apply
to objects of the same type.

> I personally consider *very* important that hash(5.0) == hash(5) (and
> that 5.0 == 5, of course).

Well, sure, but That's Different.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"I disrespectfully agree."  --SJM

From nmm1 at cus.cam.ac.uk  Fri Jan 26 23:41:57 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 26 Jan 2007 22:41:57 +0000
Subject: [Python-Dev] Python's C interface for types
Message-ID: <E1HAZld-00030T-Tu@libra.cus.cam.ac.uk>

Having looked into the answers a bit more deeply, I am afraid that
I am still a bit puzzled.

1) As I understand it, PyMem_Malloc won't cause trouble, but won't
be automatically freed, either, as it doesn't return a new reference.
I don't think that immediately following it by PyCObject_FromVoidPtr
(which is what I do) helps with that.  What I need is some standard
type that allows me to allocate an anonymous block of memory; yes,
I can define such a type, but that seems excessive.  Is there one?

2) _PyLong_FromByteArray and _PyLong_AsByteArray aren't in the API
and have no comments.  Does that mean that they are unstable, in the
sense that they may change behaviour in new versions of Python?
And will they be there in 3.0?

Thanks for any help, again.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From martin at v.loewis.de  Sat Jan 27 00:43:06 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 27 Jan 2007 00:43:06 +0100
Subject: [Python-Dev] Python's C interface for types
In-Reply-To: <E1HAYZE-0006mG-Im@draco.cus.cam.ac.uk>
References: <E1HAYZE-0006mG-Im@draco.cus.cam.ac.uk>
Message-ID: <45BA920A.8030609@v.loewis.de>

Nick Maclaren schrieb:
>> I personally consider *very* important that hash(5.0) == hash(5) (and
>> that 5.0 == 5, of course).
> 
> It gets a bit problematic with floating-point, when you can have
> different values "exactly 5.0" and "approximately 5.0".  IEEE 754
> has signed zeroes.  And so it goes.

[not sure what "And so it goes" means in English]

It may be a bit problematic to implement, but I think a clean
specification is possible. If a and b are numbers, and a==b,
then hash(a)==hash(b). I'm not sure whether "approximately 5.0"
equals 5 or not: if it does, it should hash the same as 5,
if it doesn't, it may or may not hash the same (whatever is
easier to implement).
For 0: hash(+0.0)==hash(-0.0)==hash(0)=hash(0L)=0

Regards,
Martin

From tim.peters at gmail.com  Sat Jan 27 01:03:34 2007
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 26 Jan 2007 19:03:34 -0500
Subject: [Python-Dev] Python's C interface for types
In-Reply-To: <E1HAZld-00030T-Tu@libra.cus.cam.ac.uk>
References: <E1HAZld-00030T-Tu@libra.cus.cam.ac.uk>
Message-ID: <1f7befae0701261603x81e623fg9a4b73fc87b833ac@mail.gmail.com>

[Nick Maclaren]
> ...
> 2) _PyLong_FromByteArray and _PyLong_AsByteArray aren't in
> the API

They're not in the public API, which is equivalent to that their names
begin with a leading underscore.  They're in the private API :-)

> and have no comments.

The behavior of these functions, including return value and error
conditions, is specified in longobject.h.

> Does that mean that they are unstable, in the sense that they may
> change behaviour in new versions of Python?

They /may/ change, but they won't (== only common sense guarantees
they won't change ;-)).

> And will they be there in 3.0?

Almost certainly so.

From greg.ewing at canterbury.ac.nz  Sat Jan 27 02:34:31 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 27 Jan 2007 14:34:31 +1300
Subject: [Python-Dev] Problem with signals in a single
	threaded	application
In-Reply-To: <45B70EE1.7080809@v.loewis.de>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
	<45B70EE1.7080809@v.loewis.de>
Message-ID: <45BAAC27.6080302@canterbury.ac.nz>

Martin v. L?wis wrote:

> Please
> try to come up with a patch (e.g. by putting a while(is_tripped) loop
> around the for loop).

That isn't going to fix it. What's needed is to somehow
atomically test and clear is_tripped at the beginning.
You can put a while loop around it as well if you want,
but it's not strictly necessary.

--
Greg


From greg.ewing at canterbury.ac.nz  Sat Jan 27 02:35:21 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 27 Jan 2007 14:35:21 +1300
Subject: [Python-Dev] Problem with signals in a single
	threaded	application
In-Reply-To: <E1H9gCf-00072A-1B@libra.cus.cam.ac.uk>
References: <E1H9gCf-00072A-1B@libra.cus.cam.ac.uk>
Message-ID: <45BAAC59.8020005@canterbury.ac.nz>

Nick Maclaren wrote:

> This one looks like an oversight in Python code, and so is a bug,
> but it is important to note that signals do NOT work reliably under
> any Unix or Microsoft system.

That's a rather pessimistic way of putting it. In my
experience, signals in Unix mostly do what they're
meant to do quite reliably -- it's just a matter of
understanding what they're meant to do.

There may be bugs in certain systems that cause
signals to get lost under obscure circumstances, but
that's no reason for Python to make the situation
worse by introducing bugs of its own.

> Two related signals received between two 'checkpoints' (i.e. when
> the signal is tested and cleared).  You may only get one of them,
> and 'related' does not mean 'the same'.

In the case where they're the same, I wouldn't
say that the second signal has been "lost". Rather,
it's simply redundant -- a call to the handler is
already pending, and will happen eventually. (If
you're expecting one call per signal, then you're
using the wrong mental model for signals.)

I wasn't aware that this could happen between
different signals. If it can, there must be some
rationale as to why the second signal is considered
redundant. Otherwise there's a bug in either the
design or the implementation.

> A second signal received while the first is being 'handled' by the
> operating system or language run-time system.

That one sounds odd to me. I would expect a signal
received during the execution of a handler to be
flagged and cause the handler to be called again
after it returns. But then I'm used to the BSD
signal model, which is relatively sane.

> A signal sent while the operating system is doing certain things to
> the application (including, sometimes, when it is swapped out or
> deep in I/O.)

That sounds like an outright bug. I can't think
of any earthly reason why the handler shouldn't
be called eventually, if it remains installed and
the process lives long enough.

--
Greg


From greg.ewing at canterbury.ac.nz  Fri Jan 26 23:10:46 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 27 Jan 2007 11:10:46 +1300
Subject: [Python-Dev] Object creation hook
In-Reply-To: <4E9372E6B2234D4F859320D896059A9504F58E83@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>
	<45B69AF9.3030607@v.loewis.de>
	<4E9372E6B2234D4F859320D896059A9504F58E83@exchis.ccp.ad.local>
Message-ID: <45BA7C66.1070507@canterbury.ac.nz>

Kristj?n V. J?nsson wrote:

> (I was a bit dismayed that I couldn't assign to object.__init__ 
> post-hoc from a python script, I'm fairly sure that is possible in Ruby :)  

It wouldn't do you much good anyway, because no existing
subclass of object having its own __init__ method would
call it.

--
Greg


From nnorwitz at gmail.com  Sat Jan 27 04:24:37 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Fri, 26 Jan 2007 19:24:37 -0800
Subject: [Python-Dev] Object creation hook
In-Reply-To: <4E9372E6B2234D4F859320D896059A9504F58E83@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>
	<45B69AF9.3030607@v.loewis.de>
	<4E9372E6B2234D4F859320D896059A9504F58E83@exchis.ccp.ad.local>
Message-ID: <ee2a432c0701261924i1f426907u513c1fff2b445121@mail.gmail.com>

Short of using a memory debugger such as Valgrind or Purify, have you
considered looking for reference leaks?  These may be the cause and
can be checked with pure python code.  See how Lib/test/regrtest.py
handles the -R option.

n
--

On 1/24/07, Kristj?n V. J?nsson <kristjan at ccpgames.com> wrote:
> Thanks, but the question is really, "how do I build a better debug hook than sys.getobjects?" so I argue this is a valid python-dev question.
>
> We have been using gc.get_objects() but it has several problems:
> 1) It returns all objects in the system.  This results in a list so long that it often kills the system.  Our system is of a scale that makes this very risky.
> 2) There is no way to frame certain operations and get just those objects that were created during their execution.  In our case, we would like to get the server cluster running, then frame a certain operation to get a callback for all created objects, so that we could track that they were later destroyed correctly.  I have done this previously by storing the id()s of all objects returned from gc.get_objects() and comparing them "before" and "after" but this suffers from 1) above, and the ambiguity of id() in the face of objects being created and destroyed.
>
>
> Working with the metaclasses sounds reasonable if one has the luxury of changing the entire codebase to use a different metaclass.  It also doesn't work with old style classes (plenty of those), c extensions, and builtins.
>
> (I was a bit dismayed that I couldn't assign to object.__init__ post-hoc from a python script, I'm fairly sure that is possible in Ruby :)  but I digress...)
>
> My latest attempt was to modify _PyObject_GC_TRACK(o) in objimpl.h, adding this final line:
> if (PyCCP_CreationHook) PyCCP_CreationHookFunc(o);\
>
> The function then looks like this:
>
> void PyCCP_CreationHookFunc(PyObject * obj)
> {
>         if (PyCCP_CreationHook) {
>                 PyObject *result, *tmp = PyCCP_CreationHook;
>                 PyCCP_CreationHook = 0; //guard against recursion
>                 result = PyObject_CallFunctionObjArgs(PyCCP_CreationHook, obj, 0);
>                 Py_XDECREF(result);
>                 if (!result)
>                         PyErr_Clear();
>                 PyCCP_CreationHook = tmp;
>         }
> }
>
> Doing this, and setting a no-op function as as the PyCCP_CreationHook, does seem to work for a while in the interactive python shell, but it then crashes and I haven't been able to work out the crash.  In any case, doing stuff at the point of GC list insertion is very hairy, especially in the face of __del__ methods and such (of which we don't have many)
>
> I am waiting to get Rational Purify set up on my machine and then I'll maybe be able to understand the crash case better.
>
> Cheers,
> Kristj?n
>
>
> -----Original Message-----
> From: "Martin v. L?wis" [mailto:martin at v.loewis.de]
> Sent: 23. jan?ar 2007 23:32
> To: Kristj?n V. J?nsson
> Cc: 'python-dev at python.org'
> Subject: Re: [Python-Dev] Object creation hook
>
> Kristj?n V. J?nsson schrieb:
> > I am trying to insert a hook into python enabling a callback for all
> > just-created objects.  The intention is to debug and find memory leaks,
> > e.g. by having the hook function insert the object into a WeakKeyDictionary.
>
> I'd like to point out that this isn't a python-dev question, but more
> appropriate for comp.lang.python (as it is of the "how do I x with
> Python?" kind).
>
> I would use a debug build, and use sys.getobjects to determine all
> objects and find leaks.
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From kbk at shore.net  Sat Jan 27 05:24:22 2007
From: kbk at shore.net (Kurt B. Kaiser)
Date: Fri, 26 Jan 2007 23:24:22 -0500 (EST)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200701270424.l0R4OM59020656@bayview.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  421 open ( -2) /  3549 closed (+10) /  3970 total ( +8)
Bugs    :  943 open (-17) /  6471 closed (+25) /  7414 total ( +8)
RFE     :  260 open ( +2) /   250 closed ( +1) /   510 total ( +3)

New / Reopened Patches
______________________

rlcompleter tab completion in pdb  (2007-01-22)
       http://python.org/sf/1641544  opened by  Stephen Emslie

logging leaks loggers  (2007-01-22)
CLOSED http://python.org/sf/1641790  opened by  TH

Fix error/crash in AST: syntaxerror in complex ifs  (2007-01-23)
       http://python.org/sf/1642547  opened by  Thomas Wouters

comments to clarify complexobject.c  (2007-01-23)
       http://python.org/sf/1642844  opened by  Jim Jewett

Fix Bug 1362475 Text.edit_modified() doesn't work  (2007-01-24)
       http://python.org/sf/1643641  opened by  Matthias Kievernagel

ctypes leaks memory  (2007-01-24)
CLOSED http://python.org/sf/1643874  opened by  Thomas Heller

file -> open in stdlib  (2007-01-25)
       http://python.org/sf/1644218  opened by  Daniel Nogradi

Allow importing built-in submodules  (2007-01-25)
       http://python.org/sf/1644818  opened by  Miguel Lobo

Patches Closed
______________

Fix for #1601399 (urllib2 does not close sockets properly)  (2007-01-03)
       http://python.org/sf/1627441  closed by  gbrandl

C99 _Bool support for struct  (2006-12-07)
       http://python.org/sf/1610575  closed by  loewis

logging leaks loggers  (2007-01-22)
       http://python.org/sf/1641790  closed by  vsajip

Patch for #1586414 to avoid fragmentation on Windows  (2006-10-31)
       http://python.org/sf/1587674  closed by  gustaebel

smtplib email renames  (2007-01-16)
       http://python.org/sf/1637162  closed by  gbrandl

urllib2: email.Utils->email.utils  (2007-01-16)
       http://python.org/sf/1637159  closed by  gbrandl

urllib: change email.Utils -> email.utils  (2007-01-16)
       http://python.org/sf/1637157  closed by  gbrandl

tarfile extraction does not honor umask  (2006-06-16)
       http://python.org/sf/1507247  closed by  gustaebel

Fix crash when replacing sys.stdout in sitecustomize  (2007-01-08)
       http://python.org/sf/1630975  closed by  twouters

ctypes leaks memory  (2007-01-24)
       http://python.org/sf/1643874  closed by  theller

New / Reopened Bugs
___________________

2.3.6.4 Error in append and extend descriptions  (2007-01-21)
CLOSED http://python.org/sf/1641109  opened by  ilalopoulos

Python 2.5 gets curses.h warning on HPUX  (2007-01-22)
       http://python.org/sf/1642054  opened by  Roy Smith

Grammatical Error  (2007-01-23)
CLOSED http://python.org/sf/1643150  opened by  Steve Miller

function breakpoints in pdb  (2007-01-24)
       http://python.org/sf/1643369  opened by  decitre

Emphasize buffering issues when sys.stdin is used  (2007-01-24)
       http://python.org/sf/1643712  opened by  Raghuram Devarakonda

Problem with signals in a single-threaded application  (2007-01-24)
       http://python.org/sf/1643738  opened by  Ulisses Furquim

strptime %U broken  (2007-01-24)
CLOSED http://python.org/sf/1643943  opened by  Brian Nahas

Error arrow offset wrong  (2007-01-25)
CLOSED http://python.org/sf/1644239  opened by  Cees Timmerman

./configure --prefix=/ breaks, won't build C modules  (2007-01-25)
       http://python.org/sf/1644987  opened by  Jim Shankland

MIME renderer: wrong header line break with long subject?  (2007-01-26)
       http://python.org/sf/1645148  opened by  kxroberto

Bugs Closed
___________

Newline skipped in "for line in file"  (2007-01-16)
       http://python.org/sf/1636950  closed by  bcannon

Over-zealous keyword-arguments check for built-in set class  (2006-05-11)
       http://python.org/sf/1486663  closed by  gbrandl

urllib2 does not close sockets properly  (2006-11-22)
       http://python.org/sf/1601399  closed by  gbrandl

subprocess: error redirecting i/o from non-console process   (2006-11-27)
       http://python.org/sf/1603907  closed by  astrand

Problem running a subprocess  (2007-01-13)
       http://python.org/sf/1634739  closed by  astrand

subprocess.py: O(N**2) bottleneck  (2006-11-17)
       http://python.org/sf/1598181  closed by  astrand

2.3.6.4 Error in append and extend descriptions  (2007-01-21)
       http://python.org/sf/1641109  closed by  gbrandl

asyncore.py and &quot;handle_expt&quot;  (2002-12-16)
       http://python.org/sf/654766  closed by  sf-robot

Segfault provoked by generators and exceptions  (2006-10-18)
       http://python.org/sf/1579370  closed by  loewis

gen_iternext: Assertion `f->f_back != ((void *)0)' failed  (2006-05-06)
       http://python.org/sf/1483133  closed by  nnorwitz

Python polls unnecessarily every 0.1 second when interactive  (2006-09-05)
       http://python.org/sf/1552726  closed by  akuchling

subprocess interpreted double quotation wrong on windows  (2006-03-09)
       http://python.org/sf/1446119  closed by  astrand

tarfile.extract() may cause file fragmentation on Windows XP  (2006-10-28)
       http://python.org/sf/1586414  closed by  gustaebel

subprocess.py Errors with IDLE  (2005-02-17)
       http://python.org/sf/1126208  closed by  astrand

subprocess.Popen fails inside a Windows service  (2005-07-15)
       http://python.org/sf/1238747  closed by  astrand

mailbox.py uses old email names  (2007-01-16)
       http://python.org/sf/1637167  closed by  gbrandl

subprocess.py fails on Windows when there is no console  (2005-11-16)
       http://python.org/sf/1358527  closed by  astrand

mailbox.py _fromlinepattern regexp does not support positive  (2007-01-11)
       http://python.org/sf/1633678  closed by  akuchling

rfc822 module, bug in parsedate_tz  (2005-08-01)
       http://python.org/sf/1249573  closed by  gbrandl

an extra comma in condition command crashes pdb  (2007-01-03)
       http://python.org/sf/1627316  closed by  gbrandl

Use of &quot;except:&quot; in logging module  (2001-03-28)
       http://python.org/sf/411881  closed by  sf-robot

segfaults when using __del__ and weakrefs  (2005-12-10)
       http://python.org/sf/1377858  closed by  bcannon

Grammatical Error  (2007-01-24)
       http://python.org/sf/1643150  closed by  quiver

modulefinder is not documented  (2004-03-11)
       http://python.org/sf/914375  closed by  theller

strptime %U broken  (2007-01-24)
       http://python.org/sf/1643943  closed by  bcannon

Error arrow offset wrong  (2007-01-25)
       http://python.org/sf/1644239  closed by  gbrandl

ctypes: Pointer-to-pointer unchanged in callback  (2006-10-10)
       http://python.org/sf/1574588  closed by  theller

New / Reopened RFE
__________________

recursive urlparse  (2007-01-24)
       http://python.org/sf/1643370  opened by  anatoly techtonik

RFE Closed
__________

Add registry functions to windows postinstall  (2007-01-14)
       http://python.org/sf/1635335  closed by  loewis


From martin at v.loewis.de  Sat Jan 27 09:22:48 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 27 Jan 2007 09:22:48 +0100
Subject: [Python-Dev] Problem with signals in a single
	threaded	application
In-Reply-To: <45BAAC27.6080302@canterbury.ac.nz>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
	<45B70EE1.7080809@v.loewis.de> <45BAAC27.6080302@canterbury.ac.nz>
Message-ID: <45BB0BD8.3040503@v.loewis.de>

Greg Ewing schrieb:
>> Please
>> try to come up with a patch (e.g. by putting a while(is_tripped) loop
>> around the for loop).
> 
> That isn't going to fix it. 

Why not?

> What's needed is to somehow
> atomically test and clear is_tripped at the beginning.

How would that help? The case in question was a signal raised
inside a signal handler. With my proposed solution, that would
be processed in the next while loop; if it is cleared at the
beginning, the call will wait for the next CheckSignals
invocation.

Also, why does it need to clear is_tripped atomically? If
it is only cleared if it is set, I see no need to make
the test and the clearing atomic.

Regards,
Martin

From martin at v.loewis.de  Sat Jan 27 09:36:06 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 27 Jan 2007 09:36:06 +0100
Subject: [Python-Dev] Object creation hook
In-Reply-To: <4E9372E6B2234D4F859320D896059A9504F58E83@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A9504F58E7D@exchis.ccp.ad.local>	<45B69AF9.3030607@v.loewis.de>
	<4E9372E6B2234D4F859320D896059A9504F58E83@exchis.ccp.ad.local>
Message-ID: <45BB0EF6.70803@v.loewis.de>

Kristj?n V. J?nsson schrieb:
> We have been using gc.get_objects() but it has several problems:
> 1) It returns all objects in the system.

Actually, it doesn't. It only returns objects that participate
in cyclic GC (i.e. container objects).

> 2) There is no way to frame certain operations and get just those
> objects that were created during their execution.  In our case, we
> would like to get the server cluster running, then frame a certain
> operation to get a callback for all created objects, so that we could
> track that they were later destroyed correctly.  I have done this
> previously by storing the id()s of all objects returned from
> gc.get_objects() and comparing them "before" and "after" but this
> suffers from 1) above, and the ambiguity of id() in the face of
> objects being created and destroyed.

As Neal says, people are typically successful in debugging memory
leaks by not looking at the entire list of objects, but looking
first at the number of objects (perhaps by type). It's very easy
to take snapshots of these numbers and then check whether they
grow. Only when you see that the numbers grow in an unreasonable
way, you can try to find out why a specific type has too many
objects created.

> void PyCCP_CreationHookFunc(PyObject * obj) { if (PyCCP_CreationHook)
> { PyObject *result, *tmp = PyCCP_CreationHook; PyCCP_CreationHook =
> 0; //guard against recursion result =
> PyObject_CallFunctionObjArgs(PyCCP_CreationHook, obj, 0); 
> Py_XDECREF(result); if (!result) PyErr_Clear(); PyCCP_CreationHook =
> tmp; } }

I think this is not thread-safe.

Regards,
Martin

From martin at v.loewis.de  Sat Jan 27 09:41:23 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 27 Jan 2007 09:41:23 +0100
Subject: [Python-Dev] Python's C interface for types
In-Reply-To: <1f7befae0701261603x81e623fg9a4b73fc87b833ac@mail.gmail.com>
References: <E1HAZld-00030T-Tu@libra.cus.cam.ac.uk>
	<1f7befae0701261603x81e623fg9a4b73fc87b833ac@mail.gmail.com>
Message-ID: <45BB1033.3000700@v.loewis.de>

Tim Peters schrieb:
>> Does that mean that they are unstable, in the sense that they may
>> change behaviour in new versions of Python?
> 
> They /may/ change, but they won't (== only common sense guarantees
> they won't change ;-)).

That, of course, is true for any API. For the documented API, there
is certainly a stronger desire to provide backwards compatibility in
the face of changes, but in some cases, it just means that the change
also gets documented.

Regards,
Martin

From nmm1 at cus.cam.ac.uk  Sat Jan 27 11:12:39 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Sat, 27 Jan 2007 10:12:39 +0000
Subject: [Python-Dev] Problem with signals in a single threaded
	application
Message-ID: <E1HAkY4-0003MT-07@virgo.cus.cam.ac.uk>

I apologise for going off-topic, but this is an explanation of why
I said that signal handling is not reliable.  The only relevance to
Python is that Python should avoid relying on signals if possible,
and try to be a little defensive if not.  Signals will USUALLY do
what is expected, but not always :-(

Anything further by Email, please.


Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
> > This one looks like an oversight in Python code, and so is a bug,
> > but it is important to note that signals do NOT work reliably under
> > any Unix or Microsoft system.
> 
> That's a rather pessimistic way of putting it. In my
> experience, signals in Unix mostly do what they're
> meant to do quite reliably -- it's just a matter of
> understanding what they're meant to do.

Yes, it is pessimistic, but I am afraid that my experience is that
it is so :-(  That doesn't deny your point that they MOSTLY do
'work', but car drivers MOSTLY don't need to wear seat belts, either.
I am talking about high-RAS objectives, and ones where very rare
failure modes can become common (e.g. HPC and other specialist uses).

More commonly, there are plain bugs in the implementations which are
sanctioned by the standards (Linux is relatively disdainful of such
legalistic games).  Because they say that everything is undefined
behaviour, many vendors' support mechanisms will refuse to accept
bug reports unless you push like hell.  And, as some are DIABOLICALLY
difficult to explain, let alone demonstrate, they can remain lurking
for years or decades.

> There may be bugs in certain systems that cause
> signals to get lost under obscure circumstances, but
> that's no reason for Python to make the situation
> worse by introducing bugs of its own.

100% agreed.

> > Two related signals received between two 'checkpoints' (i.e. when
> > the signal is tested and cleared).  You may only get one of them,
> > and 'related' does not mean 'the same'.
> 
> I wasn't aware that this could happen between
> different signals. If it can, there must be some
> rationale as to why the second signal is considered
> redundant. Otherwise there's a bug in either the
> design or the implementation.

Nope.  There is often a clash between POSIX and the hardware, or
a cause where a 'superior' signal overrides an 'inferior' one.
I have seen SIGKILL flush some other signals, for example.  And, on
some systems, SIGFPE may be divided into the basic hardware exceptions.
If you catch SIGFPE as such, all of those may be cleared.  I don't
think that many (any?) current systems do that.

And it is actually specified to occur for the SISSTOP, SIGTSTP,
SIGTTIN, SIGTTOU, SIGCONT group.

> > A second signal received while the first is being 'handled' by the
> > operating system or language run-time system.
> 
> That one sounds odd to me. I would expect a signal
> received during the execution of a handler to be
> flagged and cause the handler to be called again
> after it returns. But then I'm used to the BSD
> signal model, which is relatively sane.

It's nothing to do with the BSD model, which may be saner but still
isn't 100% reliable, but occurs at a lower layer.  At the VERY lowest
level, when a genuine hardware event causes an interrupt, the FLIH
(first-level interrupt handler) runs in God mode (EVERYTHING disabled)
until it classifies what is going on.  This is a ubiquitous misdesign
of modern hardware, but that is off-topic.  Hardware 'signals' from
other CPUs/devices may well get lost if they occur in that window.

And there are other, but less extreme, causes at higher levels in the
operating system.  Unix and Microsoft do NOT have a reliable signal
delivery model, where the sender of a signal checks if the recipient
has got it and retries if not.  Some operating systems do - but I don't
think that BSD does.

> > A signal sent while the operating system is doing certain things to
> > the application (including, sometimes, when it is swapped out or
> > deep in I/O.)
> 
> That sounds like an outright bug. I can't think
> of any earthly reason why the handler shouldn't
> be called eventually, if it remains installed and
> the process lives long enough.

See above.  It gets lost at a low level.  That is why you can cause
serious time drift on an "IBM PC" (most modern ones) by hammering
the video card or generating streams of floating-point fixups.  Most
people don't notice, because xntp or equivalent fixes it up.

And there are worse problems.  I could start on cross-CPU TLB and ECC
handling on large shared memory systems.  I managed to get an Origin
in a state where it wouldn't even power down from the power-off button,
and I had to flip breakers, due to THAT one!  I have reason to believe
that all largish SMP systems have similar problems.

Again, it is possible to design an operating system to avoid those
issues, but we are talking about mainstream ones, and they don't.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From nmm1 at cus.cam.ac.uk  Sat Jan 27 11:27:48 2007
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Sat, 27 Jan 2007 10:27:48 +0000
Subject: [Python-Dev] Python's C interface for types
Message-ID: <E1HAkmi-0003QN-Us@virgo.cus.cam.ac.uk>

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
>
> [not sure what "And so it goes" means in English]

I apologise.  I try to restrain myself from using excessive idiom,
but sometimes I forget.  It means "That is how things are, and there
is and will be more of the same."

> It may be a bit problematic to implement, but I think a clean
> specification is possible. If a and b are numbers, and a==b,
> then hash(a)==hash(b). I'm not sure whether "approximately 5.0"
> equals 5 or not: if it does, it should hash the same as 5,
> if it doesn't, it may or may not hash the same (whatever is
> easier to implement).
> For 0: hash(+0.0)==hash(-0.0)==hash(0)=hash(0L)=0

Unfortunately, that assumes that equality is transitive.  With the
advanced floating-point models, it may not be.  For example, if you
want to avoid the loss of error information, exact infinity and
approximate infinity (the result of overflow) have different
semantics.  Similarly with infinitesimals.

Even at present, Python's float (Decimal probably more so) doesn't
allow you to do some things that are quite reasonable.  For example,
let us say that I am implementing a special function and want to
distinguish -0.0 and +0.0.  Why can't I use a dictionary?

>>> a = float("+0.0")
>>> b = float("-0.0")
>>> print a, b
0.0 -0.0
>>> c = {a: "+0.0", b: "-0.0"}
>>> print c[a], c[b]
-0.0 -0.0

Well, we all know why.  But it is not what some quite reasonable
programmers will expect.  And Decimal (with its cohorts and variant
precisions) has this problem quite badly - as do I.

No, I don't have an answer.  You are damned if you do, and damned
if you don't.  It is an insoluble problem, and CURRENTLY doesn't
justify two hashing mechanisms (i.e. ANY difference and EQUALITY
difference).



Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From greg.ewing at canterbury.ac.nz  Sun Jan 28 01:59:50 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 28 Jan 2007 13:59:50 +1300
Subject: [Python-Dev] Problem with signals in a single
	threaded	application
In-Reply-To: <45BB0BD8.3040503@v.loewis.de>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
	<45B70EE1.7080809@v.loewis.de> <45BAAC27.6080302@canterbury.ac.nz>
	<45BB0BD8.3040503@v.loewis.de>
Message-ID: <45BBF586.4060501@canterbury.ac.nz>

Martin v. L?wis wrote:
> Greg Ewing schrieb:
> 
>>>Please
>>>try to come up with a patch (e.g. by putting a while(is_tripped) loop
>>>around the for loop).
>>
>>That isn't going to fix it. 
> 
> Why not?

Correct me if I'm wrong, but what I got from the OP
was that the current method does

   if (is_tripped) {
     for each signal {
       if the signal has occurred, call its handler
     }
     is_tripped = 0;
   }

and the problem is that any setting of is_tripped that
occurs in the midst of calling the handlers gets
wiped out at the end.

Changing this to

   while (is_tripped) {
     for each signal {
       if the signal has occurred, call its handler
     }
     is_tripped = 0;
   }

doesn't solve that, because is_tripped still gets
set to 0 before it's tested again.

> Also, why does it need to clear is_tripped atomically?

Thinking about it more, probably it doesn't. What's
important is to clear it *before* testing whether any
handlers need to be called, i.e.

   if (is_tripped) {
    is_tripped = 0;
    for each signal {
       if the signal has occurred, call its handler
     }
   }

If you really care, you can make that a while instead
of an if so that you don't have to wait until the next
CheckSignals. But if the signal had arrived a few
microseconds later you'd have to do that anyway, so
I don't see it as a big deal.

--
Greg

From ulissesf at gmail.com  Sun Jan 28 02:18:00 2007
From: ulissesf at gmail.com (Ulisses Furquim)
Date: Sat, 27 Jan 2007 22:18:00 -0300
Subject: [Python-Dev] Problem with signals in a single threaded
	application
In-Reply-To: <45BBF586.4060501@canterbury.ac.nz>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>
	<45B70EE1.7080809@v.loewis.de> <45BAAC27.6080302@canterbury.ac.nz>
	<45BB0BD8.3040503@v.loewis.de> <45BBF586.4060501@canterbury.ac.nz>
Message-ID: <d23110170701271718o2bcc51a1p83b5e416f3637476@mail.gmail.com>

On 1/27/07, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > Why not?
>
> Correct me if I'm wrong, but what I got from the OP
> was that the current method does
>
>    if (is_tripped) {
>      for each signal {
>        if the signal has occurred, call its handler
>      }
>      is_tripped = 0;
>    }
>
> and the problem is that any setting of is_tripped that
> occurs in the midst of calling the handlers gets
> wiped out at the end.
>
> Changing this to
>
>    while (is_tripped) {
>      for each signal {
>        if the signal has occurred, call its handler
>      }
>      is_tripped = 0;
>    }
>
> doesn't solve that, because is_tripped still gets
> set to 0 before it's tested again.

Agreed.

> Thinking about it more, probably it doesn't. What's
> important is to clear it *before* testing whether any
> handlers need to be called, i.e.
>
>    if (is_tripped) {
>     is_tripped = 0;
>     for each signal {
>        if the signal has occurred, call its handler
>      }
>    }
>

That's exactly what my patch does as you can see here:

http://www.python.org/sf/1643738

Regards,

-- Ulisses

From sluggoster at gmail.com  Sun Jan 28 11:53:35 2007
From: sluggoster at gmail.com (Mike Orr)
Date: Sun, 28 Jan 2007 02:53:35 -0800
Subject: [Python-Dev] Unipath package
Message-ID: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>

I finally finished my path package (Unipath) and put it in the Cheeseshop.
http://sluggo.scrapping.cc/python/unipath/

There's a Path class for pathname calculations, and a FSPath subclass
for filesystem calls.  I'm hoping Path -- or something resembling it
-- will find its way into os.path in Python 2.6 or 3.0.  FSPath is
full of convenience methods so it may not be everybody's cup of tea,
but perhaps something similar can go into Python in the farther future

Unipath is an early alpha release so the API may change as it gets
more real-world use.  There's an extensive unittest suite, which
passes on Python 2.5 and 2.4.4 on Linux.  Windows and Macintosh
testers are needed.

Following are highlights from the python-3000 discussion and deviations from it:

-  Path subclasses unicode, or str if the platform can't handle
Unicode pathnames.

- FSPath subclasses Path.  This allows you to do "from unipath import
FSPath as Path" and pretend there's only one class.  I find this much
more convenient in applications, so you don't have to cast objects
back and forth between the classes.  Also, it just became infeasable
not to let them inherit, because so many methods call other methods.

- Nevertheless, you can use Path alone and rest assured it will never
touch the filesystem.  You can even use Path objects with os.*
functions if you are so heretically inclined.  If Path is accepted
into the stdlib, FSPath will inherit it from there.

- I tried splitting FSPath into several mixins according to type of
operation, but the need for methods to call other methods in a
different category sabotaged that too.  So FSPath proudly has about
fifty methods.  (Path has 10 public methods and 4 properties.)

- The dirname property is called .parent.  The basename property is
.name.  The extension property is .ext.  The name without extension is
.stem.

- .components() returns a list of directory components.  The first
component is "/", a Windows drive root, a UNC share, or "" for a
relative path.  .split_root() returns the root and the rest.

- PosixPath, NTPath, and MacPath are Path subclasses using a specific
path library.  This allows you to express non-native paths and convert
paths.  Passing a relative foreign path to a *Path constructor
converts it to the destination type.  Passing an absolute foreign path
is an error, because there's no sane way to interpret "C:\\" on Posix
or "/" on Windows.  I'm skeptical whether this non-native support is
really worth it, because .norm() and .norm_case() already convert
slashes to backslashes (which is what Talin really wanted to do --
have Posix paths in a config file and automatically convert them to
the native format).  And if Python is going to drop Mac OS 9 soon then
MacPath is obsolete.  So maybe this non-native path code will prove
less than useful and will be deleted.  On the other hand, if someone
is burning to write a zippath or ftppath library, you can use it with
this.

- Setting Path.auto_norm to true will automatically normalize all
paths on construction.  I'm not sure if this should be the default,
because normalizing may produce the wrong path (e.g., if it contains a
symlink).  You can always pass norm=True or norm=False to the
constructor to enable/disable it.

- p.child("subdir", "grandkid") is Glyph's favorite "safe join" method
that prevents creating a path pointing above 'p'.  You can use it as a
.joinpath if you don't mind this restriction.  The constructor allows
multiple positional arguments, which are joined using os.path.join().

- Listing is p.listdir(pattern=None, filter=None, names_only=False).
This returns a non-recursive list of paths.  If 'names_only' is true
it returns the same as os.listdir().  p.walk(pattern=None,
filter=None, top_down=True) is the recursive counterpart, yielding
paths.  In both cases 'pattern' is a glob pattern, and 'filter' is one
of the constants (FILES, DIRS, LINKS, FILES_NO_LINKS, DIRS_NO_LINKS,
DEAD_LINKS) or a custom boolean function.  I spent a lot of time going
back and forth between Orendorff's six listing methods and Raphael's
one, and finally decided on this, partly because I'm not satisfied
with how either Orendorff's class or Raphael's or os.walk() handle
symlinks -- sometimes you want to ignore the links and then iterate
them separately.

- .read_file(mode) and .write_file(content, mode) are a compromise
between Orendorff's seven methods and purists' desire for zero
methods.

- .mkdir(), .rmdir(), .rmtree(), .copy(), .copy_stat(), .copy_tree(),
and .move() are more fancy than their stdlib/Orendorff counterparts.
They silently succeed if the operation is already done, and have
arguments to smartly create/delete intermediate directories, etc.

- Two extra functions are in 'unipath.tools'.  'dict2dir' creates a
directory hierarchy modeled after a dict.  'dump_path' displays an
ASCII tree of a directory hierarchy, with file sizes and symlink
targets showing.

Enjoy!  and please provide feedback.

--Mike Orr <sluggoster at gmail.com>

From thobes at gmail.com  Sun Jan 28 15:06:25 2007
From: thobes at gmail.com (Tobias Ivarsson)
Date: Sun, 28 Jan 2007 15:06:25 +0100
Subject: [Python-Dev] Unipath package
In-Reply-To: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
Message-ID: <9997d5e60701280606m7c5c1656k9ca1ed1039e09a70@mail.gmail.com>

Hi!

I am a MSc of Computer Engineering student from Sweden. A frequent reader of
python-3000, and an occasional reader of python-dev for about a year now. I
have wanted to get involved in the development of python for a while but
haven't found the time before. Now, when I'm in the process of trying to
figure out what project to do for my thesis, I have actually found myself
having more time than before. Therefore I thought that now is a better time
than never to get involved. After that compulsory first-post-introduction of
myself, lets get to the reason I choose to send the post at all.

I noticed that you wanted Mac OS tests, here are the test results from my
Mac OS X (10.4), running on an Intel Core Duo MacBook, using python 2.5:

FF...F....................................................
======================================================================
FAIL: test_absolute (__main__.TestCalculatingPaths)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 235, in test_absolute
    eq(p1, p2)
AssertionError: FSPath('/private/tmp/tmpH9fppC/images') !=
FSPath('/tmp/tmpH9fppC/images')

======================================================================
FAIL: test_chdir_absolute_relative (__main__.TestCalculatingPaths)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 220, in test_chdir_absolute_relative
    eq(FSPath.cwd(), self.d)
AssertionError: FSPath('/private/tmp/tmpRJzqbt') != FSPath('/tmp/tmpRJzqbt')

======================================================================
FAIL: test_relative (__main__.TestCalculatingPaths)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 240, in test_relative
    eq(p, "images")
AssertionError: FSPath('../../../tmp/tmpdw9juF/images') != 'images'

----------------------------------------------------------------------
Ran 58 tests in 1.587s


Please let me know if there are any further tests or improvements I could
help you with.

/Tobias

On 1/28/07, Mike Orr <sluggoster at gmail.com> wrote:
>
> I finally finished my path package (Unipath) and put it in the Cheeseshop.
> http://sluggo.scrapping.cc/python/unipath/
>
> There's a Path class for pathname calculations, and a FSPath subclass
> for filesystem calls.  I'm hoping Path -- or something resembling it
> -- will find its way into os.path in Python 2.6 or 3.0.  FSPath is
> full of convenience methods so it may not be everybody's cup of tea,
> but perhaps something similar can go into Python in the farther future
>
> Unipath is an early alpha release so the API may change as it gets
> more real-world use.  There's an extensive unittest suite, which
> passes on Python 2.5 and 2.4.4 on Linux.  Windows and Macintosh
> testers are needed.
>
> Following are highlights from the python-3000 discussion and deviations
> from it:
>
> -  Path subclasses unicode, or str if the platform can't handle
> Unicode pathnames.
>
> - FSPath subclasses Path.  This allows you to do "from unipath import
> FSPath as Path" and pretend there's only one class.  I find this much
> more convenient in applications, so you don't have to cast objects
> back and forth between the classes.  Also, it just became infeasable
> not to let them inherit, because so many methods call other methods.
>
> - Nevertheless, you can use Path alone and rest assured it will never
> touch the filesystem.  You can even use Path objects with os.*
> functions if you are so heretically inclined.  If Path is accepted
> into the stdlib, FSPath will inherit it from there.
>
> - I tried splitting FSPath into several mixins according to type of
> operation, but the need for methods to call other methods in a
> different category sabotaged that too.  So FSPath proudly has about
> fifty methods.  (Path has 10 public methods and 4 properties.)
>
> - The dirname property is called .parent.  The basename property is
> .name.  The extension property is .ext.  The name without extension is
> .stem.
>
> - .components() returns a list of directory components.  The first
> component is "/", a Windows drive root, a UNC share, or "" for a
> relative path.  .split_root() returns the root and the rest.
>
> - PosixPath, NTPath, and MacPath are Path subclasses using a specific
> path library.  This allows you to express non-native paths and convert
> paths.  Passing a relative foreign path to a *Path constructor
> converts it to the destination type.  Passing an absolute foreign path
> is an error, because there's no sane way to interpret "C:\\" on Posix
> or "/" on Windows.  I'm skeptical whether this non-native support is
> really worth it, because .norm() and .norm_case() already convert
> slashes to backslashes (which is what Talin really wanted to do --
> have Posix paths in a config file and automatically convert them to
> the native format).  And if Python is going to drop Mac OS 9 soon then
> MacPath is obsolete.  So maybe this non-native path code will prove
> less than useful and will be deleted.  On the other hand, if someone
> is burning to write a zippath or ftppath library, you can use it with
> this.
>
> - Setting Path.auto_norm to true will automatically normalize all
> paths on construction.  I'm not sure if this should be the default,
> because normalizing may produce the wrong path (e.g., if it contains a
> symlink).  You can always pass norm=True or norm=False to the
> constructor to enable/disable it.
>
> - p.child("subdir", "grandkid") is Glyph's favorite "safe join" method
> that prevents creating a path pointing above 'p'.  You can use it as a
> .joinpath if you don't mind this restriction.  The constructor allows
> multiple positional arguments, which are joined using os.path.join().
>
> - Listing is p.listdir(pattern=None, filter=None, names_only=False).
> This returns a non-recursive list of paths.  If 'names_only' is true
> it returns the same as os.listdir().  p.walk(pattern=None,
> filter=None, top_down=True) is the recursive counterpart, yielding
> paths.  In both cases 'pattern' is a glob pattern, and 'filter' is one
> of the constants (FILES, DIRS, LINKS, FILES_NO_LINKS, DIRS_NO_LINKS,
> DEAD_LINKS) or a custom boolean function.  I spent a lot of time going
> back and forth between Orendorff's six listing methods and Raphael's
> one, and finally decided on this, partly because I'm not satisfied
> with how either Orendorff's class or Raphael's or os.walk() handle
> symlinks -- sometimes you want to ignore the links and then iterate
> them separately.
>
> - .read_file(mode) and .write_file(content, mode) are a compromise
> between Orendorff's seven methods and purists' desire for zero
> methods.
>
> - .mkdir(), .rmdir(), .rmtree(), .copy(), .copy_stat(), .copy_tree(),
> and .move() are more fancy than their stdlib/Orendorff counterparts.
> They silently succeed if the operation is already done, and have
> arguments to smartly create/delete intermediate directories, etc.
>
> - Two extra functions are in 'unipath.tools'.  'dict2dir' creates a
> directory hierarchy modeled after a dict.  'dump_path' displays an
> ASCII tree of a directory hierarchy, with file sizes and symlink
> targets showing.
>
> Enjoy!  and please provide feedback.
>
> --Mike Orr <sluggoster at gmail.com>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/thobes%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070128/7299dcf1/attachment.html 

From fuzzyman at voidspace.org.uk  Sun Jan 28 17:07:32 2007
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sun, 28 Jan 2007 16:07:32 +0000
Subject: [Python-Dev] Unipath package
In-Reply-To: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
Message-ID: <45BCCA44.9040809@voidspace.org.uk>

Mike Orr wrote:
> I finally finished my path package (Unipath) and put it in the Cheeseshop.
> http://sluggo.scrapping.cc/python/unipath/
>   
Hello Mike,

Looking through the docs it looks like a great package. However, didn't 
Guido (on this list anyway) rule that he wouldn't accept a solution 
which subclassed a string type?

Michael Foord

> There's a Path class for pathname calculations, and a FSPath subclass
> for filesystem calls.  I'm hoping Path -- or something resembling it
> -- will find its way into os.path in Python 2.6 or 3.0.  FSPath is
> full of convenience methods so it may not be everybody's cup of tea,
> but perhaps something similar can go into Python in the farther future
>
> Unipath is an early alpha release so the API may change as it gets
> more real-world use.  There's an extensive unittest suite, which
> passes on Python 2.5 and 2.4.4 on Linux.  Windows and Macintosh
> testers are needed.
>
> Following are highlights from the python-3000 discussion and deviations from it:
>
> -  Path subclasses unicode, or str if the platform can't handle
> Unicode pathnames.
>
> - FSPath subclasses Path.  This allows you to do "from unipath import
> FSPath as Path" and pretend there's only one class.  I find this much
> more convenient in applications, so you don't have to cast objects
> back and forth between the classes.  Also, it just became infeasable
> not to let them inherit, because so many methods call other methods.
>
> - Nevertheless, you can use Path alone and rest assured it will never
> touch the filesystem.  You can even use Path objects with os.*
> functions if you are so heretically inclined.  If Path is accepted
> into the stdlib, FSPath will inherit it from there.
>
> - I tried splitting FSPath into several mixins according to type of
> operation, but the need for methods to call other methods in a
> different category sabotaged that too.  So FSPath proudly has about
> fifty methods.  (Path has 10 public methods and 4 properties.)
>
> - The dirname property is called .parent.  The basename property is
> .name.  The extension property is .ext.  The name without extension is
> .stem.
>
> - .components() returns a list of directory components.  The first
> component is "/", a Windows drive root, a UNC share, or "" for a
> relative path.  .split_root() returns the root and the rest.
>
> - PosixPath, NTPath, and MacPath are Path subclasses using a specific
> path library.  This allows you to express non-native paths and convert
> paths.  Passing a relative foreign path to a *Path constructor
> converts it to the destination type.  Passing an absolute foreign path
> is an error, because there's no sane way to interpret "C:\\" on Posix
> or "/" on Windows.  I'm skeptical whether this non-native support is
> really worth it, because .norm() and .norm_case() already convert
> slashes to backslashes (which is what Talin really wanted to do --
> have Posix paths in a config file and automatically convert them to
> the native format).  And if Python is going to drop Mac OS 9 soon then
> MacPath is obsolete.  So maybe this non-native path code will prove
> less than useful and will be deleted.  On the other hand, if someone
> is burning to write a zippath or ftppath library, you can use it with
> this.
>
> - Setting Path.auto_norm to true will automatically normalize all
> paths on construction.  I'm not sure if this should be the default,
> because normalizing may produce the wrong path (e.g., if it contains a
> symlink).  You can always pass norm=True or norm=False to the
> constructor to enable/disable it.
>
> - p.child("subdir", "grandkid") is Glyph's favorite "safe join" method
> that prevents creating a path pointing above 'p'.  You can use it as a
> .joinpath if you don't mind this restriction.  The constructor allows
> multiple positional arguments, which are joined using os.path.join().
>
> - Listing is p.listdir(pattern=None, filter=None, names_only=False).
> This returns a non-recursive list of paths.  If 'names_only' is true
> it returns the same as os.listdir().  p.walk(pattern=None,
> filter=None, top_down=True) is the recursive counterpart, yielding
> paths.  In both cases 'pattern' is a glob pattern, and 'filter' is one
> of the constants (FILES, DIRS, LINKS, FILES_NO_LINKS, DIRS_NO_LINKS,
> DEAD_LINKS) or a custom boolean function.  I spent a lot of time going
> back and forth between Orendorff's six listing methods and Raphael's
> one, and finally decided on this, partly because I'm not satisfied
> with how either Orendorff's class or Raphael's or os.walk() handle
> symlinks -- sometimes you want to ignore the links and then iterate
> them separately.
>
> - .read_file(mode) and .write_file(content, mode) are a compromise
> between Orendorff's seven methods and purists' desire for zero
> methods.
>
> - .mkdir(), .rmdir(), .rmtree(), .copy(), .copy_stat(), .copy_tree(),
> and .move() are more fancy than their stdlib/Orendorff counterparts.
> They silently succeed if the operation is already done, and have
> arguments to smartly create/delete intermediate directories, etc.
>
> - Two extra functions are in 'unipath.tools'.  'dict2dir' creates a
> directory hierarchy modeled after a dict.  'dump_path' displays an
> ASCII tree of a directory hierarchy, with file sizes and symlink
> targets showing.
>
> Enjoy!  and please provide feedback.
>
> --Mike Orr <sluggoster at gmail.com>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>   


From steven.bethard at gmail.com  Sun Jan 28 19:26:33 2007
From: steven.bethard at gmail.com (Steven Bethard)
Date: Sun, 28 Jan 2007 11:26:33 -0700
Subject: [Python-Dev] Unipath package
In-Reply-To: <45BCCA44.9040809@voidspace.org.uk>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
	<45BCCA44.9040809@voidspace.org.uk>
Message-ID: <d11dcfba0701281026g165c2162o48e507e864209d69@mail.gmail.com>

On 1/28/07, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
> Mike Orr wrote:
> > I finally finished my path package (Unipath) and put it in the Cheeseshop.
> > http://sluggo.scrapping.cc/python/unipath/
> >
> Hello Mike,
>
> Looking through the docs it looks like a great package. However, didn't
> Guido (on this list anyway) rule that he wouldn't accept a solution
> which subclassed a string type?

To be easily useful as a third-party module, it needs to subclass
str/unicode so you can pass it directly to something like open().
That inheritance could be dropped if the module was included in the
stdlib, as long as the inclusion was also accompanied by appropriate
modifications to open()/file() to allow Path objects.

STeVe
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
        --- Bucky Katt, Get Fuzzy

From sluggoster at gmail.com  Sun Jan 28 19:30:19 2007
From: sluggoster at gmail.com (Mike Orr)
Date: Sun, 28 Jan 2007 10:30:19 -0800
Subject: [Python-Dev] Unipath package
In-Reply-To: <9997d5e60701280606m7c5c1656k9ca1ed1039e09a70@mail.gmail.com>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
	<9997d5e60701280606m7c5c1656k9ca1ed1039e09a70@mail.gmail.com>
Message-ID: <6e9196d20701281030x11dbcf9fl263c71fa0287230c@mail.gmail.com>

On 1/28/07, Tobias Ivarsson <thobes at gmail.com> wrote:
> I am a MSc of Computer Engineering student from Sweden. A frequent reader of
> python-3000, and an occasional reader of python-dev for about a year now. I
> have wanted to get involved in the development of python for a while but
> haven't found the time before. Now, when I'm in the process of trying to
> figure out what project to do for my thesis, I have actually found myself
> having more time than before. Therefore I thought that now is a better time
> than never to get involved. After that compulsory first-post-introduction of
> myself, lets get to the reason I choose to send the post at all.

Hi Tobias!  You've got the first step in Python development -- reading
python-dev.  Now you just have to pick an interesting task from the
dozens hinted on the list and do it.  Of course, there are many tasks
not on python-dev that are also worthwhile.  Bugfixes in the Python
core, adopting a package that lost its maintainer, being active in the
SIGs, testing code (as you've just done) -- these are just a few
things off the top of my head.  Working on third-party libraries is
just as legitimate as working on the core; both are needed.  There are
also non-technical needs like promoting/marketing Python.  If you can
make it to PyCon or EuroPython, you'll meet a lot of contacts you can
potentially collaborate with.  Laura Creighton lives in Sweden and is
highly involved in Python activities; she would no doubt have some
suggestions.  If you can't locate her I can dig up her address.  If
you'd like to write me privately about your specific interests and
talents in Python, I can try to steer you to a group that is working
on that.

> I noticed that you wanted Mac OS tests, here are the test results from my
> Mac OS X (10.4), running on an Intel Core Duo MacBook, using python 2.5:

Thanks for the results.  The details of Unipath's code are outside the
scope of python-dev so I'll respond privately.  The issue here is what
kind of path object would be suitable for the standard library in 2.x.
 Guido has said he's not convinced that any OO-based path class is
necessarily superior to the existing functions, but left the door open
to later consideration.  The existing proposals were rejected for
reasons the Guido didn't personally elaborate, but others channeling
him speculated they were too monolithic: too many methods in one
class, too many diverse kinds of methods in one class, and especially
mixing pathname-calculation and filesystem-access methods
undifferentiated in one class.  There is general agreement that a
pathname-calculation class is more needed than a filesystem-access
class due to the nesting of pathname calculations in expressions, so a
pathname-calculation class has a greater chance of acceptance. There's
also the question of what can be put in Python 2.x vs 3.0.  There's a
parallel proposal to move the filesystem-access functions in os.path
into os, and to move everything in shlib into os, possibly with
changes.  This has the greatest chance of acceptance in 2.x.  A small
os.path.Path class has a middling chance.

The discussion has been hampered by the lack of released code.  Only
Orendorff's class has been widely used in production systems.  The
others have either never been used or only by their authors; they
haven't made it to the Cheeseshop.  Unipath is merely to say "Here's
another way to do it; see if it works for you."  Certainly the Path
methods need more testing and use in the real world before they'd be
ready for the stdlib.  The FSPath methods are more experimental so I'd
say they need a year of use before they can be considered sufficiently
stable.

-- 
Mike Orr <sluggoster at gmail.com>

From sluggoster at gmail.com  Sun Jan 28 19:43:07 2007
From: sluggoster at gmail.com (Mike Orr)
Date: Sun, 28 Jan 2007 10:43:07 -0800
Subject: [Python-Dev] Unipath package
In-Reply-To: <d11dcfba0701281026g165c2162o48e507e864209d69@mail.gmail.com>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
	<45BCCA44.9040809@voidspace.org.uk>
	<d11dcfba0701281026g165c2162o48e507e864209d69@mail.gmail.com>
Message-ID: <6e9196d20701281043n71270dfak72e567c7b57668a4@mail.gmail.com>

On 1/28/07, Steven Bethard <steven.bethard at gmail.com> wrote:
> On 1/28/07, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
> > Mike Orr wrote:
> > > I finally finished my path package (Unipath) and put it in the Cheeseshop.
> > > http://sluggo.scrapping.cc/python/unipath/
> > >
> > Hello Mike,
> >
> > Looking through the docs it looks like a great package. However, didn't
> > Guido (on this list anyway) rule that he wouldn't accept a solution
> > which subclassed a string type?
>
> To be easily useful as a third-party module, it needs to subclass
> str/unicode so you can pass it directly to something like open().
> That inheritance could be dropped if the module was included in the
> stdlib, as long as the inclusion was also accompanied by appropriate
> modifications to open()/file() to allow Path objects.

Character slicing and some other string methods are also immensely
useful.  Subclassing string allows you to get all of them at once,
rather than leaving out the one that was not foreseen to be needed.  I
was formerly a fan of the tuple-component approach but somebody said
it splits the path too early, makes too many platform-specific
assumptions or something like that.  In practice, you really do want a
path compatible with string so you can pass it to any function (stdlib
or third-party) that accepts a string path, without getting into cases
where you have to cast it (str()).  There's also the string/unicode
problem in Python 2.x: if you call str() on a string containing
non-ASCII characters, you get an exception, and it's not always
possible to guarantee the string you got from some other code is safe.

If the BDFL does not accept a string-based path class, it will still
have a long happy life as a third-party library.

-- 
Mike Orr <sluggoster at gmail.com>

From larry at hastings.org  Sun Jan 28 22:07:27 2007
From: larry at hastings.org (Larry Hastings)
Date: Sun, 28 Jan 2007 13:07:27 -0800
Subject: [Python-Dev] [Python-3000] Unipath package
In-Reply-To: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
Message-ID: <45BD108F.6040706@hastings.org>


I dropped the Cc: of Python-3000, because I don't think this discussion 
falls under that mailing list's charter.  As I understand it, the 
Python-3000 mailing list is for discussing the details of implementing 
Python 3000.  "Stuff I'd like to see in Python 3000" doesn't go there, 
it goes to "Python-Ideas".

One part of your message caught my eye:
> Passing an absolute foreign path is an error, because there's no sane way to interpret "C:\\" on Posix or "/" on Windows.
There is in fact a /very/ sane way to interpret "/" on Windows: the root 
directory of the "current" drive.  It's equivalent to "\".  I guess it 
isn't widely known, but nearly all Windows APIs are agnostic about 
whether you use "\" or "/" as a directory separator.  (The only 
exceptions I recall are the common dialog open / save file functions.)  
I'm a Windows programmer, and I frequently use "/".

The fact that you don't understand this about how Windows paths work 
makes me nervous about your path library.  For instance, did you 
correctly support local paths on explicit drives (e.g. "d:../foo")?


/larry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070128/0f73839d/attachment.htm 

From sluggoster at gmail.com  Sun Jan 28 23:18:51 2007
From: sluggoster at gmail.com (Mike Orr)
Date: Sun, 28 Jan 2007 14:18:51 -0800
Subject: [Python-Dev] [Python-3000] Unipath package
In-Reply-To: <45BD108F.6040706@hastings.org>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
	<45BD108F.6040706@hastings.org>
Message-ID: <6e9196d20701281418m211873ban6c4da08de359f866@mail.gmail.com>

On 1/28/07, Larry Hastings <larry at hastings.org> wrote:
>  I dropped the Cc: of Python-3000, because I don't think this discussion
> falls under that mailing list's charter.  As I understand it, the
> Python-3000 mailing list is for discussing the details of implementing
> Python 3000.  "Stuff I'd like to see in Python 3000" doesn't go there, it
> goes to "Python-Ideas".

python-ideas didn't exist when this thread was started. :)  I was just
giving a status report on existing issues we'd discussed on
python-3000.  You're right that any future work in this area will
probably involve python-ideas.  I'm waiting for Unipath to get some
use before re-raising the question of what direction we want the
stdlib to go.  I also support the other proposal to move some
functions from os.path/shutil to os, which is being worked on.

>  One part of your message caught my eye:
>
>  Passing an absolute foreign path is an error, because there's no sane way
> to interpret "C:\\" on Posix or "/" on Windows.
>  There is in fact a very sane way to interpret "/" on Windows: the root
> directory of the "current" drive.  It's equivalent to "\".

That's one way to do it, but whether Unipath should presume this is
what the programmer wants is another issue.  One can make an argument
either way.  I found it appealing to categorically refuse to translate
absolute paths to a foreign system, and force the user to make the
path relative and explicitly attach it to a new root (or
subdirectory).  "Explicit is better than implicit", as the Python
mantra goes.  If this turns out to be a hassle for programmers, we can
always be more lenient later.

> I guess it isn't
> widely known, but nearly all Windows APIs are agnostic about whether you use
> "\" or "/" as a directory separator.  (The only exceptions I recall are the
> common dialog open / save file functions.)  I'm a Windows programmer, and I
> frequently use "/".

I've heard contradictory things about this.  Some people say just use
"/" and be happy; others say you get into trouble sometimes because
it's not the Windows kernel that does this but the application or
function, so some applications support it and some don't.  In
particular I've heard that "/" on the command line is handled by the
application, which may impose the DOS semantics ("option flag") rather
than "directory separator".

>  The fact that you don't understand this about how Windows paths work makes
> me nervous about your path library.  For instance, did you correctly support
> local paths on explicit drives (e.g. "d:../foo")?

I did use Windows regularly until 1998.  I do remember drive-relative
paths.  My question is, do people seriously use them anymore?  In a
Python program?  Specifying a drive-relative path or chdir'ing to it
should work.  The one thing Unipath does is consider it absolute:
isabsolute() returns True, and .components() and .split_root() return
"d:" as the root of the filesystem -- which it is.  Joining the
components produces the original path, which Windows should correctly
interpret.  So I don't see the harm as treating it as absolute in my
model.  Maybe it does create some hardship for Windows users, and
maybe I'll have to consider.  But drive-relative paths are really an
intermediate level between absolute and relative, and the model has no
place for the relative drive specification.  E.g.,

    Path("../foo").components()  =>  ["", "..", "foo"]
    Path("C:\\foo").components()  =>  ["C:\\", "foo"]
    Path("C:../foo").components() => ["C:", "..", "foo"]

Strictly speaking, the last path is relative so the first element
should be "".  But that would leave no place for the "C:" specifier,
which would either belongs at index 0.5 or in a special attribute.
Accommodating either would require changing the entire model to
accommodate an obscure Windows-only feature, even though the model
works quite well otherwise.

It may be that Unipath just isn't suitable for manipulating Windows
drive-relative paths, and making it do so would would mess up Unipath
too much.  The former is for Windows users to decide; the latter is
for me to decide based on their input.

I have also dropped .drive, .unc, .splitdrive(), and .splitunc().  I
think .split_root() is robust enough for what people want to do.  But
I may be wrong.  If there's necessity for those methods/properties
I'll add them, but I don't want to add a bunch of legacy methods that
will rarely be called.

-- 
Mike Orr <sluggoster at gmail.com>

From sluggoster at gmail.com  Mon Jan 29 00:08:55 2007
From: sluggoster at gmail.com (Mike Orr)
Date: Sun, 28 Jan 2007 15:08:55 -0800
Subject: [Python-Dev] [Python-3000] Unipath package
In-Reply-To: <6e9196d20701281418m211873ban6c4da08de359f866@mail.gmail.com>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
	<45BD108F.6040706@hastings.org>
	<6e9196d20701281418m211873ban6c4da08de359f866@mail.gmail.com>
Message-ID: <6e9196d20701281508j9e78e13wcc11e58ab9b54548@mail.gmail.com>

On 1/28/07, Mike Orr <sluggoster at gmail.com> wrote:
> >  Passing an absolute foreign path is an error, because there's no sane way
> > to interpret "C:\\" on Posix or "/" on Windows.
> >  There is in fact a very sane way to interpret "/" on Windows: the root
> > directory of the "current" drive.  It's equivalent to "\".
>
> That's one way to do it, but whether Unipath should presume this is
> what the programmer wants is another issue.  One can make an argument
> either way.

The issue is that *maybe* the programmer wants to copy "/etc/mailcap"
on his Posix filesystem to "\etc\mailcap" on his Windows drive, but
should we assume this since \etc does not have the special status on
Windows that /etc does on Unix?  Maybe we should raise an exception
because the programmer intended to copy it somewhere else and forgot
something.  Nothing wrong with forcing him to say, "Yes, I really mean
'\'."

As for Posix-format paths in config files that should be translated to
the native NT, that sounds straightforward.  These paths should be
relative to some platform-specific root, because "C:/etc/mailcap" is
certainly not the path you intend to use on all platforms.  (Unix
would put it in the current directory, wherever that might be.)  You
can create a Path("/foo/bar") on NT and use it.  The only thing you
can't do is:

    p = Path( PosixPath("/foo/bar") )   # Illegal for absolute paths
if Path != PosixPath.

-- 
Mike Orr <sluggoster at gmail.com>

From greg.ewing at canterbury.ac.nz  Mon Jan 29 00:55:00 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 29 Jan 2007 12:55:00 +1300
Subject: [Python-Dev] [Python-3000] Unipath package
In-Reply-To: <45BD108F.6040706@hastings.org>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
	<45BD108F.6040706@hastings.org>
Message-ID: <45BD37D4.3080207@canterbury.ac.nz>

Larry Hastings wrote:

> There is in fact a /very/ sane way to interpret "/" on Windows: the root 
> directory of the "current" drive.

Whether that's sane or not is debatable -- it depends
entirely on what the application and/or user expect.
A Unix user is probably expecting "/foo" to be completely
unambiguous, and might be surprised if it gets turned
into a Windows path that's not.

In any case, there's no obvious meaning when going the other
way (i.e. translating "C:\" into a Unix path). So I think
it's reasonable to say that translation of absolute paths
is not supported in general.

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | Carpe post meridiem!          	  |
Christchurch, New Zealand	   | (I'm not a morning person.)          |
greg.ewing at canterbury.ac.nz	   +--------------------------------------+

From greg.ewing at canterbury.ac.nz  Mon Jan 29 01:13:08 2007
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 29 Jan 2007 13:13:08 +1300
Subject: [Python-Dev] [Python-3000] Unipath package
In-Reply-To: <6e9196d20701281508j9e78e13wcc11e58ab9b54548@mail.gmail.com>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
	<45BD108F.6040706@hastings.org>
	<6e9196d20701281418m211873ban6c4da08de359f866@mail.gmail.com>
	<6e9196d20701281508j9e78e13wcc11e58ab9b54548@mail.gmail.com>
Message-ID: <45BD3C14.1060000@canterbury.ac.nz>

Mike Orr wrote:
> The issue is that *maybe* the programmer wants to copy "/etc/mailcap"
> on his Posix filesystem to "\etc\mailcap" on his Windows drive,

That doesn't make sense. In order to do this, either the
Windows file system must be mounted on the Unix system, in
which case Unix paths are used for both, or the Unix file
system must be mounted on the Windows system, in which case
they're both Windows paths. In neither case is a conversion
from Unix to Windows pathnames needed.

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | Carpe post meridiem!          	  |
Christchurch, New Zealand	   | (I'm not a morning person.)          |
greg.ewing at canterbury.ac.nz	   +--------------------------------------+

From sluggoster at gmail.com  Mon Jan 29 04:12:36 2007
From: sluggoster at gmail.com (Mike Orr)
Date: Sun, 28 Jan 2007 19:12:36 -0800
Subject: [Python-Dev] Unipath package
In-Reply-To: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
References: <6e9196d20701280253h6c0b887dy5fec85e55c76c0c@mail.gmail.com>
Message-ID: <6e9196d20701281912l6b576018ra1ee00ca7763657@mail.gmail.com>

Thanks everyone who sent Windows and Mac unittest reports.  I've got
enough of them now.

95% of the Windows errors are because I forgot to remove symlinks from
the control values.
The repr() problem is more mysterious: somehow one path is turning
into unicode while the other isn't.  The chmod error probably just
needs some bits excluded from the comparision.

The Mac errors have to do with the general temp directory being a
symlink, and some paths being resolved while others aren't. I'll have
to take that up with my Mac-loving colleagues.

0.1.1 will be in the Cheeseshop soon with fixed tests.

-- 
Mike Orr <sluggoster at gmail.com>

From glyph at divmod.com  Mon Jan 29 04:41:04 2007
From: glyph at divmod.com (glyph at divmod.com)
Date: Mon, 29 Jan 2007 03:41:04 -0000
Subject: [Python-Dev] Unipath package
Message-ID: <20070129034104.28203.1940118452.divmod.xquotient.238@joule.divmod.com>


On 28 Jan, 06:30 pm, sluggoster at gmail.com wrote:
>The discussion has been hampered by the lack of released code.  Only
>Orendorff's class has been widely used in production systems.  The
>others have either never been used or only by their authors; they
>haven't made it to the Cheeseshop.  Unipath is merely to say "Here's
>another way to do it; see if it works for you."  Certainly the Path
>methods need more testing and use in the real world before they'd be
>ready for the stdlib.  The FSPath methods are more experimental so I'd
>say they need a year of use before they can be considered sufficiently
>stable.

Mike is mistaken here; Twisted has a module, twisted.python.filepath, which has been used extensively in production as well as by other projects.  You can see such a project here (just the first hit on google code search):

    http://www.google.com/codesearch?hl=en&q=+twisted+python+filepath+show:3PfULFjjkV4:Vyh6PbYbXnU:7_Cvpg7zWo4&sa=N&cd=30&ct=rc&cs_p=https://taupro.com/pubsvn/Projects/Xanalogica/ShardsKeeper/trunk&cs_f=shardskeeper.py#a0

and the source to the module itself here:

    http://twistedmatrix.com/trac/browser/trunk/twisted/python/filepath.py

Twisted's filepath implementation also provides a zipfile implementation of the same interface, so that, for example, you can copy a directory in the filesystem and put it into a zipfile with the same function.  We plan to eventually also include tarfile and in-memory implementations.

Of course, I am FilePath's author, so I have a certain bias; but I think it would be well suited to the standard library, and I would be interested to hear any feedback on it, especially that which would make it unsuitable for inclusion.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070129/83231d99/attachment.html 

From martin at v.loewis.de  Mon Jan 29 08:52:12 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 29 Jan 2007 08:52:12 +0100
Subject: [Python-Dev] Problem with signals in a
	single	threaded	application
In-Reply-To: <45BBF586.4060501@canterbury.ac.nz>
References: <d23110170701231137hc0e8ff6hfcd053721fa136f1@mail.gmail.com>	<45B70EE1.7080809@v.loewis.de>
	<45BAAC27.6080302@canterbury.ac.nz>	<45BB0BD8.3040503@v.loewis.de>
	<45BBF586.4060501@canterbury.ac.nz>
Message-ID: <45BDA7AC.8070408@v.loewis.de>

Greg Ewing schrieb:
> Correct me if I'm wrong, but what I got from the OP
> was that the current method does

Ok, I'm correcting you: This is not what the current
method does:

> 
>    if (is_tripped) {
>      for each signal {
>        if the signal has occurred, call its handler
>      }
>      is_tripped = 0;
>    }

Instead, the current method does this:

if (!is_tripped)
  return;
for each signal {
  if the signal has occurred, call its handler
}
is_tripped = 0

> and the problem is that any setting of is_tripped that
> occurs in the midst of calling the handlers gets
> wiped out at the end.
> 
> Changing this to
> 
>    while (is_tripped) {
>      for each signal {
>        if the signal has occurred, call its handler
>      }
>      is_tripped = 0;
>    }

My proposal (wrap the for loop with another is_tripped
looo) would literally give

  if (!is_tripped)
    return;
  while(is_tripped) {
    for each signal {
       if the signal has occurred, call its handler
    }
 }
 is_tripped = 0

Of course, that doesn't make much sense as it gives
an infinite loop. You need to clear is_tripped somewhere,
and I thought clearing it before the for loop would
work

  if (!is_tripped)
    return;
  while(is_tripped) {
    is_tripped = 0;
    for each signal {
       if the signal has occurred, call its handler
    }
 }

> If you really care, you can make that a while instead
> of an if so that you don't have to wait until the next
> CheckSignals. But if the signal had arrived a few
> microseconds later you'd have to do that anyway, so
> I don't see it as a big deal.

Sure. However, it's not that the signal would occur
"randomly" a few microseconds later, but instead
occurs *in* the signal handler.

I think Python currently has some "guarantee" that
a signal handler will be called quickly: either the
signal gets tripped in a blocking system call, which
ought to abort the system call, or the interpreter
is executing byte codes, when it will check for
signals every nth instruction.

A signal can get delayed significantly if the
interpreter makes a blocking system call before
handling the signal. I think this can happen
even today (if there is a blocking call between
the signal and the nth instruction, the signal
may still get delayed). However, in this specific
case, I think the chance that that the signal
gets delayed is high, and the case can be easily
implemented to avoid that risk.

Regards,
Martin

From nnorwitz at gmail.com  Tue Jan 30 08:01:32 2007
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Mon, 29 Jan 2007 23:01:32 -0800
Subject: [Python-Dev] 2.5.1 release postponed
Message-ID: <ee2a432c0701292301w14373c99i477f3f21b532f7be@mail.gmail.com>

As you may have noticed we missed the schedule for getting 2.5.1 out.
Unfortunately several of us are having scheduling problems with making
this release.  Since there don't seem to be major problems with the current
2.5 release, our current plan is to defer release until April when
everyone should have more time to do it right.

n

From lingwitt at gmail.com  Wed Jan 31 15:15:38 2007
From: lingwitt at gmail.com (lingwitt at gmail.com)
Date: Wed, 31 Jan 2007 09:15:38 -0500
Subject: [Python-Dev] Scoping [Patience, please]
Message-ID: <FACD4404-113F-4FC8-BBAE-A285F7FAD619@gmail.com>

This email is broken up into multiple sections:
	(1) Introduction
	(2) Problem
	(3) Probably Poor Solution
	(4) Tangent (Utter Tomfoolery)



#### Introduction ####

Hello,

This is my first post to the list,
and I apologize for 4 reasons:
	(1) I haven't been lurking for more than a few days.
	(2) I am by no means well versed in python.
	(3) I know nothing of language design.
	(4) This isn't a very good introduction.

Nonetheless, I have a habit of beginning with institutions
that are on the verge of  sweeping changes, and python
seems no different.

To curb the future troubles against which this unfortunate
fate of mine often forces me, I have been pouring over PEPs
(scanning really, because it's far past my bed time).

I came across the discussion of scoping, and while it seems
to be a very badly beaten and decaying horse, I ask that you'd
be kind enough to include me in this disgruntled discourse.



#### Problem ####

See http://www.python.org/dev/peps/pep-3104/

Consider the following:

 >>> x = 10
 >>> def a():
...     x = 3
...     def b():
...         global x
...         x = 4
...         print x
...
...     b()
...     print x
...
 >>> a()
4
3
 >>> print x
4

The keyword 'global' is only good for the module scope.



#### Probably Poor Solution ####

See http://www.python.org/dev/peps/pep-3104/

There seems to be a predilection towards limited
and/or general means of outer scope referencing,
such as
	(1) prefixing variables with dots.
	(2) scope override declarations.

However, I don't understand the virtue of this generality
(of course, I haven't thought much about it either, so please
lend me your expertise).

Most of the time, it would seem a person should know exactly which
outer variable he is after; he knows exactly which scope.

Fortunately, the scopes in question are always named.
Can't we just reference what's already there?

 >>> x = 10
 >>> def a():
...     x = 3
...     def b():
...         a.x = 4
...         print a.x
...
...     b()
...     print x
...
 >>> a()
4
4
 >>> print x
10

Of course, the module itself is unnamed, and it would
be completely consistent to use the single prefixed dot
to reference it:

 >>> x = 10
 >>> def a():
...     x = 3
...     def b():
...         a.x = 4
...         print a.x
...
...    .x = 5
...     b()
...     print x
...
 >>> a()
4
4
 >>> print x
5

However, I would think a placeholder is easier to spot:

 >>> x = 10
 >>> def a():
...     x = 3
...     def b():
...         a.x = 4
...         print a.x
...
...    @.x = 5
...     b()
...     print x
...
 >>> a()
4
4
 >>> print x
5

Of course, what happens if 'a' is a variable name?

 >>> x = 10
 >>> def a():
...     x = 3
...     def b():
...         a = 15
...         a.x = 4     #?
...         print a.x
...
...    @.x = 5
...     b()
...     print x
...

This would never happen in practice. Why bother considering it?
However, one could get around it with a fuller qualification:

 >>> x = 10
 >>> def a():
...     x = 3
...     def b():
...         a = 15
...         @.a.x = 4
...         print a.x
...
...    @.x = 5
...     b()
...     print x
...
 >>> a()
4
4
 >>> print x
5

In any case, those are just some thoughts off of the top of my  
(tired) head.
It seems to me that it unifies a lot of scoping ideas. everything can  
be accessed
with a qualified name.

Sorry if this is nonsensical.



#### Tangent (Utter Tomfoolery) ####

More interestingly, one sees (if one squints) the distinction between  
modules,
classes, and functions blur:

 >>> def new_b():
...     print "tee hee"
 >>> a.b = new_b()
 >>>a()
tee hee
3
 >>> print x
5

or perhaps:

 >>> # Interestingly, the following code is valid already
...
 >>> def Person(name, age, location):
...     def print_record():
...         print('The last person created had the name ' +  
Person.name_last)
...         print("However, I don't know any of the details")
...
...     def instance():
...         def print_record():
...             print('Name: ' + name)
...             print('age: ' + str(age))
...             print('location: ' + location)
...
...     if name == 'lingwitt':
...         name = age = location = 'classified'
...
...     Person.name_last = name
...
...     return instance
...
 >>>p1 = Person('spam', 80, 'eggsville')
 >>>Person.name_last
spam
 >>># Now the code becomes invalid
...
 >>>Person.print_record()
The last person created had the name spam
However, I don't know any of the details
 >>>p1.print_record()
name: spam
age: 80
location: eggsville



It would appear that modules, classes, and functions are scopes
with different magic associated with them.


#### Warning!!! This is an unlisted section! Sillyness Ensues ####

It would appear that a language could be built around scopes;
One could define a scope and then use that same scope as a template
for other scopes (actually, it sounds kind of like prototyping  
languages);
this seems to encompass the behavior of modules, classes, and functions.

Seems like all the "kinds" of scopes could be unified.

I'll stop here, because now I'm worried that I've made myself look  
silly.

Thanks for your time.

Yours truly,
Herr Witten

P.S.

Be gentle.

From lingwitt at gmail.com  Wed Jan 31 15:52:48 2007
From: lingwitt at gmail.com (lingwitt at gmail.com)
Date: Wed, 31 Jan 2007 09:52:48 -0500
Subject: [Python-Dev] Scoping [Patience, please]
In-Reply-To: <FACD4404-113F-4FC8-BBAE-A285F7FAD619@gmail.com>
References: <FACD4404-113F-4FC8-BBAE-A285F7FAD619@gmail.com>
Message-ID: <D3E141BE-C952-4A23-B804-3A094C5BF750@gmail.com>

lingwitt at gmail.com wrote:
> #### Tangent (Utter Tomfoolery) ####
>
> More interestingly, one sees (if one squints) the distinction  
> between modules,
> classes, and functions blur:
>
> >>> def new_b():
> ...     print "tee hee"
> >>> a.b = new_b()
> >>>a()
> tee hee
> 3
> >>> print x
> 5
>


After falling into bed, I realized that everything after this point  
contradictions
with my earlier usage of qualified names.

Upon further investigation, it seems what I have suggested above
conflicts with the use of func_dict.


> or perhaps:
>
> >>> # Interestingly, the following code is valid already
> ...
> >>> def Person(name, age, location):
> ...     def print_record():
> ...         print('The last person created had the name ' +  
> Person.name_last)
> ...         print("However, I don't know any of the details")
> ...

From aahz at pythoncraft.com  Wed Jan 31 16:11:04 2007
From: aahz at pythoncraft.com (Aahz)
Date: Wed, 31 Jan 2007 07:11:04 -0800
Subject: [Python-Dev] Happy Birthday, Guido!
Message-ID: <20070131151104.GA11441@panix.com>

Thanks again for giving me something fun to do with my life.  ;-)
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"I disrespectfully agree."  --SJM

From barry at python.org  Wed Jan 31 16:24:14 2007
From: barry at python.org (Barry Warsaw)
Date: Wed, 31 Jan 2007 10:24:14 -0500
Subject: [Python-Dev] Happy Birthday, Guido!
In-Reply-To: <20070131151104.GA11441@panix.com>
References: <20070131151104.GA11441@panix.com>
Message-ID: <B2A306F2-0821-4095-8229-D8E9FEF13A09@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 31, 2007, at 10:11 AM, Aahz wrote:

> Thanks again for giving me something fun to do with my life.  ;-)

Here, here!
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iQCVAwUBRcC0nnEjvBPtnXfVAQI2+AP/QcOBBiGI/zDreuFU2bjq0MNH8qIocHnA
y2XkoaXFDNYlOO6Uj39kttFl8HecMmyTyLvcyMTFBDxxN0xxQBpOPBkzFe6D44HQ
x8R/VsaSY8wB3G4NZyYOoBjJMPPTN1x6U3lg+jzHkuRE4SSFYZmoWuvUGK8hdEQ9
uDYqy/TlbT8=
=JnWB
-----END PGP SIGNATURE-----

From p.f.moore at gmail.com  Wed Jan 31 16:44:06 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 Jan 2007 15:44:06 +0000
Subject: [Python-Dev] Scoping [Patience, please]
In-Reply-To: <FACD4404-113F-4FC8-BBAE-A285F7FAD619@gmail.com>
References: <FACD4404-113F-4FC8-BBAE-A285F7FAD619@gmail.com>
Message-ID: <79990c6b0701310744y6db4835chb99d9e6b4e0b5eaa@mail.gmail.com>

On 31/01/07, lingwitt at gmail.com <lingwitt at gmail.com> wrote:
> Fortunately, the scopes in question are always named.
> Can't we just reference what's already there?
>
>  >>> x = 10
>  >>> def a():
> ...     x = 3
> ...     def b():
> ...         a.x = 4
> ...         print a.x
> ...
> ...     b()
> ...     print x
> ...
>  >>> a()
> 4
> 4
>  >>> print x
> 10

There is further research you need to do, as this option has been
discussed before. I'm not sure if it's in the PEP, but it has
certainly come up on the mailing list - you should check the archives
for more information.

Basically, you can't use "a.x" as the function a has not yet been
created - you are still executing the "def". So you don't have access
to the function object's attributes in the way you need. (The
technical details are beyond me, and that explanation probably shows
it, but I hope you get the gist :-))

> Of course, the module itself is unnamed, and it would
> be completely consistent to use the single prefixed dot
> to reference it:
>
>  >>> x = 10
>  >>> def a():
> ...     x = 3
> ...     def b():
> ...         a.x = 4
> ...         print a.x
> ...
> ...    .x = 5
> ...     b()
> ...     print x
> ...
>  >>> a()
> 4
> 4
>  >>> print x
> 5

Yuk.

> However, I would think a placeholder is easier to spot:
>
>  >>> x = 10
>  >>> def a():
> ...     x = 3
> ...     def b():
> ...         a.x = 4
> ...         print a.x
> ...
> ...    @.x = 5
> ...     b()
> ...     print x
> ...
>  >>> a()
> 4
> 4
>  >>> print x
> 5

Double yuk.

> Be gentle.

Sorry, I could probably have been more gentle :-) Seriously, thanks
for your interest, and don't be put off, but if you pick up on a
long-standing proposal like this, you really do need to do a lot of
research. Many of the obvious, a lot of the not-so-obvious, and even
some of the downright stupid options will have already been discussed,
and you can't always guarantee that everything gets captured in the
PEP. Checking the list archives is always a good idea.

If you want to get involved without needing to do quite so much
pre-work, joining in on one of the current threads is often a good way
to start.

Hope this helps,
Paul.

From guido at python.org  Wed Jan 31 17:00:29 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 31 Jan 2007 08:00:29 -0800
Subject: [Python-Dev] Happy Birthday, Guido!
In-Reply-To: <B2A306F2-0821-4095-8229-D8E9FEF13A09@python.org>
References: <20070131151104.GA11441@panix.com>
	<B2A306F2-0821-4095-8229-D8E9FEF13A09@python.org>
Message-ID: <ca471dc20701310800k7fad21d5s6f63c5c6fa252332@mail.gmail.com>

Thank you guys! Now I want something fun to do with my life again too! :-)

On 1/31/07, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Jan 31, 2007, at 10:11 AM, Aahz wrote:
>
> > Thanks again for giving me something fun to do with my life.  ;-)
>
> Here, here!
> - -Barry
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (Darwin)
>
> iQCVAwUBRcC0nnEjvBPtnXfVAQI2+AP/QcOBBiGI/zDreuFU2bjq0MNH8qIocHnA
> y2XkoaXFDNYlOO6Uj39kttFl8HecMmyTyLvcyMTFBDxxN0xxQBpOPBkzFe6D44HQ
> x8R/VsaSY8wB3G4NZyYOoBjJMPPTN1x6U3lg+jzHkuRE4SSFYZmoWuvUGK8hdEQ9
> uDYqy/TlbT8=
> =JnWB
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip at pobox.com  Wed Jan 31 17:43:37 2007
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 31 Jan 2007 10:43:37 -0600
Subject: [Python-Dev] Happy Birthday, Guido!
In-Reply-To: <20070131151104.GA11441@panix.com>
References: <20070131151104.GA11441@panix.com>
Message-ID: <17856.51001.751236.495441@montanaro.dyndns.org>


    aahz> Thanks again for giving me something fun to do with my life.  ;-)

Not to mention a very good way to earn a living. ;-)

Skip

From jcarlson at uci.edu  Wed Jan 31 20:11:42 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Wed, 31 Jan 2007 11:11:42 -0800
Subject: [Python-Dev] Happy Birthday, Guido!
In-Reply-To: <ca471dc20701310800k7fad21d5s6f63c5c6fa252332@mail.gmail.com>
References: <B2A306F2-0821-4095-8229-D8E9FEF13A09@python.org>
	<ca471dc20701310800k7fad21d5s6f63c5c6fa252332@mail.gmail.com>
Message-ID: <20070131110404.5A56.JCARLSON@uci.edu>


"Guido van Rossum" <guido at python.org> wrote:
> Thank you guys! Now I want something fun to do with my life again too! :-)

You could hand BDFL responsibilities off onto perhaps Tim, Raymond, or
Martin for a few months and try to convince Google to open an office in
Hawaii.  Pick up one of those Toughbook laptops, and you could be coding
on the beach all year round.  Need to wait for some unittests to run?
Catch a wave or two.  Feeling a little stressed?  Have a mimosa. 
Getting too much sun?  Sit in the shade.

The possibilities are endless. :)


 - Josiah


From steve at holdenweb.com  Wed Jan 31 23:04:05 2007
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 31 Jan 2007 22:04:05 +0000
Subject: [Python-Dev] Happy Birthday, Guido!
In-Reply-To: <B2A306F2-0821-4095-8229-D8E9FEF13A09@python.org>
References: <20070131151104.GA11441@panix.com>
	<B2A306F2-0821-4095-8229-D8E9FEF13A09@python.org>
Message-ID: <45C11255.8090207@holdenweb.com>

Barry Warsaw wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Jan 31, 2007, at 10:11 AM, Aahz wrote:
> 
>> Thanks again for giving me something fun to do with my life.  ;-)
> 
> Here, here!

Where, where?

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com
See you at PyCon?         http://us.pycon.org/TX2007

From steve at holdenweb.com  Wed Jan 31 23:04:05 2007
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 31 Jan 2007 22:04:05 +0000
Subject: [Python-Dev] Happy Birthday, Guido!
In-Reply-To: <B2A306F2-0821-4095-8229-D8E9FEF13A09@python.org>
References: <20070131151104.GA11441@panix.com>
	<B2A306F2-0821-4095-8229-D8E9FEF13A09@python.org>
Message-ID: <45C11255.8090207@holdenweb.com>

Barry Warsaw wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Jan 31, 2007, at 10:11 AM, Aahz wrote:
> 
>> Thanks again for giving me something fun to do with my life.  ;-)
> 
> Here, here!

Where, where?

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Blog of Note:          http://holdenweb.blogspot.com
See you at PyCon?         http://us.pycon.org/TX2007


From aahz at pythoncraft.com  Wed Jan 31 23:13:57 2007
From: aahz at pythoncraft.com (Aahz)
Date: Wed, 31 Jan 2007 14:13:57 -0800
Subject: [Python-Dev] Happy Birthday, Guido!
In-Reply-To: <45C11255.8090207@holdenweb.com>
References: <20070131151104.GA11441@panix.com>
	<B2A306F2-0821-4095-8229-D8E9FEF13A09@python.org>
	<45C11255.8090207@holdenweb.com>
Message-ID: <20070131221357.GA24074@panix.com>

On Wed, Jan 31, 2007, Steve Holden wrote:
> Barry Warsaw wrote:
>>On Jan 31, 2007, at 10:11 AM, Aahz wrote:
>>>
>>>Thanks again for giving me something fun to do with my life.  ;-)
>>
>>Here, here!
> 
> Where, where?

Therewolf
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"I disrespectfully agree."  --SJM